By Kurt Lender, Marketing Work Group Co-Chair, PCI-SIG® and Casey Morrison, PCI-SIG member

The first installment of the PCI-SIG® educational webinar series, “Retimers to the Rescue: PCI Express® Specifications Reach Their Full Potential” premiered on October 9, 2019. Attendees learned about the diagnostic capabilities of Retimers, how to solve signal integrity problems and the status of Retimer Compliance Testing. The webinar also examined insertion loss issues, how to design with a Retimer and compared reach extension solutions.

We received a multitude of insightful questions from the audience but ran out of time to answer them during the live presentation. The answers to all questions received on and off screen are below. 

PCI-SIG Retimer Webinar Audience Questions

Retimer Design

1. When designing a Retimer, can you provide the calculations that went into step two of determining which slots required a Retimer?

It comes down to comparing the length and loss of each port in the topology against the length/loss which is possible with a given Printed Circuit Board (PCB) material. To start, the following table lists representative loss-per-inch figures for different PCB materials.

Next, if you take the PCI Express (PCIe®) 4.0 and PCIe 5.0 maximum channel specification, subtract the Root Port package loss, add-in card (AIC) loss, CEM connector loss, and the loss of two vias (0.7 dB each), then the remainder for the system board, in terms of inches, is listed in the table below.

With this knowledge, we then look at the system topology—the different ports and the distance from the ports to the CPU. If the distance from the port to the CPU exceeds the “Max system board trace, worst-case (WC) and 15% safety margin” row in the table above for the given data rate, then a Retimer may be required, depending on how much of a safety margin the system designer wants to reserve.

The two “Maybe” cases highlighted below in red barely meet the 8-inch reach capability of ultra-low-loss PCB material; however, if a Riser card will be used, the system budget will be exceeded and a Retimer will be required.

The “Maybe” case highlighted in purple will need a Retimer even if a Riser is not used, assuming the system designer wants to reserve a 15% safety margin.

Of course, if all these ports must support 32 GT/s speed, then more will require a Retimer.

2. Do Retimers need to support Separate Reference Independent Spread (SRIS) as per the PCIe specification?

According to Section 4.3.9 of the PCIe Base Specification, “Retimers are permitted but not required to support SRIS.” There are special latency requirements for Retimers which do support SRIS (see Base Specification table 4-30). It is worth noting that, when SRIS is enabled in a system, the Root Complex, Retimer and Endpoint must all support SRIS.

3. Why would you use a Retimer when an internal cable is used?

This depends on the topology and system requirements. Internal cables are often used for storage applications. If the individual channel components (i.e. the storage mid-plane, cable connector, the cable itself, and the connector into which the cable plugs into on the system board side) are able to meet the loss requirement and some amount of safety margin, then arguably a Retimer would not be required. Many times, when there are multiple connectors in the path, each of these connectors—including the cable connectors—introduce loss, discontinuities and cross-talk. It’s often a safer option to use an active break-out card instead of a passive one. The advantage of leveraging a Retimer is that users can drive a longer cable, deal with the signal from a loss perspective and compensate for reflections and cross-talk introduced along the way.

Retimer Size and Latency

4. Is there a standard footprint/pinout definition for all Retimers? What are the Retimer sizes (x4, x8, x16)?

Retimers do generally come in different widths and footprints/pinouts. There is a trend in the industry towards using a common footprint for x16, x8, and x4 Retimer form factors.

5. What’s the difference between using multiple x4 or x8 Retimers in the system versus x16?

Logically, using two x8 Retimers versus one x16 Retimer, which is bifurcated into two x8 Retimers, is the same. Practically speaking, the total solution size will be different. The industry-standard x16 footprint is approximately 10% smaller than two x8s or four x4s, and if we consider all of the surrounding components like decoupling, REFCLK, pin-straps, etc., the space savings from using a single x16 to support multiple ports can be substantial.

6. What is the main difference between a x16 Retimer and a 32-lane PCIe switch using 1 x16 upstream port and 1 x16 downstream port?

The main difference is latency. A Retimer is required to have no more than 64 nanoseconds of input-to-output latency at 16 GT/s and 32 GT/s. PCIe switches, which generally inspect the packets at the Data Link and Transaction protocol layers, will have significantly higher latency. In addition, power consumption and solution size (IC footprint, surrounding components, etc.) will generally be higher for switches than Retimers.

7. Are Retimers considered non-root ports in terms of their package loss?

Technically, the PCIe Base Specification considers Retimers as non-Root port devices when it comes to package loss. Since Retimer packages are typically much smaller than traditional Endpoints (for example, just 2 square cm for a x16), the loss in the Retimer’s package will probably be much less, more like <2 dB.

8. It was stated that the PCIe 6.0 specification reach would be the same between the PCIe 5.0 specification. Are you referring to 36-dB insertion loss budget?

The on-going discussions in the PCIe electrical working group (EWG) indicate that the same physical reach (i.e. typical server dimensions) will be targeted for the PCIe 6.0 specification; however, an exact loss budget in dB has not yet been defined.

9. In a simple number, how many dB boosts can a Retimer add?

For the PCI Express 3.0 specification, Retimers were ECN post the base specification and a compliance program was never done for them. For the PCIe 4.0 specification, Retimers were put into the base specification and the compliance program will ripple out. The first Integrators List came out of the PCI-SIG Compliance Workshop #110 in August, so it will be late this year or early on in 2020 for the Retimer test specification. We expect them to be in the PCIe 5.0 specification and will certainly be in the PCIe 6.0 specification. The channel loss is defined by the specification, so for the PCIe 4.0 specification it’s about 28 dB and for the PCIe 5.0 specification it’s about 36 dB. Essentially, a Retimer will provide another full loss channel.

10. Can you expand on the latency of a Retimer?

The standard does have some limits for latency. For the PCIe 4.0 and PCIe 5.0 specifications, it’s 64 nanoseconds in non-SRIS applications. Of course, for various applications, especially emerging applications with different kinds of workloads and some protocol extensions that are making use of the PCI Express physical layer, much lower latency is often desired and required. It’s fair to say that Retimer vendors are aware of the need to keep latency to a minimum, but the standard itself has an upper limit of 64 nanoseconds.

11. Do devices like PCIe Switches offer Retimer capabilities?

Essentially, one could look at PCIe switches as an expensive Retimer. Retimers will boost signals in some way, shape or form. Looking at switches today, they’re usually >x16, so there’s a different usage model there, and that’s not what a switch is generally used for. In theory, a switch could be used as a Retimer if it’s configured in some kind of 1:1 fashion. However, switches generally have more logic, which includes more processing in the middle of the two pseudo-ports. As a result, meeting the 64-nanosecond latency requirement would be a challenge. Even if the 64-nanosecond latency were met, many applications require much lower latency and lower power consumption.

Protocol

12. Are Retimers transparent regarding error correction and flow control protocols? If so, could the additional latency could impact RX and replay buffer sizing?

Yes, Retimers are transparent regarding error correction and flow control. The added latency (64 ns, maximum, according to the PCIe base specification), will need to be accounted for in replay buffer sizing.

13. Why is a link limited to two Retimers?

The primary reason is protocol related. For example, Control SKP Ordered Sets (OSs) have fields for “First” and “Second” Retimer Data Parity, which the Root Complex and Endpoint use to check for data parity errors in each link segment (i.e. the physical channels between Retimers). There is no field in Control SKPs for “third,” “fourth,” etc. Retimers.

Another reason is that a Retimer is permitted to add or remove SKP symbols for the purposes of frequency correction. A PCIe Receiver expects to receive SKP Ordered Sets with between one and five SKP symbols. When an SKP Ordered Set is transmitted by a PCIe Root Complex or Endpoint, it is transmitted with three SKP symbols. If two Retimers are present in the Link, each may remove one SKP symbol, leaving just one SKP symbol in the SKP OS received by the downstream device. If a third Retimer were present in the Link, then it may remove the final SKP symbol, which would relegate the SKP OS to having no SKP symbols—therefore it would be an invalid SKP OS.

14. Is there any clock jitter criteria for Retimers? Can Retimers recover their signal if the clock jitter rate is bad?

According to the base specification, the Retimer is expected to receive a reference clock, which complies with the reference clock requirements, just like any other PCI Express device like a CPU or an endpoint. The quality of the reference clock could potentially impact the Retimer’s ability to recover data, error-free, and it would affect the transmit jitter characteristics. While it may be possible for a Retimer to recover data if the clock jitter is bad, the expectation from the PCI Express specification, is that the Retimer is supplied with a compliant reference clock.

15. Is there any way to measure the Bit Error Rate (BER) of a Retimer?

Yes, there are a couple ways to measure the Bit Error Rate (BER) of a Retimer. When a Retimer is going through compliance testing, a BER tester is used and that test will measure the BER on both pseudo-ports that are being tested. Since they fully recover the data and the clock, Retimer often have various types of pattern checking capabilities. Some of the patterns that Retimers check for include Pseudo Random Binary Sequence (PRBS) patterns (PRBS 31 or PRBS 23). Retimers can often check for PRBS patterns and tabulate and accumulate any errors that may be seen. Another way is to have a BER tester or component (endpoint or root complex) generate a test pattern and have the Retimer check it. The third way to measure the BER is to use the loopback feature where a root complex, an endpoint, or some test equipment generates a pattern that is looped back at the Retimer’s pseudo-port and then tested for errors back at the source.

PCI Express Specification Status

16. What is the Retimer ecosystem enabling status on the PCIe 4.0 and PCIe 5.0 architecture?

Retimer requirements are specified in the Base Specification and a compliance test program is being developed. As of Q4 2019, that program is in “FYI” state and is expected to progress past the “FYI” state by early 2020.

We hope the webinar and these questions provided helpful information about Retimer technology. If you have any further questions about Retimers or have suggestions for future topics for the PCI-SIG educational webinar series, please contact us.

If you would like to watch the presentation again or subscribe to our BrightTALK channel to learn about future webinars, the recording of the live webinar is available on our channel here.