Direct-to-chip liquid cooling: where the piping and flanges are
Direct-to-chip cooling puts a cold plate on each processor and pumps coolant through it. Most joints at the rack are quick disconnects, but behind the racks sit headers, CDUs and mains that are welded and flanged.
In a direct-to-chip system, coolant flows from the coolant distribution unit (CDU) through row or rack manifolds and quick disconnects to cold plates on the processors, and back. The flanged joints are upstream of the racks: CDU nozzles, technology cooling system (TCS) headers, isolation valves, strainers and filters, and the facility water mains. On the TCS those flanges are 316L or 304L stainless; on the facility side they follow the operator's specification.
How the coolant reaches the chip
| Component | What it does | Typical joints |
|---|---|---|
| Cold plate | Metal block on the processor with fine internal channels | Brazed or threaded fittings, made by the server or cold plate maker |
| Rack manifold | Shares coolant between the servers in a rack | Quick disconnect couplings to each server, for example to the Open Compute Project UQD specification |
| Row manifold or TCS header | Feeds the racks in a row or a data hall | Butt welded or flanged, with flanged isolation valves at branches |
| CDU | Heat exchanger, pumps, filters and controls between TCS and facility water | Flanged inlet and outlet nozzles on both sides |
| Facility water mains | Carry the heat from the CDUs to chillers, dry coolers or heat reuse | Welded, flanged or grooved, to the facility specification |
The CDU is the boundary. Everything between the CDU and the cold plates is the technology cooling system, which has stricter rules for materials and cleanliness than the facility water side. FWS vs TCS explained.
Cold plates take the heat from the processors and other plated parts. Memory, drives and power supplies still shed heat to air, so a liquid-cooled hall still needs air cooling, fan coils or rear-door heat exchangers for the rest. The other liquid route, immersion cooling, puts the whole server in a dielectric fluid instead.
Flow: how much coolant per rack
Water carries about 4.2 kJ per kilogram for each degree it warms. With a 10 K rise across the cold plates, each kilowatt of chip heat needs about 1.4 liters per minute of water. A glycol mix such as PG25 carries a little less heat, so it needs a few percent more flow.
| Heat to liquid | Flow | Flow (US gpm) |
|---|---|---|
| 1 kW | 1.4 L/min | 0.38 |
| 50 kW rack | 72 L/min | 19 |
| 100 kW rack | 143 L/min | 38 |
| Row of ten 100 kW racks | 1,430 L/min | 380 |
A row like the last one needs a header of roughly NPS 5 to NPS 6 at normal water velocities, which is why TCS headers are flanged stainless pipe rather than hose. Halve the temperature rise and the flow doubles.
Materials on each side of the CDU
- Technology cooling system: stainless steel and copper alloys only. The Open Compute Project sets stainless pipework of at least grade 304 with surfaces treated to ASTM A380. Order 316L or 304L flanges, pickled and passivated, with 100% PMI.
- Gaskets: wetted parts too. Use EPDM or PTFE from the CDU supplier's approved list. Flange gaskets for water and glycol loops.
- Facility water: carbon steel A105 is common, with stainless near the white space on some projects. Where the two meet, consider an insulating joint.
Cold plate channels are about 100 microns wide. A few grains of scale, weld oxide or rust are enough to block them, so the TCS has to arrive clean and stay clean. That is why pickled and passivated flanges, purged welds and a full flush come before the first rack is connected. TCS flushing and cleaning.
Where the flanges are, joint by joint
| Location | Usual flange | Read more |
|---|---|---|
| CDU secondary nozzles (TCS side) | 316L weld neck or lap joint with stub end, Class 150 or PN16 | CDU flanges |
| TCS headers, branches and valves | 316L weld neck, slip-on or lap joint, Class 150 | TCS flanges |
| Header ends left for future racks | 316L blind flanges | Blind flanges |
| CDU primary nozzles (facility side) | A105 or stainless weld neck, Class 150 or PN16 | Facility water flanges |
| Facility water mains | A105 weld neck or slip-on; B16.47 or AWWA C207 at large sizes | Carbon steel flanges |
Frequently asked questions
Are quick disconnects part of the flange specification?
No. Quick disconnects connect each server or rack to the manifold and are bought with the rack or manifold. The headers, valves and CDU connections they hang off are welded or flanged, and those flanges belong in the piping specification.
Is direct-to-chip cooling single-phase or two-phase?
Most direct-to-chip systems today are single-phase: water or a water and propylene glycol mix stays liquid through the cold plate. Two-phase systems boil a refrigerant or engineered fluid in the cold plate and use different, mostly brazed or threaded, joints on the rack side.
Does direct-to-chip cooling remove all of the server heat?
No. Cold plates take the heat from the processors and other plated parts, but memory, drives and power supplies still shed heat to air, so the room still needs air cooling, rear doors or fan coils for the rest.
Which flange material suits the technology cooling system?
316L or 304L stainless steel, pickled and passivated. The Open Compute Project guidance for the TCS calls for stainless pipework of at least grade 304 with internal surfaces treated to ASTM A380, and plain carbon steel is ruled out because it rusts.
Sources
- Open Compute Project, guidelines for pre-commissioning preparation of TCS row manifolds; Universal Quick Disconnect (UQD) specification
- ASHRAE TC 9.9, liquid cooling guidance for data processing environments