Liquid Cooling Cluster Monitoring Project at a Supercomputing Center

Supercomputing facilities combine extremely high computational performance with high thermal loads. Maintaining suitable operating temperatures is therefore an essential part of keeping computing nodes available.

The supercomputing centre in this project provides 10 EFLOPS of computing power and operates 300 liquid-cooled computing nodes. Rather than relying solely on conventional air cooling, the facility uses fully immersive liquid cooling technology with customised PG25 coolant.

In an immersion environment, the requirements placed on monitoring equipment can be particularly demanding. Components may remain continuously exposed to coolant, so protection against liquid ingress and reliable long-term operation become critical considerations. At the same time, coolant condition needs to be monitored.

A liquid cooling water quality sensor can provide information about selected coolant parameters and help operators identify changes that may require attention. pH monitoring, for example, can be used to observe changes in coolant acidity or alkalinity.

For a facility containing hundreds of monitoring locations, however, collecting individual measurements is not enough. Data also needs to reach a central platform where operators can observe conditions across the cooling infrastructure. The project therefore required a monitoring solution that combined physical durability, communication capability and predictive monitoring.

The first major problem involved the protection rating of the previous monitoring equipment. The immersive cooling environment required sensors to remain continuously submerged in coolant. However, the existing sensors offered only IP67 protection.

Over six months of operation, 17 sensors suffered water ingress and were damaged.

Repeated sensor failures can create gaps in coolant monitoring and increase the amount of maintenance required across a large facility. With hundreds of monitoring locations distributed throughout the infrastructure, replacing damaged devices can also increase the workload of operations personnel.

The new coolant water quality sensor solution therefore needed stronger protection for continuous immersion.

Power quality presented a second challenge. The supercomputing centre experienced significant voltage fluctuations. The conventional monitoring devices supported only a 12–24 VDC input range. When the supply voltage became unstable, sensor operation could be interrupted and monitoring data could be lost. The resulting data gaps made it more difficult for operators to maintain a continuous view of coolant conditions.

For a monitoring system expected to support trend analysis, continuity is especially important. Missing measurements can weaken the data record used to understand how conditions change over time. The facility therefore needed sensors capable of operating across a wider voltage range.

The cooling infrastructure contained 300 monitoring points distributed across different cabinets.

Under the previous arrangement, centralised management was difficult. Maintenance personnel had to inspect individual cabinets, creating a substantial operational workload. As computing infrastructure grows, this type of manual monitoring becomes increasingly inefficient.

The new monitoring solution therefore needed to transmit sensor information to the centre’s existing operations and maintenance platform, allowing multiple monitoring locations to be managed centrally.

The existing monitoring system was mainly reactive.

It lacked trend-analysis capabilities that could help maintenance personnel anticipate changes in coolant pH. As a result, teams generally responded only after an abnormality or fault had already occurred.

This approach limited opportunities for preventive action. The centre therefore wanted to move beyond basic monitoring towards a system capable of analysing pH trends and providing earlier warning of abnormal changes.

The challenges identified at the facility created several clear monitoring requirements.

First, the sensors needed to tolerate continuous immersion. The previous IP67 devices had already demonstrated that insufficient protection could result in water ingress and premature sensor failure.

Second, the monitoring equipment needed a wider acceptable input-voltage range. Voltage fluctuations were a known characteristic of the site’s power environment, so sensors needed to continue operating when supply conditions changed.

Third, communication needed to support centralised management.

With hundreds of monitoring points spread across different cabinets, operators required a way to bring monitoring information into the existing operations and maintenance platform rather than checking each cabinet separately. The system also needed to support a more proactive maintenance strategy.

Instead of simply reporting the current pH value, collected data needed to support trend analysis so that unusual changes could potentially be identified before they developed into more serious conditions.

The main monitoring requirements therefore included:

• Reliable continuous immersion in customised PG25 coolant
• Improved protection against liquid ingress
• Wider input-voltage tolerance
• Centralised collection of monitoring data
• RS485 communication
• Modbus-RTU protocol compatibility
• pH trend analysis
• Early warning of abnormal pH changes
• Reduced dependence on individual cabinet inspections

Together, these requirements formed the basis of the new liquid coolant quality monitoring system.

The supercomputing centre selected the RK500-12 liquid cooling-specific pH sensor for the project.

A total of 120 sensors were deployed in three phases. The sensors were installed on the side walls of the cooling systems and connected through an RS485 bus to the centre’s existing operations and maintenance platform. This architecture allowed pH measurements from the deployed monitoring locations to be managed centrally.

The RK500-12 sensors provided IP68 protection, addressing the water-ingress problems experienced with the previous IP67 devices. This protection level enabled the sensors to operate continuously while immersed in coolant.

The sensors also supported a 7–30 VDC wide-voltage power input. This wider input range was particularly important because the facility experienced substantial voltage fluctuations. By accommodating a broader range of supply conditions, the sensors could maintain operation where the previous 12–24 VDC devices had experienced interruptions.

Modbus-RTU protocol support provided another important part of the solution. It allowed sensor data to be integrated with the centre’s existing operations and maintenance platform.

However, the project extended beyond hardware deployment. A pH trend-analysis and early-warning model was also developed using the monitoring data. This gave the facility the ability to analyse changes over time and identify developing abnormalities earlier.

The 120 RK500-12 sensors were deployed in three phases rather than through a single large-scale installation. The sensors were installed on the side walls of the cooling systems to support continuous monitoring of the customised PG25 coolant. Following physical installation, the devices were connected through an RS485 bus.

Modbus-RTU communication enabled the monitoring network to interface with the supercomputing centre’s existing operations and maintenance platform. This created a centralised monitoring environment for the deployed sensors.

The wide 7–30 VDC input range addressed the power instability experienced at the facility. Rather than relying on sensors with the previous narrower 12–24 VDC range, the new devices could accommodate more variable power conditions.

IP68 protection addressed another important commissioning requirement: reliable operation during continuous coolant immersion. Once monitoring data was available through the central platform, the project could move to the next stage: trend analysis.

A pH trend-analysis and early-warning model was developed to analyse monitoring data and identify abnormal pH changes. This added a predictive element to the monitoring strategy and allowed the operations team to move away from responding only after problems appeared.

The RK500-12 liquid cooling-specific pH sensor formed the core of the monitoring deployment.

For this project, one of its most important characteristics was its IP68 protection rating. The previous IP67 sensors had experienced water ingress during continuous immersion, so improved protection was essential for the operating environment. The RK500-12 also supported a 7–30 VDC power input. This helped the monitoring system accommodate the voltage fluctuations present at the supercomputing centre.

RS485 connectivity and Modbus-RTU protocol support allowed the sensors to communicate with the existing operations and maintenance platform. As a specialised coolant pH sensor, the device provided the pH data required for the project’s trend-analysis and early-warning model.

While this particular project focused on pH monitoring, liquid cooling systems can require different water-quality measurements depending on their coolant chemistry and operational requirements.

For example, a conductivity sensor for coolant monitoring can be considered when operators need to observe electrical conductivity as part of coolant management. A wider data center coolant water quality sensor strategy may combine pH with other relevant parameters when required by the application.

The appropriate sensor combination should be determined by the coolant type, cooling-system design, monitoring objectives and operating conditions rather than using the same configuration for every liquid cooling installation.

The completed project delivered measurable improvements in monitoring reliability and system management.

Before the project, 17 IP67 sensors had suffered water ingress and damage within six months. Following deployment of the IP68-protected sensors, the water-ingress damage rate dropped to 0%. The sensors operated continuously for 12 months without failures. Mean Time Between Failures reached 15,000 hours, demonstrating improved reliability under the facility’s continuous immersion conditions.

The previous monitoring devices had struggled with the facility’s voltage fluctuations. With the 7–30 VDC wide-voltage design, data loss associated with unstable power conditions was eliminated. The overall data loss rate decreased from 12% to 0%. This provided a more continuous monitoring record and supported more reliable analysis of pH trends.

Connecting the sensors to the operations and maintenance platform through an RS485 bus addressed the difficulty of managing distributed monitoring points. Rather than depending on individual cabinet inspections for routine observation, sensor information could be integrated into the centre’s central platform. This provided a more practical foundation for managing coolant monitoring across the supercomputing infrastructure.

The trend-analysis and early-warning model introduced a predictive capability that was absent from the previous system. According to the supplied project results, the model could predict abnormal pH changes up to seven days in advance. This allowed the maintenance strategy to shift from reactive response towards proactive prevention. Instead of waiting until an abnormal condition or fault had already developed, personnel could use trend information to identify potential pH changes earlier and plan appropriate action.

The combination of improved sensor reliability, elimination of power-related data loss and predictive pH monitoring contributed to improved system availability. Following implementation, supercomputing system availability reached 99.995%. For infrastructure delivering 10 EFLOPS of computing power, reliable monitoring supports the wider objective of maintaining stable and highly available computing operations.

This supercomputing centre project demonstrates how the requirements for a liquid cooling water quality sensor can change when monitoring equipment must operate continuously inside an immersive cooling environment.

The previous monitoring arrangement suffered from three fundamental limitations: insufficient protection against continuous immersion, sensitivity to power-supply fluctuations and limited centralised management. It also provided little capability for predicting changes in coolant conditions.

Deploying 120 RK500-12 liquid cooling-specific pH sensors addressed these challenges through IP68 protection, a 7–30 VDC input range, RS485 connectivity and Modbus-RTU integration.

The results were measurable. Water-ingress damage fell to 0%, power-related data loss was eliminated, the overall data loss rate dropped from 12% to 0%, and MTBF reached 15,000 hours.

Most importantly, the pH trend-analysis and early-warning model could identify abnormal pH changes up to seven days in advance. This enabled the centre to move from responding to coolant abnormalities after they occurred towards a more proactive monitoring approach.

For supercomputing centres and other high-density computing facilities, effective liquid coolant quality monitoring is therefore not limited to collecting individual measurements. Reliable sensors, suitable environmental protection, stable communication, centralised data management and trend analysis can work together to provide a more complete approach to managing coolant conditions in demanding liquid cooling systems.