Opportunities for improving energy efficiency in existing data centres
21 October 2019
Dublin is one of top locations for building DCs in Europe along with London, Frankfurt, Amsterdam, Paris and Sweden. In addition to this demand for new DCs there is an ongoing need for modernising of older DCs built in the early 2000s.
Brendan Dervan writes on the opportunities for improvements in energy efficiency during data centre modernisation programmes.
The advent of cloud computing, social media, online retail, video streaming, cloud storage and the like are driving the demand for new data centres (DCs).
Dublin is one of top locations for building DCs in Europe along with London, Frankfurt, Amsterdam, Paris and Sweden.
In addition to this demand for new DCs there is an ongoing need for modernising of older DCs built in the early 2000s.
Besides the obvious need to employ newer ICT technologies many of DCs are approaching 20 years in operation and much of the M&E plant has reached end of life and needs to be replaced.
This leads to opportunities to carry out other upgrade works including improvements in energy efficiency and resilience and various risk abatement works.
Increased efficiency in power and cooling systems frees up power capacity for ICT equipment. This article focuses on the opportunities for improvements in energy efficiency during DC modernisation programmes.
Energy usage metrics
The two energy efficiency metrics which are widely used in the DC industry are (i) Power Usage Effectiveness (PUE) and (ii) DC Infrastructure Efficiency (DCIE). PUE, the more common metric, is the ratio of Total Facilities Energy to IT energy.
For example, a DC with an ICT equipment consumption of 100 GWH and a facilities consumption of 100 GWH would result in: a PUE of (100+100) / 100 = 2.
Figure 1 shows typical energy usage in an existing legacy DC resulting in a PUE of 2. The modern high performance DCs built in the last 10 years have PUEs of 1.2 and lower. The aim of many DC modernisation programmes is to reduce the PUE as low as possible.
The Uptime Institute (UI) is a consortium of companies that engage in the DC industry and is best known for its widely adopted tier classification in relation to resilience. Four tiers, I to IV are recognised based on their level of resilience and redundancy.
Redundancy in the context of DCs refers to the ability of the power or cooling system to stay operational in the event of a ‘module’ or ‘path’ failure or both.
The level of redundancy is expressed in terms of N, N+1, 2N or 2N+1 as indicated in the example in Figure 4 below. The example is based on an essential load of 200 KVA which is supplied through a number of 100 KVA UPS modules.
There is no redundancy provided with the ‘N’ arrangement. The 2N arrangement with four modules each carrying 25 per cent of the load is typically what we would expect in a Tier III site whereas the 2N+1 arrangement with six modules each carrying 16.66 per cent of the load is more typical of a Tier IV site.
The point is, higher resilience means higher capital outlay and higher energy costs due to plant running inefficiently at very low part-load.
ASHRAE temperature and humidity ranges
One of the most significant developments in the DC industry in recent years has been the acceptance of increased temperature and humidity ranges. This development has been largely driven by ASHRAE TC 9.9 which is summarised in their 2015 white paper.
These wider ranges have resulted from improved design in the heat dissipation characteristics of ICT equipment. Adopting wider ranges of temperature and humidity results in significant savings in energy and brings the PUE closer to unity.
The traditional air flow requirement for many years has been in the order of 70 l/s per kW of ICT equipment load based on temperature difference (Δt) of 11˚C.
However, due to improvements in the thermal and air management capabilities of ICT equipment, higher temperatures and therefore lower air flow rates are now acceptable.
Furthermore, fan controls within the ICT equipment allow reduced fan speed at low utilisation levels and/or low inlet temperatures. All of the above factors have led to a reduction in overall fan power required.
After the ICT load the largest single consumer of energy in DCs is the mechanical cooling systems. When it comes to cooling solutions there are numerous options available.
Figure 3 summarises some of these options starting with the heat rejection from within the IT rack, the transport fluid (that is, chilled water, refrigerant or air) and the final heat rejection to the atmosphere.
The diagram covers all options from the traditional chilled water systems up to the more common and more energy efficient solution of using direct or indirect evaporative cooling systems.
A point worth noting at this stage is the difference between a CRAC (computer room air conditioning) unit and a CRAH (computer room air handling) unit.
A CRAC is defined in some data centre literature as a computer room air handling unit with integral compressor, that is, a DX system, while a CRAH is defined as a computer room air handling unit with a chilled water coil.
The term CRAC throughout this report shall mean computer room air handling unit designed to condition the air by means of a chilled water or DX cooling coil and a humidifier if required.
Most existing DCs use either chilled water or ‘DX’ (direct expansion) systems. Both of these are refrigerant based systems which use compressors to achieve cooling.
The traditional approach has been to locate the CRACs around the internal perimeter of the data hall. Variations of this design include roof top units and external perimeter units.
Following the relaxation of temperature and humidity set-points by ASHRAE a number of ‘high efficiency cooling’ trends have developed in an effort to improve energy efficiency in DCs.
These are often referred to as ‘free cooling systems’, however this is really a misnomer as fans still have to be used to deliver the required air flow.
Economisation is another term which is widely used and perhaps is more appropriate. Systems can be broadly divided into ‘air-side’ and ‘water-side’ economisation and in each case there are direct and indirect methods.
With direct air-side economisation, the external air is used to cool the space. Cooling may still be required to deal with humidity and high external temperatures if necessary.
The exhaust air can be recirculated and mixed with fresh air to avoid unnecessary humidification and/or dehumidification.
Water-side economisation systems generally comprise an air to water heat exchanger, which can be either part of the chiller or a separate dry cooler.
The heat exchanger removes heat from the chilled water loop whenever the return water is higher than the ambient air temperature. This reduces the demand on the mechanical refrigeration cycle and saves considerable energy.
Evaporative or adiabatic cooling systems typically as shown in Figure 4, are the most popular choice for most new DCs. These systems use the latent heat of evaporation to produce lower air temperatures.
To avoid the obvious hazard of excessive humidity the approach is generally to use air to air heat exchangers with the evaporative cooling facility reducing the temperature of the primary (non-DC) air.
In some conditions, evaporative coolers can use as little as a quarter of the electrical energy that an equivalent vapour compression type refrigerant cooler uses.
However, the downside is that substantial structural openings need to be formed in the external walls and extensive ducting is required.
As a result evaporative cooling is not normally a viable option for existing DCs. On the other hand, water-side economisation systems can be easily deployed without any major risk to the DC itself as the system modifications take place out in the chiller compound and therefore there is no impact on the existing internal cooling infrastructure.
Due to the increasing power densities in modern DCs there is a growing trend towards placement of the cooling units ‘in-row’, that is, within the row of cabinets.
This option requires hot or cold aisle containment as discussed below. In very high density applications, a dedicated cooling unit may be provided for a rack or a small group of adjacent racks.
The air is contained within the racks and therefore there is no need for aisle containment. Both ‘in-row’ and ‘n-rack’ cooling are regarded as scalable designs which allow the cooling capacity to grow in tandem with the ICT load.
Air flow management
Early generation DCs used racks are bayed together in rows with intervening uncontained hot and cold aisles as indicated in Figure 5.
The air flow strategy is based on delivering cooled air through a pressurised raised access floor plenum which in turn delivers it to the racks via perforated floor tiles located in a shared aisle which is referred to as the ‘cold aisle’.
The lack of containment results in a number of air management issues. First, ‘short-circuiting’ occurs when hot return air from the rear of the rack recirculates to the front mixing with cold air and resulting in inefficient diluted warm air entering ICT equipment.
Second, ‘bypassing’ occurs when cooled airflow returns to CRACs without passing through the ICT equipment. Both of these conditions result in low return air temperatures which reduces the overall efficiency of the cooling system.
Due to the increasing power densities in ICT racks and the poor energy efficiency associated with short-circuiting and bypassing contained aisle solutions have become the norm in the past decade.
Many DCs have successfully modified existing uncontained hot and cold aisle arrangements to contained systems by adding doors at each end of the aisle and a roof.
A more basic approach using cold room curtains has been employed in some instances. In addition to aisle containment, sealing of floor openings and installation of blanking panels in unused cabinet bays is an essential part of these works.
Considerable energy savings can be achieved by the simple aisle containment solutions described above. Research has shown that payback period as short as six months can be achieved in some legacy DCs which makes this one of the most attractive of all the investments considered.
Distribution in DCs is generally at 10kV or 20kV depending on the site configuration. Larger campus type sites with long cable runs would tend to use 20kV whereas DCs within a single building are more likely to opt for 10kV.
Two types of transformers are commonly used, oil filled and dry cast-resin type. The choice is generally dependent on the location in the building and the hazards associated with a transformer oil leak.
Recent design developments have resulted in lower iron and copper losses in transformers, however it is unlikely that capital cost for replacement of a transformer would outweigh the benefit from the improved efficiency.
Generator systems and demand side management
A typical standby generator installation for a DC comprises the diesel engine powered generator set, the enclosure or plant room, the fuel storage and distribution system, the ventilation system the switchgear and the control system.
Generator replacement would not normally form part of the scope of DC energy efficiency upgrade, however there is potential to use the generators to allow the DC operator to participate in a demand side management (DSM) scheme and in turn earn considerable revenue which could contribute to the capital cost of the energy efficiency upgrade.
DSM generally refers to the management of consumer demand in an energy system with a view to reducing peaks in the system.
In general it involves offering financial incentives to the major consumers such as DC operators to reduce their demand during peak periods.
The incentives are usually in the form of savings in the tariff structure.
In the current DS3 Programme a Demand Side Unit (DSU) is an individual demand site or aggregated demand site with a demand reduction capability of at least 4 MW.
DSUs reduce demand when requested by the EirGrid using an Electronic Dispatch Instruction Logger (EDIL) system. Payments are made to DSUs for simply being available to reduce their demand while further payments are made when demand reduction is actually implemented at EirGrid’s request.
Customers can sign up to a fixed or variable demand reduction. For ‘fixed’ subscriber there is an agreed price per MW of reducible load which is typically in the order of €40,000 – €50,000 per MW per year depending on the speed of reduction, the number of outages per year and so on.
For variable subscribers the rate per MW of reducible demand is based on the half hour period in which the order is dispatched. In general a higher rate applies to times of peak demand which is typically winter peak demand period of 5-7pm Monday to Friday, November to February inclusive.
The DSU aggregator also pays the subscriber for any generator fuel consumed during the demand reduction period. In general those individual periods will not exceed two hours in duration and no more than 20 demand reduction events will be requested in a 12-month period.
DCs with their high levels of resilience are ideal customers for participation in these DSM schemes which allows them to convert an otherwise standby asset into a revenue stream which can assist in the modernisation programme.
An area worthy of further research is the electrical energy consumed by the generator block heaters and heating of the generator plant room itself.
There is very little data available on this at present. However, based on reasonable assumptions, research has shown that significant amounts of electrical energy is used to maintain generators at their optimum temperature and this could be easily displaced with recovered heat from the UPS plant rooms or elsewhere in the DC.
An uninterruptible power supply (UPS) system is an essential element of the power distribution infrastructure in every DC and is a prime target for improvement in energy efficiency.
Its primary function is to provide a clean uninterruptible supply to the ICT equipment and protect it from voltage disturbances such as transients, sags, surges, and brownouts which are associated with utility networks.
Online systems are also known as double conversion systems offer the highest level of power quality and protection against interruptions. They are classified as VFI meaning their loads are ‘voltage and frequency independent’ of the mains.
Online systems usually comprise five main elements; a rectifier, an inverter, a battery, a static bypass switch and a maintenance bypass switch. Figure 6 shows a simplified diagram of how each of the elements is interconnected to form the complete UPS system.
Within the online category there are two further sub-categories namely ‘transformer based’ and ‘transformerless’ systems.
To generate an RMS phase voltage of 230VAC a DC voltage of twice the peak value is required, that is, (230/0.707)*2 = 650VDC.
However most UPS systems limit the DC voltage to around 380V to keep the battery to a minimum size and therefore the inverter output needs to be stepped up using a transformer.
The iron and copper losses in these transformers reduce the overall efficiency of the UPS and increase the input power factor at low load.
In recent years there has been a move towards transformerless designs which use booster converters to step up the rectifier output before it feeds into the inverter.
For DC applications, batteries are normally 10- to 15-year valve regulated sealed lead acid (VRSLA) type. Very often it is more economical to replace the entire UPS system at the same time as the battery given the current generation of UPS systems are considerably more efficient than the older legacy systems.
Many DCs are still using UPS systems with power conversion technology developed in the late 1990s. Efficiencies can be as low as 85 per cent compared with current generation technology which can achieve efficiencies in the order of 96 per cent at similar part load conditions.
Based on a 1MW critical load running 24X7 and say a cost of €120/MWh the difference in running costs between a legacy UPS system running at 85 per cent and a modern modular system operating at 96 per cent efficiency is in the order of €141,000 per year as shown in Table 2.
Financial appraisal techniques
Investment in energy improvements like any other investment must be subjected to a detailed financial appraisal to determine the project costs and the likely benefits that will arise.
DC operators will want to fund the option that will yield the greatest financial return. In some cases the complexity and associated costs of upgrading a live DC can exceed the cost of building a new facility and therefore detailed financial appraisal of the competing options needs to be carefully considered in all cases.
Payback period is by far the simplest financial appraisal method and the one that is most widely used despite its short comings. It basically indicates the length of time it takes for the cost savings resulting from an investment to equate to the original capital outlay.
It is ideal for making a simple comparison of investments however it has a number of obvious flaws. First of all, it does not indicate what savings are achievable after the payback period which could be substantially different for two competing investments.
Second, it does not consider the time value of money, that is, that the sum invested could have earned interest by investing elsewhere. The net present value (NPV) method, unlike the payback method above, does take into account the effect of time on future cash flows resulting from an investment.
The total cost of ownership (TCO) of competing options should always be compared. TCO is the total life cycle cost and includes the initial capital outlay, energy costs, maintenance costs and final decommissioning.
The initial capital outlay of a UPS system is a very small portion of its TCO so comparing two competing options on capital cost along would not be recommended.
Replacement of end of life M&E plant in DCs provides an ideal opportunity to improve the overall energy efficiency. Upgrading cooling and UPS systems and improving air management within the data halls are the obvious choices to achieving a lower PUE.
This in turn results in additional power capacity becoming available for data processing equipment which in theory should yield more revenue for the DC.
Consideration should be given to participation in a demand side management scheme which is an effective revenue stream to fund some of the energy improvement works.
In some cases the complexity and associated costs of upgrading a live DC can exceed the cost of building a new facility and therefore detailed financial appraisal of competing options needs to be carefully considered in all cases.
Author: Brendan Dervan is a chartered engineer with more than 40 years’ experience in building services. Since starting his own firm, Dervan Engineering Consultants (DEC) in 1999, he has been involved in the design and project management of M&E services for existing and new DC facilities. He retired from mainstream consultancy in 2019 and set up Best Training which provides specialist CPD services to the M&E sector. Courses available include: Role of the M&E Consultant; Data Centres – Introduction to M&E Services; LV Power Distribution Design; MV Power Distribution Design.