Data center rack density: How high can it go?
本文分析了数据中心单机架功率密度越来越高的来由，以及发展的趋势。人工智能(AI)和高性能计算(HPC)在各行各业的应用越来越广泛，同时各行业用户对时延的需求，使得靠近用户的高密度数据中心将成为一种趋势。就算是再小规模的高密机架部署，对于不管是提高用电密度、增大机柜尺寸还是通过改变冷却方式来提高散热能力，要求都越来越高。
人工智能(AI)和高性能计算(HPC)将对计算、存储能力和网络资源的需求推向了极限。算力更强大的芯片已成为驱动机架功率密度升高的主要因素。十年前，处理器的平均功耗还不到100瓦。如今，它们的能耗约为500瓦。
Artificial intelligence (AI) and high-performance computing (HPC) have pushed requirements to the limit for compute, storage capacity and networking resources. More powerful chips have been a primary factor in driving rack densities higher and higher. A decade ago, the average processor consumed less than 100 watts of power. Today, they consume around 500 watts.
在更小的空间中可用的功能越多，应用程序开发人员就会想出更多的方法来利用它——然后他们就需要更多的算力。Hyperion Research的首席执行官Earl Joseph表示，高性能计算的增长很大程度上是由于更高要求的应用的出现。
The more power that becomes available in a smaller space, the more application developers come up with ways to harness it – and then they demand even more compute power. Earl Joseph, CEO of Hyperion Research, said HPC growth is largely surging due to the advent of more demanding applications.
Earl Joseph说:”人工智能、机器学习和深度学习正在以每年近30%的速度增长。”
“AI, machine learning and deep learning are growing at close to 30% a year,” Joseph said.
让我们来看看（机架功率）密度是如何演变的，以及它可能会达到多高。
Let’s look at how density has evolved and how high it might go.
规划、订购和交付的数据中心数量创历史新高。据Uptime Institute的数据显示，到2025年，数据中心的单位面积功耗预计将增加50%，2019年至2025年期间全球产生的数据将增长500%。数据中心正在以前所未有的数量和远高于以往的机架密度建设。为满足这样的需求，数据中心需要在每个机架或每平方英尺安装更多的芯片。
Data centers are being planned, ordered and commissioned in record numbers. This is shown in statistics such as a 50% increase being forecast for the power footprint of data centers by 2025, as well as 500% growth in global data generated between 2019 and 2025, according to the Uptime Institute. Data centers are being built in unprecedented numbers and at far higher compute densities than ever before. Data centers need to pack more punch per rack or per square foot to keep up with demand.
Uptime Institute十多年前的数据显示，每个机架的平均功率密度在4-5kW。到2020年，每个机架的功率达到了8-10kW。但是请注意，有三分之二的美国数据中心调查表示，他们曾经收到过的峰值功率密度需求达16 - 20kW。2022年的最新需求，10%的数据中心的机架密度为20-29kW，7%的数据中心的机架密度为30-39kW，3%的数据中心的机架密度为40-49kW，5%的数据中心的机架密度为50kW或以上。显然，现代应用程序和数据量正在将机架密度推向前所未有的高度。
Uptime Institute numbers from just over a decade ago put the average power densities per rack in the 4–5 kW range. By 2020, that was up to 8–10 kW per rack. Note, though, that two-thirds of U.S. data centers surveyed said that they were already experiencing peak demands in the 16–20 kW per rack range. The latest numbers from 2022 show 10% of data centers reporting rack densities of 20–29 kW per rack, 7% at 30–39 kW per rack, 3% at 40–49 kW per rack and 5% at 50 kW or greater. Clearly, modern applications and data volumes are pushing rack density to unprecedented heights.
亚马逊(Amazon)、Facebook、谷歌(Google)和微软(Microsoft)等公司的超大规模数据中心在过去十年里一直是机架密度增长的领头羊。他们开创了数据中心更好地冷却和供电，同时在很小的占地面积内提供尽可能多的算力。有些已经进化出50kW甚至更高的机架密度。
Hyperscale data centers from the likes of Amazon, Facebook, Google and Microsoft have been among the leaders in density growth over the past decade. They have pioneered ways to better cool and power data centers while providing as much compute power as possible in a tiny footprint. Some have evolved racks of 50 kW and even higher.
然而，你会发现，超大规模数据中心的公司正在放弃超高机架密度。他们将该行业的机架密度提升到了一个新的水平，但现在更倾向于将机架功率密度稳定在30kW或者更低。因为这样的运行规模大、效率高，可以满足大多数用户的需求。超大规模数据中心在探寻最优解。他们想要一个高(但不是非常高)的机架密度，是可以规模复制，并且具备一个有吸引力的售价。
What you tend to find, however, is that the hyperscalers are now leaving the highest densities to others. They took the industry up to new levels of density but now prefer to flatten out at densities around the 30 kW range and slightly below. As they operate at massive scale and high efficiency, they can satisfy most user demands. What the hyperscalers want is optimization. They want a high (but not too high) rack density that is scalable and available at an attractive price point.
对于高性能计算和人工智能市场，正在出现的是迎合最高密度的专业提供商。这些公司正在从意想不到的来源获得业务。
For the HPC and AI markets, what is emerging are specialist providers that cater to the highest densities. These companies are gaining business from unexpected sources.
Oper8 Global首席执行官Mike Andrea表示:“虽然高性能计算曾经是企业和研究机构的专利，市场高达10亿美元，但现在小型企业正在利用它来获得竞争优势。”
“While HPC used to be the province of businesses and research entities with a billion-dollar turnover, much smaller businesses are now using it for competitive advantage,” said Mike Andrea, CEO of Oper8 Global.
他指出，从资金雄厚的大型公司到小型研发机构，以及航空航天、地震工程、3D建模、自动驾驶汽车仿真、各种人工智能用例、能源、石油和天然气生产、天气预报、分析、医疗保健和3D电影渲染等领域，高性能计算应用正在不断普及。高性能计算的需求仍在不断增加。
He noted the democratization of HPC applications from large, well-funded organizations down into smaller-scale research and development outfits, as well as into fields such as aerospace, seismic engineering, 3D modeling, autonomous vehicle simulation, various AI use cases, energy, oil and gas production, weather forecasting, analytics, healthcare, and 3D film rendering. The list of those making heavy usage of HPC continues to grow.
Mike Andrea表示：“时延仍然是高性能计算的主要驱动因素，此外，数据中心还能够支持超过100 kW /机架的机架密度。”Andrea的公司(Oper8)正在与多个客户合作，这些客户对机架密度的要求从80 kW到200 kW不等。
“Latency remains a primary driver in HPC, along with the ability of data centers to support rack densities in excess of 100 kW per rack,” said Andrea, whose company is working with several customers that are demanding anywhere from 80 kW to 200 kW per rack.
为了在这个市场上竞争，高密数据中心必须靠近客户以消除时延。因此，这些高性能计算的专业公司可能只会出现在某些区域，这些区域是聚集在一起的高密用户。
To compete in that market, highly dense data centers must be situated near to their customers to eliminate latency. Thus, these HPC specialists are likely to be found only in certain regions where there are clusters of customers making outrageous density demands.
另一个趋势是，数据中心通过只部署一两个高密度机架来迎合不断增长的高性能计算市场的一小部分。一些边缘和托管数据中心开始实现一个由2到12个机架组成的极高密度高性能计算方舱，或者一个由几个高性能计算机架组成的集群，以及更多中等密度的机架。这样的策略可以帮助数据中心满足一个或两个客户的需求，并建立他们的高性能计算的资源池，而不必大量资源投入到数据中心的重新设计。
Another trend is for data centers to cater to a small subset of the growing HPC market by deploying only a rack or two of high-density gear. Some edge and colocation data centers are beginning to implement one extreme-density HPC pod of two to 12 racks, or a cluster of several HPC racks alongside racks of more modest densities. Such a strategy helps data centers satisfy the requirements of one or two clients and build up their HPC arsenal without having to massively invest in a complete redesign of the entire data center.
尽管如此，即使是引入一两个机架，也需要大量的工作。除了新的服务器和配套设备外，还需要布线和其他改造。假定数据中心仍有足够的电力资源，并且高性能计算机架还需要更充分的冷却。因此，高密数据中心可能需要计算流体动力学(CFD)技术来提升冷却空气的气流组织，以避免局部热点。
That said, there is plenty of work involved in bringing in even a rack or two. As well as new servers and supporting gear, cabling and other improvements will be needed. It also presumes that sufficient additional power is on hand and that HPC racks can be served by enough cooling. Hence, high-density data centers will probably need to invest in computational fluid dynamics (CFD) technology to enhance air and cooling flows to avoid hot spots.
此外，它们需要先进的冷却技术，甚至可能需要某种液体冷却来保持新机架的低温。高温机架可能会导致电源分配单元(PDUs)的故障，因为机架背面为热空气。特别是机架密度大于35kW的情况。
Further, they require advanced cooling technologies and perhaps even liquid cooling of some kind to keep temperatures down in their new racks. Very hot racks can lead to failures in power distribution units (PDUs) due to the presence of so much rear-rack hot air. This is especially the case with racks more than 35 kW.
Dell Technologies的高性能计算和新业务工程高级总监Onur Celebioglu表示:“在空间有限的情况下，高功耗元件必须使用液冷。
“Where high-power consumption elements are packed into limited space, liquid cooling may become a necessity,” said Onur Celebioglu, senior director of engineering, HPC, and emerging workloads at Dell Technologies.
以水基为冷却媒介的方式，如冷冻水背板门，冷板液冷，浸没液冷等在高性能计算机架中变得越来越普遍。但请注意。液冷高性能计算应用可能需要更宽更深的机柜来容纳额外的电源和分歧管（manifolds）。成本，再加上对现有数据中心进行认真的重新配置，可能会导致一些人放弃高性能计算市场。
Water-based cooling options such as actively cooled rack doors, direct liquid cooling with cold plates, and immersion cooling are becoming commonplace in HPC racks. But be warned. Liquid-cooled HPC applications may necessitate wider and deeper cabinets to accommodate extra power feeds and fluid manifolds. The cost, coupled with the need for serious reconfiguration of an existing data center, may cause some to avoid the HPC market.
不久前，高密机架还只被认为是10kW左右。现在看起来微不足道。没有人知道高密机架能达到什么程度。但准备好迎接未来几年一些惊讶的机架密度吧。
Not so long ago, highly dense racks were thought be 10 kW or more. That looks positively miniscule in light of modern density numbers. No one knows how high density can go. But get ready for some outrageous figures in the coming years.
Mike Andrea表示:“如今，高密机架的功率约为40到125 kW，而极密机架的功率高达200 kW，甚至更高。”
“Today, high-density racks are roughly 40 kW to 125 kW and extreme-density racks are up to 200 kW and even beyond,” said Andrea.
翻译：
江秋健
广东浩云长盛网络股份有限公司 解决方案部高级经理
DKV（DeepKnowledge Volunteer）精英成员
校对：
王舜
秦淮数据 产品规划&研发总监
DKV（DeepKnowledge Volunteer）精英成员
公众号声明：
联系客服
