Data center en silo: avantages de la verticalité

Datacentre en Silo:avantages de la verticalité

Marc Parizeau, Ph.D.

1Montréal, 16 avril 2012

circa 1965

accélérateur Van de Graaff

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Construction en 1962 du pavillon Vachon et de son accélérateur de particules

2

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012 • Produit un potentiel

électrostatique grâce au frottement d’une courroie

• À l’époque (1965), une infrastructure de 20 millions de dollars!

• Toujours fonctionnelle en 2006, mais totalement désuète

Accélérateur Van de Graaff

3

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Centre de recherche nucléaire(en sous-sol)

4

salle des commandes

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

5

PDP-15 (circa 1970)

Compute room

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

6

salle des cibles(3000pi2)

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

7

Compute Canada — Calcul CanadaA proposal to the

Canada Foundation for Innovation – National Platforms Fund

Hugh Couchman (McMaster University, SHARCNET)Robert Deupree (Saint Mary’s University, ACEnet)Ken Edgecombe (Queen’s University, HPCVL)Wagdi Habashi (McGill University, CLUMEQ)Richard Peltier (University of Toronto, SciNet)Jonathan Schae�er (University of Alberta, WestGrid)David Senechal (Universite de Sherbrooke, RQCHP)

Executive Summary

The Compute/Calcul Canada (CC) initiative unites the academic high-performance comput-ing (HPC) organizations in Canada. The seven regional HPC consortia in Canada —ACEnet,CLUMEQ, RQCHP, HPCVL, SciNet, SHARCNET and WestGrid— represent over 50 institutionsand over one thousand university faculty members doing computationally-based research. TheCompute Canada initiative is a coherent and comprehensive proposal to build a shared distributedHPC infrastructure across Canada to best meet the needs of the research community and en-able leading-edge world-competitive research. This proposal is requesting an investment of 60 M$from CFI (150 M$ with matching money) to put the necessary infrastructure in place for fourof the consortia for the 2007-2010 period. It is also requesting operating funds from Canada’sresearch councils, for all seven consortia. Compute Canada has developed a consensus on nationalgovernance, resource planning, and resource sharing models, allowing for e�ective usage and man-agement of the proposed facilities. Compute Canada represents a major step forward in movingfrom a regional to a national HPC collaboration. Our vision is the result of extensive consultationswith the Canadian research community.

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

En 2006, la Fondation canadienne pour l’innovation (FCI) annonce le financement d’une plateforme nationale pour le Calcul de Haute performance (CHP)

8

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Plateforme nationale•Création de Calcul Canada pour chapeauter

les 7 consortiums canadiens de CHP

• 150 millions à travers le Canada✓ dont 56 millions au Québec

✓ 37.5 millions pour le CLUMEQ

✓ et 12.5 millions à l’Université Laval

•CLUMEQ = McGill + Laval + réseau UQ✓ fait aujourd’hui partie de Calcul Québec

✓ qui rassemble toutes les universités québécoises

9

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Projet•Bâtir à l’Université Laval un centre pour le

calcul scientifique de haute performance✓ superordinateur de classe mondiale

✓ datacentre pour superordinateur

✓ équipe de support

• Étapes✓ début du design : janvier 2007

✓ début des travaux : septembre 2008

✓ arrivée du superordinateur : juin 2009

10

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Plan

•Design de Colosse

•Avantages de la verticalité

•Siloctet

11

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

17 octobre, 2006

12

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012 • Principaux avantages

✓ pas de coins

✓ moins de turbulences

✓ une seule allée froide / chaude

✓ espace au sol réduit

✓ câbles plus courts

•Mais comment faire?✓ refroidissement?

✓ accès physiques?

✓ etc.

Concept cylindrique

13

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Après 12 concepts et 8 mois de travaux assez intenses...

14

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

15

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Système de free air

cooling

système de refroidis-sement principal

ventilateurs

serpentins

cabinet

16

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

plénum air froid(32 m2)

plénun air chaud(25 m2)

17

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

18

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Spécifications• Nombre de cabinets: 56 max

• Puissance électrique: 1.1 MW✓ Aucune contrainte sur la densité d’énergie par cabinet

✓ capacité moyenne: 20 kW/cabinet

• Refroidissement des cabinets: 100% air✓ fonctionne avec des cabinets de plus de 30 kW

✓ vélocité d’air maximale: 2.4 m/s

✓ Asservissement de la pression différentielle entre l’allée froide et l’allée chaude

• Réutilisation de la chaleur pour chauffer le campus

• Capacité de charge des planchers: 940 lb/pi2

19

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

« InfoWorld Green 15 Award »CLUMEQ / Université Laval

20

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Superordinateur Colosse•Sun constellation system✓ 10 fully loaded Sun Blade 6048, with X6275 modules

(double Nehalem EP blade, 2.8GHz, 24GB of RAM)

✓ full-bisection IB-QDR interconnect (2xM9 switches)

✓ 1 PB of Lustre storage in a high availability configuration, using 2 MDS and 9x2 OSS

✓ Sun J4400 storage arrays

✓ >30 kilowatts/cabinet

• 77 Tflops sustained✓ < 400 kW

21

Classement basé sur le testHigh Performance Linpack (HPL)

teraflopsmesurés

teraflopsthéoriques

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

22

Nov. 2009

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Plan

•Design de Colosse✓ silo de béton

✓ 36 pieds de diamètre

✓ 65 pieds de hauteur

•Avantages de la verticalité ✓ versus autres approches

•Siloctet

23

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

APC Schneider Electricwhite paper #130

The Advantages of Row and Rack-oriented Cooling Architectures for Data Centers

24


Schneider Electric – Data Center Science Center White Paper 130 Rev 1 3

rows or groups, and with the rack level architecture CRAC units are assigned to the individual racks.

Room Row Rack

A summary of the basic operating principles of each method are provided in the following sections: Room-oriented architecture In room-oriented architecture, the CRAC units are associated with the room and operate concurrently to address the total heat load of the room. A room-oriented architecture may consist of one or more air conditioners supplying cool air completely unrestricted by ducts, dampers, vents, etc. or the supply and/or return may be partially constrained by a raised floor system or overhead return plenum. For more information see White Paper 55, Air Distribu-tion Architecture Options for Mission Critical Facilities. During design, the attention paid to the airflow typically varies greatly. For smaller rooms, racks are sometimes placed in an unplanned arrangement, with no specific planned con-straints to the airflow. For larger more sophisticated installations, raised floors may be used to distribute air into well-planned hot-aisle / cold aisle layouts for the express purpose of directing and aligning the airflow with the IT cabinets. The room-oriented design is heavily affected by the unique constraints of the room, including the ceiling height, the room shape, obstructions above and under the floor, rack layout, CRAC location, the distribution of power among the IT loads, etc. The result is that perfor-mance prediction and performance uniformity are poor, particularly as power density is increased. Therefore, complex computer simulations called computational fluid dynamics (CFD) may be required to help understand the design performance of specific installations. Furthermore, alterations such as IT equipment moves, ads, and changes may invalidate the performance model and require further analysis and/or testing. In particular, the assurance of CRAC redundancy becomes a very complicated analysis that is difficult to validate. Another significant shortcoming of room-oriented architecture is that in many cases the full rated capacity of the CRAC cannot be utilized. This condition is a result of room design and occurs when a significant fraction of the air distribution pathways from the CRAC units bypass the IT loads and return directly to the CRAC. This bypass air represents CRAC airflow that is not assisting with cooling of the loads; in essence a decrease in overall cooling

Figure 1 Floor plans showing the basic concept of room, row, and rack-oriented cooling architecture. Blue arrows indicate the relation of the primary cooling supply paths to the room.

Air Distribution Architecture Options for Mission Critical Facilities

Related resource White Paper 55

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Principales approches

2D configurations

25

APC Schneider Electric white paper #130, Figure 1

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

3D vertical rows

•Critères d’analyse✓ agilité / flexibilité

✓ fiabilité

✓ coûts (TCO)

✓ maintenance

•Conclusion✓ tous les avantages de

l’approche en rangée

✓ plus grande densité

✓ mise à l’échelle

✓ distances plus courtes

26

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Avantages de la géométrie verticale

Maintenant, si la même quantité de serveurs est installés dans un espace en trois dimensions...

• 46 m x 46 m x 46 m = 100 000 m3

• Distance «d» au serveur le plus loin = 55 m (86% moins de distance à parcourir)• Longueurs de câbles informatiques et électriques réduites

46m

46m

services46m

d

• 316 m x 316 m x 1 m = 100 000 m3

• Distance «d» au serveur le plus loin : 353 m• Longues distances de câbles informatiques, électriques et de conduits de ventilation requises

316m

316m

services

d

27

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

✓ plénum = faible vélocité d’air

✓ asservissement de la pression différentielle

5 kW - 35 kW

plénum froid

plénum chaud

cabinet(productionde chaleur)

28



Agility challenges Data center users have identified the agility challenges shown in Table 1 as critical cooling-related issues. The effectiveness of the different architectures in addressing these chal-lenges is summarized as well.

Agility challenges Challenge Rack Row Room

Plan for a power density that is increasing and unpredictable

Modular; deployable at rack level increments targeted at specific density

Modular; deployable at row level increments targeted at specific density

Complex to upgrade or adapt; typically built out in advance of requirement

Reduce the extensive engineering required for custom installations

Immune to room effects; rack layout may be completely arbitrary

Immune to room effects when rows laid out according to standard designs; configure with simple tools

Complex CFD analysis required which is different for every room

Adapt to ever-changing requirements or any power density

Rack cooling capacity that is not used cannot be used by other racks

Cooling capacity is well defined and can be shared across a group of racks

Any change may result in overheating; complex analysis required to assure redundancy and density are achieved

Allow for cooling capacity to be added to an existing operating space

New loads may be added that are completely isolated from the existing cooling system; limited to rack cooling capacity

New loads may be added that are com-pletely isolated from the existing cooling system; each additional cooling system increases density for entire row

May require shutdown of existing cooling system; requires extensive engi-neering

Provide a highly flexible cooling deployment with minimal reconfiguration

Racks may need to be retrofit or IT equipment moved to accommodate new architecture

Requires the rack rows to be spaced to accom-modate or changes to overhead infrastructure for new architecture

Floor tiles can be reconfi-gured quickly to change cooling distribution pattern for power densities <3 kW

Table 1 Effectiveness of the room, row, and rack-oriented cooling architectures in addressing agility challenges. Best performance highlighted.

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

APC Schneider Electric white paper #130, Table 1

29



Availability challenges Data center users have identified the availability challenges shown in Table 2 as critical cooling-related issues. The effectiveness of the different architectures in addressing these challenges is summarized as well.

Availability challenges Challenge Rack Row Room

Eliminate hot spots Closely couples heat removal with the heat generation to eliminate mixing

The airflow is complete-ly contained in the rack Closely couples heat removal with the heat generation to minimize mixing

Supply and return paths promote mixing; engi-neered ductwork required to separate air streams

Assure redundancy when required

2N cooling capacity required for each rack; many rack cooling systems are not redun-dant capable

Utilizes shared N+1 capacity across com-mon air return

Complex CFD analysis required to model failure modes; requires localized redundancy

Eliminate vertical temperature gradients at the face of the rack

Heat captured at the rear of the rack before mixing with cold supply air

Heat captured at the rear of the rack before mixing with cold supply air

Warm air may recirculate to front of rack as a result of insufficient heat removal or supply

Minimize the possibili-ty of liquid leaks in the mission critical instal-lation

Operates at warmer return temperatures to reduce or eliminate moisture removal and make-up sources. Rack targeted cooling requires additional piping and leakage points

Operates at warmer return temperatures to reduce or eliminate moisture removal and make-up sources

Mixed air return promotes the production of conden-sate and increases require-ment for humidification

Minimize human error Standardized solutions are well documented and can be operated by any user

Standardized solutions are well documented and can be operated by any user

Uniquely engineered system requires a highly trained and specialized operator

Table 2 Effectiveness of the room, row, and rack-oriented cooling architectures in addressing availability challenges. Best performance highlighted.

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012


30



Lifecycles cost challenges Data center users have identified the lifecycle cost challenges shown in Table 3 as high priority cooling-related issues. The effectiveness of the different architectures in addressing these challenges is summarized as well.

Lifecycle cost challenges Challenge Rack Row Room

Optimize capital investment and available space

Dedicated system for each rack may result in oversizing and wasted capacity

Ability to match the cooling requirements to a much higher percen-tage of installed capaci-ty

System performance is difficult to predict, resulting in frequent oversizing

Accelerate speed of deployment

Pre-engineered system that eliminates or reduc-es planning and engineer-ing

Pre-engineered system that eliminates or reduces planning and engineering

Requires unique engineer-ing that may exceed the organizational demand

Lower the cost of service contracts

Standardized compo-nents reduce service time and facilitate the ability for user serviceability. Likely higher number of units with 1:1 ratio to IT racks enclosures.

Standardized compo-nents reduce service time and facilitates the ability for user servi-ceability

Specialized service con-tracts required for custom components

Quantify the return on investment for cooling system improvements

Standardized compo-nents for accurate measurement of system performance

Standardized compo-nents for accurate measurement of system performance

Customer engineered solutions makes system performance difficult to predict

Maximize the operational efficiency by matching capacity to load

Cooling system will likely be oversized and full potential not realized.

Right-sized cooling capacity to the cooling load matching heat load to installed capacity

Air delivery dictates oversized capacity; pres-sure requirements for under floor delivery are a function of the room size and floor depth.

Table 3 Effectiveness of the room, row, and rack-oriented cooling architectures in addressing lifecycle cost challenges. Best performance highlighted.

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012


31



Serviceability challenges Data center users have identified the serviceability challenges shown in Table 4 as high priority cooling-related issues. The effectiveness of the different architectures in addressing these challenges is summarized as well.

Serviceability challenges Challenge Rack Row Room

Decrease Mean-Time-To-Recover (includes repair time plus technician arrival, diagnosis, and parts arrival times)

Modular components reduces downtime; 2N redundancy required for system repair and maintenance

Modular components reduces downtime; N+1 or excess capacity allows for repair without interruption to system performance

Custom spare parts are not readily available and require trained technician extending recovery time

Simplify the complexity of the system

Standardized compo-nents reduce the technic-al expertise required for routine service and maintenance

Standardized compo-nents reduce the technical expertise required for routine service and mainten-ance

Operation and repair of the system requires trained experts.

Simpler service procedures

In-house staff can perform routine service procedures. Modular subsystems with interfaces that mistake-proof service procedures.

In-house staff can perform routine service procedures. Modular subsystems with interfaces that mistake-proof service procedures.

Routine service procedures require disassembly of unrelated subsystems. Some service items are not easy to access when the system is installed. Highly experienced personnel are required for many service procedures.

Minimize vendor interfaces

Modular units designed to integrate with a small set of ancillary systems

Modular units designed to integrate with a small set of ancillary systems

Engineered solution with multi-vendor subsystems

Learn from past problems and share learning across systems

Standardized building block approach with single rack and cooling unit interaction maximiz-es learning

Standardized building block approach with low interactions increases learning but with fewer systems to learn from

Unique floor layouts all have unique problems, limiting learning

Table 4 Effectiveness of the room, row, and rack-oriented cooling architectures in addressing serviceability challenges. Best performance highlighted.

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012


32

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Plan•Design de Colosse✓ silo de béton

✓ 36 pieds de diamètre

✓ 65 pieds de hauteur

•Avantages de la verticalité ✓ versus autres approches

•Siloctet✓ Rehab

✓ prefab

33

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

34

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

35

La rencontre du silo et de l’octet...

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012 S i l o # 5 Montréal . . .

« Quossé dont qu’on fait avec ça ? »

36

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Fiche technique :• 1,2 mégawatt par silo• 2 688 serveurs par silo• 446 watts par serveur

Projet global :• 52,8 mégawatts charge dédiée aux serveurs• 118 272 serveurs 1U

Objectif : T ier I I I

37

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

38

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

39

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

VERT.COM, inc.

innove avec . . .

pré-ingéniéré, standardisé et préfabriqué, a!n

de données dans un environnement vertical et modulaire

enceinte de refroidissement intelligente avec une technologie hybride simple et !able.

Le tout complété par SiloSmartTM, un système de gestion spéci!que permettant de mesurer, de contrôler

optimiser l’énergie consommée.

Le data center éco-conçu en silo ®

* Des

ign

simpl

i!é.

40

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Montréal 514 788.8975Canada & U.S.A. 1 877 753.8975 Europe 00 800 55.66.33.33

Courriel [email protected]

facebook.com/siloctet

twitter: @siloctet

www.blogue.vert.com

www.vert.com

conception | design

Astr

olite

PC

100%

.

Vert.com conçoit le

“Frigidaired’ Internet”

41

Une con!guration modulaire et un déploiement progressif

SiloSmart®Une configuration modulaire et un déploie-

ment progressif à grande échelle

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Une enceinte de refroidissement hybride et intelligente

42

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

Conclusion

• La verticalité permet de✓ minimiser l’emprise au sol

✓ maximiser l’efficacité énergétique

✓ réduire les distances

✓ modulariser le déploiement progressif d’un datacentre à grande échelle

✓ réduire les coûts d’opération et d’exploitation

43

Mar

c Pa

rize

au, D

atac

entr

e en

Silo

: ava

ntag

es d

e la

ver

tical

ité, 1

6 av

ril 2

012

ATTENTION

calculs

intenses

Personnel

autorisé

seulement

Questions?

électroaimant de l’ancien accélérateur de particules

44

Education

Data center en silo: avantages de la verticalité