Guide to CERN Computing Center
CERN Computing Center


Introduction

CERN Computing Center The CERN Computing Center is located in Building 513 since year 1972. Some facts:
  • Dimensions: Main room 1400m2; Basement 1200m2
  • Power Capacity:
    • Nominal current capacity: 2.5 MW (240 kW of critical power)
    • Extended usable capacity: 2.9 MW (600 kW of critical power)
    • As of 2013 increased to 3.5MW (450kW of critical power) (Read more and more about this upgrade)
Despite the several upgrades performed throughout the years and the ongoing optimisations of the current facility, it became clear that in order to face future computing demands, another building would be needed. Different possibilities were considered, like building a new facility in Prevessin, and finally in 2011, CERN launched a formal Call for Tender to find a place that would host extra CERN equipment. The Wigner Research Center for Physics in Budapest was eventually chosen and it will operate as an extension to the CERN Data Centre as of 2013. The Wigner Research Center will be adding 20.000 cores and 5.5 PB of disk data, doubling this after 3 years. CERN has signed a contract for 3 years that could be extended up to 7 years. CERN is the owner of the equipment installed at the Wigner Research facilities.

VIP Itinerary

VIPs and Press visits are allowed to go inside the computing center if IT explicitly confirms it. This is the recommended itinerary on the ground floor (left) and basement (right):

Map for VIP visits: ground floor Map for VIP visits: basement
Entrance to the exhibition area VIP visits in building 513 could be organised as follows:
  • The balcony on the 1st floor with a view of the main room, where you could talk about:
    • Processing power and disk capacity
    • CIXP
    • Openlab Project
  • The exhibition area on the ground floor, where you could talk about:
    • Operators room
    • On-line WLCG Monitor and Grid Computing
    • Exhibition about the history of computing at CERN
  • The ground floor inside the computing center, where you could talk about:
    • Cooling system
    • CERN network
    • Emergency power
  • The basement, where you could talk about:
    • Tape capacity
    • Water cooled racks

General public visits Itinerary

View from the gallery General public visits should be restricted to the following areas:
  • The balcony on the 1st floor with a view of the main room (19 people maximum for security reasons)
  • The exhibition area on the ground floor

History of Computing at CERN

Building 513 in the 70s Some pictures about the evolution of computing at CERN throughout the years can be found on the corridor that goes from building 31 to building 513. Visitors can see the evolution from the first 'human computer' to the grid.

The exhibiton area shows punch cards, modems and hard disks used 50 years ago! It also contains the computer used by Tim Berners Lee to invent the web.

Some interesting links about the history of computing at CERN:

Operators room

Operators room There is always someone working in the Operators room. 24 hours a day, 365 days a year. It's normally just one person. There are documented procedures to operate the different IT services. Critical IT services are also covered by an on-call service managed by experts and contacted by the operator when necessary. The computing center is managed mostly by automated and semi-automated software (anything else would be impossible on this scale!)

IT Services for CERN

Some of the IT services that are run at the CERN computing center:

Openlab

CERN openlab summer students in the computer centre CERN openlab is a framework for evaluating and integrating cutting-edge IT technologies or services in partnership with industry, focusing on future versions of WLCG. Through close collaboration with leading industrial partners, CERN acquires early access to technology that is still years from general computing market. In return, CERN offers expertise and highly demanding computing environment for pushing new technologies to their limits and provides a neutral ground for carrying out advanced R&D with various partners.

This initiative has been organised into successive three-year phases. In openlab-I (2003-2005) the focus was on the development of an advanced prototype called opencluster, and openlab-II (2006-2008) addressed a range of domains from platforms, databases and interoperability to security and networking. We are now in the third phase openlab-III (2009-2011) which is not only capitalizing on and extending the successful work carried out in openlab-II, but is also tackling new crucial areas.
There are currently four industrial partners: Intel, Oracle, HP and Siemens. The technical activities are organised in different domains of competence:

To know more, please check the Openlab web.

Internet Exchange Point

CIXP CERN's primary mission is to provide facilities for high energy particle physics experiments. CERN is open to scientists from its 20 member states and from all other countries of the world. This makes CERN one of the largest sources of numerical scientific data in the world. Computer networking, and in particular Internet connectivity, is therefore a mission-critical requirement. CERN operates an IXP in order to facilitate the exchange of Internet traffic in the region and to maximize its own Internet connectivity.

An Internet exchange point is a physical infrastructure through which Internet service providers exchange internet traffic between their networks.

CERN IXP provides peering between Internet Service Providers and Telecom Operators like Sunrise or France Telecom on a commercial basis.
For more information, please check the CIXP pages.

Computing Power

Machines in the computer center Some numbers about the computing power at the CERN computing center (July 2013):
  • Number of machines: 17,000 processors with 85,000 cores (Source)
  • All physics computing is done using the Linux operating system and commodity PC hardware. There are few Solaris server machines as well, especially for databases (Oracle).

The LHC grid

LHC data is stored and analized using the computing resources of the WLCG grid infrastructure (Worldwide LHC grid). WLCG is a worldwide collaboration of sites belonging to two major grid infrastructures: the European Grid Initiative (EGI) and the Open Science Grid (OSG) in US. WLCG sites are organised in a Tier model. Tier 0 is the CERN computing center where the data from the 4 LHC experiments is at first stored. Then there are 11 Tier-1 sites where LHC data is also replicated on tape. Finally, there are more than 100 Tier 2 sites providing extra computing resources. LHC data is then processed using the computing power provided by all these sites.

Check the WLCG web for more details on the project.

For general information about the Grid, please visit The Grid cafe.

WLCG activity Some useful numbers for WLCG:
  • Total number of EGI + OSG sites: 500 (157 WLCG sites)
  • Average number of WLCG jobs executed every month: 40M
For real time information, you can check:
  • The WLCG Google Earth Dashboard that it is also installed at the entrance of the computer center. It shows real time data transfers and jobs being executed in the WLCG infrastructure. It also contains detailed information about the different WLCG sites when you click on a particular site. The dashboard can be installed in any computer, check the User Guide for more details.

Cooling

Air cooled racks Air conditioning is a major problem for data centres everywhere in the world at the moment. As processors get faster they also get hotter and at the moment we are getting a greater increase in heat than in performance. Rack machines are even worse as they are densely packed with processors.

Some of the racks at the computing center contain only a few machines in them since there's not enough cooling capacity now to fill them with more machines. The room was designed with one supercomputer in a corner in mind, not several thousand processors!

It's interesting to mention how the racks are placed. They use a Hot/Cold aisle configuration: the front of the racks are facing each other on the 'cold' aisle and expel heat out in their backs to the 'hot' aisle. The doors and roofs placed in the cold aisles increase efficiency by preventing warm air from mixing unhelpfully with the cold air. The cold air comes out from the floor inside the 'cold' aisle. The cold air is introduced in the building through the big blue pipes coming from the roof and going down to the floor. 3 chillers are responsible for cooling down the air. This process consumes no energy during the winter months where cold air is directly taken from outside.
Water cooled racks In the last upgrade works of the computing center, a new special IT room has been built on the back of the main room on the ground floor. This special room hosts critical IT equipment. This room uses water-cooled racks with passive heat exchangers (they don't consume extra power). These racks are more efficient and are able to cool down 10kW of equipment, compared to 4kW equipment that can be cooled with the hot/cold aisle racks. Water-cooled racks are also more expensive and it is more complicated to install them since water pipes are needed. In the basement, water pipes are easily accesible and water-cooled racks have also been installed, these ones can be shown during the visit.

Check the following interesting articles and presentations:
  • Article by the Google Vice President of Operations in the CERN courrier, about power consumption in computer centers.
  • Article giving an overview of CERN's approach to energy efficient computing.
  • Presentation explaining the cooling approach of the computer center.

Emergency power

In the right back part of the main room is the 'critical area', backed by diesel capacity. Everything else has UPS (Uninterruptible power supply). This is an electrical apparatus that provides emergency power when the main input poweer source fails, but only for a few minutes, which is enough to switch between the French and the Swiss power in case of problems. Some (but not all) of the chillers are backed by UPS and diesel as well.

Networking

Network switch The CERN network is made of several networks:
  • General Purpose network: offers internet connectivity protected by the CERN firewall. Open to everybody. Desktop computing and central services (mail, web, etc). Fiber optic. 8 main star-points. Each building is connected by fiber to the closest starpoint. One star-point per building. Twisted pair cables used within the buildings.
  • Technical network: offers connectivity for industrial systems and accelerator control devices. No connection from outise. Restricted to special users. Fiber optic.
  • Experiments network: 10 GE dedicated links to the computing center. Optic fiber.
  • WLCG network: 10 GE dedicated links to the Tier-1s using National Research Networks and Geant. Optic fiber.
  • Connection to Wigner in Budapest
  • : two connections 100 GE. Optic fiber.
Check this link to see a diagram explaining the network connections between CERN and the Tier-1 centres.

Data Storage

IBM tape Some numbers about storage capacity at the CERN computing center (July 2013):
  • Tape: Reaching 100PB of data, including non LHC data. (Source)
  • Disk: 120PB. This number includes system disks as well as storage disks. Moreover, RAID is used to increase redundancy and stability, so the amount of disk space actually available is less than this number. (Source)
The LHC has generated 20PB of data in 2011 and 30PB of data in 2012. This is all stored on tape together with analysis data, which means that more data will be generated also during the LHC shutdown . A copy of the data is always available at CERN and it will be kept during the whole lifetime of the LHC. Replicas of part of the data are copied among the 11 Tier-1 sites.

There are three different tape vendors in the computing center. Some facts about two of them:

  • Linear IBM 3584 libraries: There are 3 in building 513 and 1 in building 613. They have 2 robotic accessors going up and down.
  • Sun SL8500 interconnected 'u-bend track': There are 2 in building 513 and 2 in building 613. They have 4 robotic accessors per u-track
Storagetek tape Tape is a much more lasting and durable (30 years) device than disk and RAM memory. It also involves much less power consumption, and therefore cooling. It's also cheaper than disk per GB. CERN will use tapes for 5 years before upgrading to newer, higher capacity tapes.

Access tape data is very slow, about 1-3 minutes. So latest data is made available for physicists in disk servers where they can get it immediately.

What happens when we run out of tapes and LHC keeps on generating data?

When a new type of higher capacity tape media is introduced, or a new drive is purchased we can copy all the data from the 'obsolete' media to the new higher capacity media, and avoid buying extra robotic library space for a while. Or we can copy all the data from the media that can be re-written at a higher density by the new drive, rewrite it, and avoid buying more new media for a while. However, this can take a year to move data onto new tapes and a lot of work.....

Other Material

Acknowledgements

Special thanks to: