Planning the Upgrade to “50 kHz”

Introduction

We will install in 2008 8 slices of 72/80 RUs each and with 3 GbE? rails from the RUs and up to 1200 Total BUFUs. The GbE? Switches will be 8 as in the final configuration, each equipped with 6 linecards to start. Each Slice will serve 4 or 5 BUFU racks, the number of BUFUs per rack will be decided after the selection of the PCs, depending on the power consumption.
We will only cable the RU racks and the 32 or 40 BUFU racks, the rest of the racks will be installed and cabled later (the racks are not physically installed for now.)

In addition the Storage Manager rack will be cabled to each of the 8 switches. The HLT validation farm rack as well as the new Heavy Ions HLT rack will be cabled to 2 of the 8 slices.

Everything can be installed and cabled as soon as the relevant material will be procured. Probably the last item that will arrive will be the BUFU PCs.

Urgent Things

• Decide the lengths and the number of needed Ethernet patch cables
• Order as soon as possible the patch cords for the racks
• Test the FU PCs and prepare the tender (who does it?)

Procurement:

• (JAN) Switches and line cards (ORDERED already) (Received January 8 2008)
• Check Force 10 order
• Check Cable Management Kits (+1) and prepare supports for mounting them
• (MAR-JUN) Blue Cables (Requesting offers)
• Offer Requested for now to Israel (Fibernet)
• Check if we want to add a few 25m Cables
• Check when cables are expected (beg. March?)
• (JAN-FEB) Patch Cables (To be decided and ordered) (Ordered end of December)
• Two per PC (control + data) +3rd rail for A, B, E and F5-8 rows
• Power cords should be ready (CHECK)
• Power sequencers (CHECK)
• How many per rack? Four?
• Check if we have enough and procure if necessary
• HP Switches
• There should be enough, make sure
• Install them in the racks as soon as convenient
• Switch installation
• Double the power on the switch racks; Started I think
• Prepare the cables for the switch rectifiers; Started too I think
• Cable labels - Only needed for the blue cables
• Make them standard or get a label maker to be free to write what we like
• Rack vertical supports to install patch panels
• Get more: Attila (requested 60 + ~300 screws January 9 2008)
• Get more screws for them
• Replace the cannibalized ones
• (MAR-AUG) BUFU PCs
• Test them, including power consumption at full CPU usage
• Make the tender
• Select them
• Wait for the PCs
• When do we expect them (???)
• Should probably get everything ready before Installing the PCs, perhaps including the rack cabling
• Get enough cable ties and cable labels for the patch cords (some labelling already provided by the suppliers)

Pre-Installation

• Get vertical rack bars to install patch panels
• Cut them
• Prepare the support plates (6) for the Force 10 Cable Management Kits
• Double the power on the switch racks. Started I guess
• Prepare the cables for the switch rectifiers
• Can already prepare the racks with HP switches at least
• Check alternative to VLANs
• Test as soon as possible new BUFU PCs and decide number of PC per rack
• Must start tests in December (Not started yet)
• Organize moving 10 racks + cooling vents in case additional racks will be needed
• Prepare the detailed plan of cabling in the racks and to the Force 10 switches
• Consider who will be available for this installation from UCSD (students as well?), from the CERN DAQ group and between other people in the DAQ group

Installation

• Install HP switches
• Install 6 new FORCE 10 Switches (Decide position of line cards) (1.5 weeks; 2 people)
• Send back 2 broken line-cards to be fixed (could be done now)
• Add patch panels to SM Rack (0 weeks)
• Prepare all BUFU 4x8=32 Racks (1.5 weeks -2 people)
• HP switch
• Power sequencers
• Patch panel (1 per rack)

AFTER RECEIVING RJ21 CABLES

• Cable Storage Manager Rack to switches (before or after the others?) (1 day 2 people)
• Put the blue cables for 5x8=40 racks (3 weeks 2 people)
• First E and F rows (Slices 1-4)
• Move the cables of RUs E and F (1 week 2 people)
• Add cables for the 3rd rail RUs E and F (1 week 2 people)
• Cable inside the racks the 3rd rail for RUs E and F (1 week 2 people) (CAN BE DONE AS SOON AS WE HAVE THE PATCH CABLES)
• Cable 4 switches slices 1-4 (1 week 2 people)
• Get Slices 1-4 to work (1-4 weeks)
• Then A and B rows (Slices 5-8)
• Move the cables of RUs A and B (1 Week 2 people)
• Add cables for the 3rd rail RUs A and B (1 Week 2 people)
• Cable inside the racks the 3rd rail for RUs A and B (1 week 2 people) (CAN BE DONE AS SOON AS WE HAVE THE PATCH CABLES)
• Cable 4 switches slices 5-8 (1 weeks 2 people)
• Get Slices 5-8 to work (1-2 weeks)
• TOTAL TIME Sequential 13-17 Weeks. People needed 2
• USING 4 PEOPLE SHOULD TAKE ~6-10 WEEKS
• Possibly cable the additional 10 racks when moved
• Heavy Ions HLT + FU validation farm: add patch panels, maybe not counted but we have enough spares

AFTER RECEIVING BUFU

• Install BUFU PCs in the racks (1 Week 4 people) (probably done by the vendors)
• Cable the BUFU PCs = power cords + 2 Ethernet cables (4 Weeks 3 people) can be done in advance
• Check BUFU PCs – bios - install OS – fix broken PCs (4+ ??? Weeks 2 people)

Commissioning

Continue with the current setup: 2 slices with roughly the right number of BUFUs

• Atcp: add some monitoring and a debug hyperdaq page
• Fixed the network bug in the configurator, use it now
• Verify again the performance
• Prepare all the scripts to check all what is possible to check.
• Test the multiple slices
• Implement the quasi-real time usage of hot spares
• Check well the large configurations with CMSSW FUs and SM
• Decide if we need a switch in the SM Rack or not

After re-cabling of the first 4 slices we will have 4 32x48 slices plus one of the current slices that will be taken down after a short while

• Just verify the networking and the cabling, scripts can be ready in advance
• Verify usage of Storage Managers
• Also multi slice operation (up to 4 or 5)

After re-cabling all 8 new slices we will have 8 32x48 slices

• Just verify the networking and the cabling, scripts can be ready in advance
• Also multi slice operation (up to 8)
• Also test 16 Slices of 40 RUs (maybe use folded at the beginning)

After installing the final BUFUs (~1200) check the final performance of the system

-- MarcoPieri - 9 January 2008