Some note I wanted to save for posterity on a datacenter migration project I lead several years ago:
What we did well:
- Everything was set into the racks correctly on the first try (pretty incredible if you think about it).
- Everyone arrived to the pre-meeting and customer sites on time (started at midnight).
- Communication with the customer was consistent.
- We got the backup filer up before 7am!
- We got the equipment all up and handed over by 4:30pm (our expected was 4pm and deadline was 7pm)
- We handled several mini-crises in stride:
- Motherboard death
- Loop “login delay” issue
- FC cable shortage
- Loop combination
What we need to remember next time:
- Starting at midnight and going all day is very, very different from starting at 8am and going all day.
- It’s important for a team to be familiar with one another and how the work tasks will flow.
- It’s important to have a good blend of experienced and inexperienced team members.
- Fly the team into town the day before – You can’t expect someone to travel all day and then work all night.
- Make sure the team understands how to read the documentation before you get onsite.
- Check to see if the team is experienced with the specific technologies relevant.
- Have a pre-job meeting (preferably with pizza) to explain expectations and game plan.
- Make sure everyone is in the same hotel, close to the jobsite.
- Make sure there is food/drink arriving regularly for the guys who are working.
- Add an extra guy to do physical work if you have someone supervising/project managing.
- Drills are important.
- Get all of the rails into the racks beforehand if possible.
- Plan to combine loops/stacks if the system is complex and spread out. Don’t underestimate how long it takes to cable per loop!
- Cabling loops between racks is significantly more time consuming.
- Don’t expect to be able to salvage any/all cables that run under/over racks. Was a total rat’s nest.
- Have a rested, standby team member to handle hardware failures/support issues
- Make sure you’re aware of the plan for switches/patch panels that are in-rack, and for disconnecting the PDU’s.
