I moved back to Colorado and took over two call center sites in September. Just as a little background I work for a outsourcing company that manages call centers for other companies (amongst many other things). It provides some IT challenges since we use the customers software on our machines alongside our internal software to run the business.
I received notice a few months after I took over these sites that our client wanted all of our desktop machines upgraded from Windows XP to Windows 7 by “April 15th” (we will get to the quotation marks later). Half way kidding I told my boss we would get it done in 60 days. I thought the 90 days that we were given was more than enough time even running as lean as we do, with two desktop technicians supporting a site of over 1000 desktops. I had built XP images and rolled it out to 800 machines in less than 60 days. 90 days for Windows 7 should be a piece of cake. I was in for a big surprise.
First 30 Days
The first 30 days consisted of upgrading the 3 domain controllers from Windows Server 2003 R2, to Windows Server 2008 R2. We also built a new WDS server since our old WDS server was also still running Windows Server 2003 R2 and built the base image and did internal testing. We built new group policies and WMI filters to make sure these new policies only affected the new machines. Everything was smooth sailing. I sent my boss a status email outlining our plan of completion by “April 15th”
Week 5 and 6
Over the next week we rolled it out to about 5% of the first two business unit groups (about 30 desks) and made a few small tweaks to the GPO’s and the image. The next week we rolled it out to the remaining machines in those two groups, and like that we were sitting at over 60% of our machines completed. Proud of our status and since we met the goal I set for the first two business groups I sent an update to my boss reassuring him we were on track for the “April 15th” deadline.
Week 7 and 8
After completing the first two business groups, we had to slightly modify the group policies for the next two business groups. Luckily we didn’t have to modify the image at all so it went pretty quickly. We took the two weeks we had slated for these last two groups (around 100 machines) and made sure it was done right. We also pushed it to another 140 training machines. Again I sent my boss an email with our status and reassuring him of the “April 15th” deadline. This time I got an email back that we needed to get it done by April 7th. When I asked my boss why it changed he told me the date had always been April 7th, and he wondered why I kept forecasting completion for the 15th. I am not sure where I got the 15th from or why he didn’t mention it before now but hey, 1 week won’t make THAT much different, right?
The final 3 weeks
The final 3 weeks we had the least amount of machines to deploy but we had to modify the base image with additional software and when we rolled it out we also had to use Symantec PGP WDE (I hate this software) to fully encrypt the station. This adds about 3.5 hours to each machine setup time. We got the image modified in week 9, rolled it out to a test group half way through week 10 and then finished rolling it out to the support team, supervisors and managers in week 11 and finished with 12 hours to spare on April 7th. Everything was good (so we thought).
The Aftermath
It has only been a few days since we completed. After the roll out we received some strange reports from everything from corrupt OST files, corrupt Office installs to programs stating they were not valid Windows applications. None of which that were reported in our extremely short 5 day “soak period”. It appears the problems were caused by how we encrypted the stations and PGP wasn’t happy with it. To get up and running in time we had to log in as an administrator, install PGP, start the encryption process and use Windows Fast User Switching to Switch User and allow the users to immediately start using the machine. We are still troubleshooting and testing but we believe that was the cause.
Conclusion
There are a few things I learned throughout this process. It was a good experience and I know next time I will do a few things different.
- When you are faced with a roll out of this size, take your firstestimate of time and double it just to be safe.
- Make sure you always let the image “soak” for a minimum of a week (two would be better) with a test group before you start the roll out.
- During the soak period make sure you restart the soak timer anytime you make a change.
- Think through the roll out and make sure you setup the machine to soak exactly how you will deploy it to the floor. There should be no variation.
- Make sure you get your boss to respond to any deployment plans that have to do with time frames. Silence is not golden.