It's hot in the city!

Here's an email representation of how the week at work is going;


Sent: Sunday, July 04, 2010 10:22 AM
To: ITS TechAlert
Subject: RE: Server issues

As a result of the partial power failure in the datacenter, several servers and appliances went down. Application owners; please verify for apps and/or contact me
Thanks Mike, Dmitriy, Yury, Foroozan, Zaki, Ernando, Munir and Joey for participating (directly and remotely) in this unscheduled Sunday event.

Sent: Monday, July 05, 2010 2:38 PM
Subject: High Temperatures in Data Centre Require Systems to be Shut Down

Good afternoon,
The failure of one of the compressors on one of the A/C units for the Data Centre and the increased loading caused by the high outdoor temperatures has caused the remaining units to reduce their operating capacity by 50%. This is causing temperatures in the Data Centre to approach 100 degrees and may result in systems going into thermal failure.


Sent: Tuesday, July 06, 2010 9:26 AMTo: ITS TechAlert
Subject: RE: datacenter shutdown, an update

SERVERS UP - 129
SERVERS DOWN – 164
DATACENTER room temp at 9.30am = 75F, 24C

Sent: Tuesday, July 06, 2010 July-06-10 22:51
Subject: RE: datacenter shutdown, an update, July 6, 19.00h

Unfortunately, power interruption last night at 21.00h seems to have triggered AC unit C malfunction, as a result, the room temperature is now reaching 83F.

There are now 113 servers UP with 180 DOWN

Sent: July-06-10 13:54
Subject: RE: datacenter shutdown, an update

The room temperature is now (13.45h) 77F, slowly going up.
147 servers are UP
146 servers are DOWN


Sent: Wednesday, July 07, 2010 7:01 AM
Subject: RE: datacenter shutdown, an update, July 7, 6.50h

At the present time, the temperature in the datacenter is 83F with very hot spots between racks.

Sent: Wednesday, July 07, 2010 2:02 PM

To: ITS TechAlert
Subject: RE: datacenter shutdown, an update, July 7, 14.00h


Servers
UP = 162
DOWN = 131
Room temp, 75F