Hot Hot Hot!

Dealing with power and computing is one of those areas where you really want the right thing to happen.

Usually it is fair to say that you are going to be switching between sources, batteries, generator... stuff, in the interest of keeping the servers / cooling / switches / infrastructure running without interruption. Less boat rocking makes for a nicer day, and less grumpy people. Success.

However - a general rule of thumb is that this should not be at the expense of catching fire.

Some legacy UPS units we have - mostly in storage now -  are the APC Smart-UPS RT 3000's - 3U 3kVA weighty rack mounts that have 8 13 Amp kettle leads out, and two 16 Amp Square ones. We used to make use of them on sites that had unreliable local or no Uninteruptable Power Systems (UPS)... thankfully not something we have to put up with any more (short of obscure transit points of presence where they expect everything to be DC powered).

So. "While I am not saying it would not have turned off before it caught fire but..." the game of 'I will name that smell in three' - a world of stinking-of-epoxy-going-off it was apparent that this particular RT 3000 still racked, still powering some end off life internal hardware that we had been avoiding interrupting, was in a bad way and was not long for this world.

Fan seemed to be turning in the front (thus helping circulate the smell). Unit uncomfortably hot to the touch - battery sleds crazy hot. Having uncoupled it - taken the downtime hit, we dragged out outside and away from the building. Sat outside on paving slabs to cool down - its guts unplugged and battery sleds sat on the concrete. 2 hours later still noticeably hot to the touch (lead acid - this should not be a thermal run away scenario). A lucky escape all in all.

Dear APC  - given that we used to pull data from these things to plot with Munin using your smart serial cables - we know you monitor all manner of goodness. Watts, volts, temperatures, battery capacity, charge, so on. So with this in mind - HAVE A WORD - if it's hot enough to start melting things... it's probably reached that point where it is okay to suffer an outage. Do I have any seconders.


Leave a Reply

Your email address will not be published. Required fields are marked *