"They did WHAT?" Topic

Was a phrase repeated often today at the office. The reason goes back to Friday night.

I work in the systems group of a large call center that has sister facilities scattered about the country. We've got a nice room full of servers that house our applications and databases and generally serve to keep things rolling across the company. Its fairly nice and even has video surveillance to keep an eye on things.

We use a cleaning company to do a lot of the maintenance. Many companies do and its often done at night as the best time to run the sweepers is when no one is around. Its overwhelmingly staffed with immigrants with poor to no English skills.

So early Saturday morning, one of the cleaning staff is sweeping the floor of the server room when he accidentally whacks the broom handle into a plexiglass housing on wall fixture, knocking it off the wall. An alarm sounds.

The guy is looking around, the fixture that the plexiglass covered has a big button and some flashing lights. He's confused. Why is that alarm going off? How do I make it go away? Hey, these lights are flashing – maybe pushing the button will stop the alarm! <Insert visual of button being pushed>…

…and visualize every server in the room simultaneously powering down as the emergency shut off is activated. If only he could have read the sign above the button.

EMERGENCY POWER SHUT OFF.

Lucky for him it wasn't the button for the halon extinguisher.

Lucky for me I didn't have to roll out of bed in the middle of the night and re-start the servers and all the associated applications.

My first guess from your description was tthat it was going to be the Halon, that would have been bad.

Can't really blame him. Mr Bean does it all the time.

We had something like that happen a few years ago. A piece of evidence that caused a fire, and many millions of $$ in loss in a factory fire was left on an investigator's desk. The cleaning service saw a piece of burned trash and threw it in the trash. The investigator, the cleaning service, and a large sum of money are gone as a result.

I once saw somebody set the Halon on a rather old-school server room off – they were groping round for a light switch to turn off the lights. There was a loud noise – almost like an explosion – and the room instantly filled with condensation. In this particular set-up there were teletype terminals with paper, which got blown everywhere like some sort of large-scale party poppers.

We once had a member of staff, I repeat a member of staff, plug in a giant power drill inside the UPS/surge protection etc. Fried a few servers from the inside. He was popular…

People are morons. It's our superpower.
-Kle.

We moved offices once. People were instructed to mark all things to be thrown aways as "Trash" and everything else to be moved with your name so that they'd find their way to the proper cube. A lady named Trish did exactly what she was told. And she came to new office the next Monday to a cube with no computer. Seems the cleaning service threw it away. Between me and you, I think our company gave the poor cleaning crew way too hard of a time.

In the early 90's I was working on a project setting up new blocking data for the railroad yard I was working at. The program was not new but had never been applied to to what I was doing. As I was doing my data entry work one day every so often my terminal would lock up and I would get an ABEND message. This happened may seven or eight times over a period of several hours. It got to the point that I was getting very frustrated with the main frame going down like that. I then got a call from a Systems guy at our headquarters. He asked if I was using such and such a program on my terminal. I told him yes why, and he said "STOP!!" He told me every time I made a specific type of data entry update I took the main frame down. I had lots of people in an up roar, up to the Systems VP.

They found a fix and I was back working on my project two days later.

Terry

Yeah, a likely story Terry. I know you're really an evil mastermind using the servers to take over the internet, only their low computing power saved us from Callahan's Cult of 6mm Wargaming.

Yeah, a likely story Terry

A little unfair Streitax – I used to work on the kind of comms emulator code that the early remote terminals used, although a little earlier than the 90's – and crash-causing sequences werent unknown. Some of them were a bit like the "buffer overrun" exploits that can still crash modern web servers.

I have no doubt that the simplest things can crash the most complex systems, it's their fate due to their complexity.

The offices I work in don't have a clean power supply- everything plugs into normal sockets. Sitting at one desk I caught the cable on the EXTENSION LEAD and lost 15 minutes work. Plus once a month they test the emergency generator. They run the generator for a few hours, but don't tell you the exact time of the test. You just have to remember to save every thing every few minutes. I'm not even sure how much damage it is doing EVERY electrical device in the place having a powercut then brought back up again is doing.

I have three of my own. A looong time ago, I worked for a small company in a large office high-rise. We had an early mini-computer that was powered from a standard 110v outlet. Every night for about two weeks, the system would go down, and we would have to restart it in the morning. Finally, someone was working late (11pm) and noticed that the lights dimmed. It turns out that the building was reducing the voltage to save money, and didn't bother informing the tenants.

The next one is even worse. The same company built a dedicated computer room, but the contractor located the emergency shutdown/halon button behind the inward-opening door, perfectly aligned with the door handle. How long do you think it took for someone open that door just a little too hard…

Finally, we had a server in a the office kitchen/break room (another really small company) and every Monday morning, the server was down. We tried power monitors, etc. It turns out that the cleaning crew was unplugging the server to use the outlet for their floor scrubber.

Years ago I worked for the Fred Harvey Co, which was a part of AMFAC which ran the lodging, and stuff for the tourists at the Grand Canyon South Rim.
One day the entire network went blooey! I was the late night shift person and I had to call the Manager, who then tried to talk me into getting it back up..nada…
She came in, the asst. came in, etc…
FINALLY we had to go to the routers….

When they set this new network up…they put the ENTIRE set of routers for Arizona Steakhouse, El Tovar, Bright Angel, etc…IN THE KITCHEN of the EL TOVAR…

So we wander into the kitchen of the El Tovar and find the cleaning crew…with a high pressure water hose…hosing down the floor…
Oh, and did I mention that they put the routers in the rack REALLY LOW..as in…near the floor???

And folks wonder why that company was taken over…

Ohhh – The BIG E-P-O strikes again.

I preferred the dats where there was a rubber handled axe (ax) behind a class plate.