Pick the best of the worst

Now's your chance to vote on the real-life DBA disaster story that you'd like to see turned into the DBA Team's Christmas special.

From a host of frighteningly strong contenders, we've picked five of the best - or should that be the worst.

Here's a teaser for each:

Your number's up

I received a call around 6:55pm, from a user who said his query had updated 1.8m rows and he only expected it to update 1 row. He nervously admitted to missing the WHERE clause.

A few queries showed that our operations team had inadvertently updated all customer phone numbers to the same number, which would cause havoc with our billing run starting at 10pm.

I had 3 hours to resolve the issue…

Slow on the uptake

The shipping company I worked for had a tracking application which relied on database replication to stay in sync.

We brought a new site online, but the location didn’t want to pay for monitoring.

One day, they weren’t able to make updates. Replication was failing. Someone had also disabled the transaction log backups to make the system "faster".

The replication settings were set to push from the Publisher. That means that the only choice we had to bring them back online was to re-sync all our locations…

Finger on the trigger

My company required a critical database change for a client. Due to poor database design, the database design triggers to try and maintain integrity.

I wrote a script to use a lookup table to make the changes. It worked fine in testing – so we were ready to run it at the client. The company I worked for was very budget minded, so they decided to send a sales person to make the updates. Fortunately, he was intelligent and readily understood what he had to do.

Before he left I told him to check that they had a backup before running the script. Something had gone wrong, so I asked him for the backup.

"It's a spreadsheet," he said.

I asked "What's a spreadsheet?”

He said "Their backup..."

Someone get me a doctor

Monday morning, 7:00am. I had just sat down with my coffee, when a colleague ran in and told me the database was down. He was speaking so fast that I couldn't catch what he was trying to say.

He told me that he accidentally deleted the LUN of the transaction log files for the mission critical hospital database. The last full backup was corrupt and restoring an earlier full backup wasn't a choice.

At that point, the CIO stumbled into my room, and started shouting and demanding to know when I was going to fix the problem…

Disaster zone

I received a call from our Windows Admin Manager about some problems with SQL Server. We had just finished a DR exercise, and he informed me that the DR site and live site were reconnected before all the DR servers were destroyed. This overwrote my Cluster computer objects.

The problems started with one instance that was part of the DR – the one responsible for producing a large number of checks, worth an extremely large amount of money. It went offline and couldn’t be brought back.

Another affected instance was responsible for scheduling ALL the jobs and nightly batch for the entire company. Another handled the CEO's call conferencing software...