Around 2000 my team was responsible for installing and maintaining a larger amount of servers in 19" racks in a data centre.
Most servers had those hot swap drive bays for convenient access from the front while the server was running. You only had to make sure no write operation occurred while you pulled the drive out of the bay.
So, I had to exchange a backup disk on a database server running quite a few rather large forums. The server had two disk bays: One for the live hard disk and one for the backup disk. I was absolutely sure at that time which one was the backup disk so I didn't bother to shut down the database server and incur a minimal downtime. Of course, I was wrong and blithely yanked the live disk from the drive bay.
I spent the rest of the night and most of the following day running various MySQL database table repair magic. It worked out surprisingly well but having to admit this error to our forum users was embarrassing, nonetheless.
Lesson: Appropriately label your servers and devices.
Most servers had those hot swap drive bays for convenient access from the front while the server was running. You only had to make sure no write operation occurred while you pulled the drive out of the bay.
So, I had to exchange a backup disk on a database server running quite a few rather large forums. The server had two disk bays: One for the live hard disk and one for the backup disk. I was absolutely sure at that time which one was the backup disk so I didn't bother to shut down the database server and incur a minimal downtime. Of course, I was wrong and blithely yanked the live disk from the drive bay.
I spent the rest of the night and most of the following day running various MySQL database table repair magic. It worked out surprisingly well but having to admit this error to our forum users was embarrassing, nonetheless.
Lesson: Appropriately label your servers and devices.