Where I’m working, they wanted to try “netoffice”, a PHP collaboration/time tracking/productivity thing. I downloaded the “dwins” version of it, and ran the install script after reading the instructions. Being a bit late, I hit the return key one too many times, and noticed some funny error messages about problems deleting files in /dev. Oh shit…

Turns out the script has the following line:

rm -rf ${INSTALL_DIR}/*

Simply brilliant. Thanks a lot, guys… So at this point I’m sitting there with two shells opened via ssh, no /bin/, no /etc/ and nothing much in /dev. For reasons I won’t discuss here we don’t have backups – suffice it to say that it wasn’t my decision.

Not good. ssh didn’t work any more, either in or out, because it needs files in /etc/ and /dev, and so I couldn’t just copy new stuff in. Luckily, I had a fully stocked /usr/bin, and I did the first thing that came to my head: a small server in Tcl that copied its input into the specified file. Using that and netcat on the other side to send the files, I recreated enough of /bin to get MAKEDEV working in /dev. Phew… a little bit better, but /etc/ was still gone. I brought in a few more bits and pieces like /etc/passwd and /etc/group using my Tcl server, which was enough to get scp working from the machine. At that point, I brought in the rest of /etc. I was starting to think about going home when I ran the fateful script, so at this point it was pretty late, I was starving, so the rest will have to wait for tomorrow.

The biggest problem, I think, is how to restore the files in /etc at least to the pristine installed state required by the packages that own them? The other problem is that the password file uid’s are now out of sync with the file system, which is causing problems here and there. Unless I think of a brilliant way to fix those two issues, I’ll probably consider myself satisfied that I got things running to a point where I can get the important data off the system, set up some temporary services on other machines, and reinstall the whole thing.

This is all on Ubuntu systems, by the way.