I originally submitted this post to Docker people in the celebration of the 2015 Sysadmin Day, and they selected it as one of their favorite war stories. Now I publish it in my own blog.
Some time ago I was working as Linux sysadmin in a major company. Our team were in charge of the operating system, but other teams were the applications administrators. So in some circumstances we allowed them some privilleged commands via sudo. The could do some services installs/patching in this manner.
One day I received a phone call from one of our users. He said me there was a server with a erratic behaviour. I tried to ssh on it. Connection refused. I tried to log in from the console, and I only could see weird messages.
So I boot the server in rescue mode with a OS iso. I mounted the filesystems. And I began to see someone was changed all the permissioms in all the system. I investigated for a while, I could discover who was the guilty, and the command that executed, a sudo chmod -R something /
How we can recover the server in a situation like this? With previous steps (changing some permissions on hand, chrooting) we do it using the rpm database:
for p in $(rpm -qa); do rpm --setperms $p; done
for p in $(rpm -qa); do rpm --setugids $p; done/sbin/conf.d/SuSEconfig.permissions