A few things went wrong today. well, not exactly wrong, just not the way I had expected.

we got a message from mcaffee saying that their copy of nessus had found that we were running apache < 2.2.9 – which we were, so it was time for us to update our system. the server hasnt been updated since at least the time I started. so I did a yum update.

bad move.

there were a few things that happened – first I had to add all the gpg keys to yum. easy enough. then I had to run yum update again. fair enough. it downloaded 600 meg of updates, and started applying them.it updated apache – which was what I wanted to happen.  then it did the wrong thing. it stopped the apache service. and it ran the quotacheck. then it had to wait to start the plesk services. wrong order. all our customers servers were down for half an hour, needlessly. if it had been stop apache, restart plesk, start apache, run quotacheck then the server would have blipped out for about 2 seconds.

i had 5 people phoning me up to tell me that their server was down, and I had to explain to them that the console said ‘this may take some time’ and that there was nothing I could do while it was running, I simply had to wait. most people understood, but one did not. Even after I explained that it was critical to their mcaffee and PCI complience they still didnt want to accept it. they said ‘ill phone your boss because this is unacceptable’ I said ‘fine there is nothing that he can do either unless you want him to hose the system completely. just wait half an hour for the check to finnish and it will be back up’.

the problem was that yum had shut down apache, installed the new version (overwriting the httpd.conf) and then ran an update that takes half an hour before it installed plesk, which generates the vhost files. so there was no path to any of the virtual hosts.

quotacheck finnished and plesk installed and the vhosts came back up. the total time was about 26 minutes. I phoned round the people that had phoned to tell them their systems were back up. most were happy. one didnt even answer the phone. same one as before.

anyway, I checked to see if plesk was running, it wasnt. it was complaining about keys. I checked arround the net, it turns out that plesk needs through the firewall to get a new key. way to go plesk for opening random ports. ill create a rule that only allows a connection to and from the plesk key server.

the only solution was to erase plesk from the rpm database, keeping the files and then to install it via yum with the port open for it to get the key. it worked.

though when plesk was installing it ran the quota check again (it didnt stop the running httpd services though so the vhosts didnt go down) – if this was the behaviour when the original update happened then it would be fine. or even if the httpd update had kept the original vhosts.conf file everything would have been fine. even if it had told me that it would overwrite the vhosts file id have kept a copy (the thing is that each vhost has its own vhosts.conf in its own directory – it would have taken me hours to write it by hand) .

my advice is – DONT UPDATE YOUR SYSTEM USING YUM. instead, update everything by hand, one by one. or find some way to stop quotacheck from running.