A few things went wrong today. well, not exactly wrong, just not the way I had expected.

we got a message from mcaffee saying that their copy of nessus had found that we were running apache < 2.2.9 – which we were, so it was time for us to update our system. the server hasnt been updated since at least the time I started. so I did a yum update.

bad move.

there were a few things that happened – first I had to add all the gpg keys to yum. easy enough. then I had to run yum update again. fair enough. it downloaded 600 meg of updates, and started applying them.it updated apache – which was what I wanted to happen.  then it did the wrong thing. it stopped the apache service. and it ran the quotacheck. then it had to wait to start the plesk services. wrong order. all our customers servers were down for half an hour, needlessly. if it had been stop apache, restart plesk, start apache, run quotacheck then the server would have blipped out for about 2 seconds.

i had 5 people phoning me up to tell me that their server was down, and I had to explain to them that the console said ‘this may take some time’ and that there was nothing I could do while it was running, I simply had to wait. most people understood, but one did not. Even after I explained that it was critical to their mcaffee and PCI complience they still didnt want to accept it. they said ‘ill phone your boss because this is unacceptable’ I said ‘fine there is nothing that he can do either unless you want him to hose the system completely. just wait half an hour for the check to finnish and it will be back up’.

the problem was that yum had shut down apache, installed the new version (overwriting the httpd.conf) and then ran an update that takes half an hour before it installed plesk, which generates the vhost files. so there was no path to any of the virtual hosts.

quotacheck finnished and plesk installed and the vhosts came back up. the total time was about 26 minutes. I phoned round the people that had phoned to tell them their systems were back up. most were happy. one didnt even answer the phone. same one as before.

anyway, I checked to see if plesk was running, it wasnt. it was complaining about keys. I checked arround the net, it turns out that plesk needs through the firewall to get a new key. way to go plesk for opening random ports. ill create a rule that only allows a connection to and from the plesk key server.

the only solution was to erase plesk from the rpm database, keeping the files and then to install it via yum with the port open for it to get the key. it worked.

though when plesk was installing it ran the quota check again (it didnt stop the running httpd services though so the vhosts didnt go down) – if this was the behaviour when the original update happened then it would be fine. or even if the httpd update had kept the original vhosts.conf file everything would have been fine. even if it had told me that it would overwrite the vhosts file id have kept a copy (the thing is that each vhost has its own vhosts.conf in its own directory – it would have taken me hours to write it by hand) .

my advice is – DONT UPDATE YOUR SYSTEM USING YUM. instead, update everything by hand, one by one. or find some way to stop quotacheck from running.


4 Comments

Jacko · August 20, 2009 at 2:38 pm

Sounds painful.

Regarding:

“anyway, i checked to see if plesk was running, it wasnt. it was complaining about keys.”

Did Plesk upgrade during the yum upgrade as well? ie. Did it go from Plesk 8 to Plesk 9? AFAIK when Plesk upgrades from 8 to 9 you need to retrieve a new plesk key from their key server. I’ve never had to open any ports on the inbound firewalls..

    jonathan · August 20, 2009 at 8:24 pm

    yeah, it upgraded, it just took a couple of hours for it to get the key =)

alan · September 30, 2013 at 8:22 am

RTFM! Plesk work fine if you know how fix some problem. Quota check you can stop easy. Kill vzquota process (start after “vzctl start VEID”) for you container from hw node.

    jonathan · October 1, 2013 at 11:31 pm

    yeah, but you shouldnt have to fix problems during an upgrade. every single time there is a major rev. update plesk breaks :| upgrading from 10 to 11 was dreadful. I thing this post was upgrading from 8 to 9.