UKC02 – Service Outage Updates

---

Latest Update

Sun 16th July 1024 : The server data restore is complete, we have provided a list of the commonly reported issues below:

Plesk Default Page

Upon completion of the restore there were a number of websites displaying the Plesk Default page, this has been resolved. If you have a ticket open regarding this issue, please try accessing your website again and let us know if you able to / unable to access your website.

Email

We have applied the new server encryption key to all mail account passwords, you will now be able to login to webmail / Outlook using the most recent password set on your mail account.

Plesk Login

We have resolved the following error which was previously displaying inside the Client Portal, you will now be able to login to Plesk from inside the Client Portal:

Error code: 0. Error message: Unable to find appropriate manager for this version of Panel. Plesk should be at least 8.0 version.

MySQL Databases

If your database restore is incomplete, please submit a ticket to the HelpDesk and we will schedule a full restore of the affected database.

We have scheduled an upgrade of MySQL from 5.1 to 5.5.49 to be completed overnight to match the version that was previously installed on the server.

If you are seeing the following error in PHPMyAdmin please login to Plesk and reset the database user password for the database that you wish to access PHPMyAdmin for:

Connection for controluser as defined in your configuration failed.

500 error / Blank page

Upon completion of the restore the PHP version was set to 5.3 on all domains, we will be changing the PHP version to 5.4 for all domains on UK02 today to match the version that was previously used. This will resolve most reports of the 500 error / blank page. If you were previously using PHP 5.6 please let the HelpDesk know and we will apply PHP 5.6 to your domain.

If you have logged a ticket and have not yet received a response, please rest assured we are aware of the issues that you have reported and are currently working them.

1: PSA database – COMPLETE
2: Restore vhosts – COMPLETE
3. Restore emails – COMPLETE
4: Restore databases – COMPLETE
5: Restore DNS zone files – COMPLETE

Thank you very much for your patience throughout this difficult time.

Will I be compensated for the downtime? Yes! Even those who have not purchased a Backup Plan or Business / Unlimited hosting service will receive some form of compensation in the form of discount codes, discounted renewals and free hosting extensions. Full details will follow once we get everything working.

Who is assigned to this incident? Our best engineers together with the most senior technical resources of our upstream DC/Server partner EUKHOST are working to bring this matter to the fastest possible conclusion.

Why is it taking so long to solve? The sheer amount of data we are working with takes considerable time to restore and verify data integrity. Each restore cycle takes up to 20 hours and unfortunately the failure experienced has required multiple restoration cycles.

What are the priorities? Our primary objective is to retain 100% data integrity for our clients. The fastest method to obtain this goal together with bringing the server back online safely and as quickly as we can are our objectives.

I have my own backup, can I get an account on a new server? Yes, please let our helpdesk know and you can have a new account setup on a new server and perform your own data restore using the Plesk backup/restore tool.

Wed 12th July – 0708 : In unison with our upstream DC/Server partner EUKHOST, another attempt is in progress to restore all data to a newly built server. Data recovery is showing 31% complete. This will be our final attempt before we switch to a manual site-by-site restore process. We will post an update at 1800. We are overwhelmed by your support and appreciate your patience during this difficult time.

Tue 11th July – 1150 : A full data restore is currently nearing completion. This will be followed by a thorough data integrity check. If all checks pass then the server will be brought back online later today. We will post an update at 1800. Thank you for your continuing patience.

Tue 11th July – 0604 : A full data restore is currently nearing completion. This will be followed by a thorough data integrity check. If all checks pass then the server will be brought back online later today. We will post an update at 1200. Thank you for your continuing patience.

Mon 10th July – 2249 : A full data restore is currently in progress. Due to the large amount of data the process is expected to take 8 hours from this point. A data integrity check will then follow. We will post a progress update at 0800. Thank you for your patience.

Mon 10th July – 1810 : A full data restore is currently in progress. Due to the large amount of data the process is expected to take 14 hours from this point. While the restore is progressing we are investigating options that could speed up the process. We will post a progress update at 2300. Thank you for your patience.

Mon 10th July – 1134 : A full data restore is currently in progress. Due to the large amount of data the process is expected to take 20 hours from this point. While the restore is progressing we are investigating options that could speed up the process. We will post an update at 1800. Thank you for your patience.

Mon 10th July – 0600 : A full data restore is currently in progress. Due to the large amount of data we expect the process to continue throughout the day. We will post an update at 1200. Thank you for your patience.

Sun 9th July – 2300 : A full data restore is currently in progress and will run throughout the night. Upon completion, comprehensive and exhaustive data integrity checks will be performed. We will post a progress update at 0600. Thank you for your continuing patience.

Sun 9th July – 1600 : We are preparing for a full data restore from the data set taken 0400 on Saturday 8th July. The process will begin this evening. We will post a progress update at 2300. Thank you for your patience.

Sun 9th July – 1100 : The Server Recovery Team have requested more time which we have granted. We are now prepared to run a full data restore from the most recent data set if required. Once the restore process begins we will have an ETA for its completion.  We will post an update at 1600. Thank you for your patience.

Sat 8th July – 2300 : The Server Recovery Team continue to work with the volumes and configuration checks which are going to run over night and in to tomorrow morning. If they not satisfied with the results we will begin a full data restore from the most recent data set taken 0400 on Saturday 8th July. We will post an update at 1100 tomorrow morning. Thank you once again for your patience.

Sat 8th July – 1800 : The Server Recovery Team continue to work with the volumes and configuration checks which could lead to a speedy recovery. However, it is possible we may need to perform a full Data Recovery from our Backup System. The most recent data set we have is 0400 Saturday 8th July. We still hope to avoid a full data restore (it’s a lengthy process so we we hope we can avoid it) however in the meantime it’s important for us to verify exactly what went wrong, to ensure we do not inherit the problem going forward. We will post an update at 2300. Thank you for your patience.

Sat 8th July – 1400 : The Server Recovery Team are currently performing an exhaustive check of the server drive units and controller. This process will take a number of hours. We will post an update at 18:00. Thank you for your patience.

Sat 8th July – 1000 : Server UKC02 has failed to start up successfully this morning after a restart. The issue is now being investigated by the Server Recovery Team. An update will be posted at 1400 hours. Thank you for your patience.