Network management Toolkit

Contractors 'killed' PlusNet data-centre power

David Meyer ZDNet.co.uk

Published: 22 Feb 2008 11:58 GMT

The PlusNet outage that hit the internet service provider's users early on Thursday was the result of human error at a third-party data centre, it now appears.

A full report on the incident is yet to be drawn up but, according to PlusNet, initial investigations show that contractors cut all power to the as-yet-unidentified data centre, in the process knocking out PlusNet's internal systems, portals and email.

"We have found out this morning that the owners of the data centre had contractors in doing some maintenance. They failed to notify us of this work," wrote PlusNet's communications manager, Mark Kelly, on the ISP's community page on Thursday.

Read this

Mobile World Congress 2008

Highlights from the Barcelona show...

"During that work it appears that the contractors killed all power to the centre which was just like hitting the big red fire button in the centre," Kelly continued. "That action caused all power to all services to die including the generators and [uninterruptible power supply]. Failover couldn't work as the power to everything was killed. We believe that this was human error; however, we are awaiting a full report from our suppliers."

Kelly added that not all services can be automatically failed over. "[Databases and so on] have to be failed over manually to ensure that data integrity is maintained," he wrote. "As we were unaware that this work was taking place we did not have anyone on standby to react. Thankfully our incident management processes and call out system kicked in very effectively. We had engineers on site within about 20 minutes and 97 percent of services were back up and running in around four hours."

"I'll reiterate that this was a catastrophic failure which we believe was caused by human error," wrote Kelly. "All the resiliency in the world could not have prevented this situation."

However, Kelly admitted that PlusNet's communications team should have become aware of the incident sooner and relayed information to customers more effectively. "I am dealing with that aspect internally this morning," he wrote.

Did you find this article useful?
11 out of 11 people found this useful

Post a comment

Full Talkback thread

0 comments

More In Network management

Company/Topic Alerts

Create a new alert from the list below:

Related Jobs

Human Resources Manager - NHS - North West - Contract

Leading NHS organisation now has a fantastic opportunity for a NHS Senior Human Resources Manager. The NHS Senior Human Resources Manager will join a ...

HCM / HR Managing Consultant - Human Capital Management consulting

You will be responsible for delivering large scale human capital management projects within clients transformation programmes.you will be expected to ...

Database Developers ( SQL / T-SQL / SSIS / ETL ) - Chatham Maritime

As and when required, carry out database administration and maintenance tasks including capacity planning, security and integrity planning, index ...

Featured Talkback

Could it be that ISP’s are making this out to be a bigger problem than it actually is? We’re a small country with an internet penetration of less than 60%, for every Youtuber there’s someone who only uses the internet to check their emails, more people surf on their mobile handsets than a few years ago. Surely things should even themselves up.