Please bear with us whilst we complete this work and repair the installations.

This will be causing additional latency and route instability. This is out of the control of any one at XILO and will be affecting most of the UK Internet population.

The scale of the problem can be seen at https://www.linx.net/pubtools/trafficstats.html?stats=day

If we have any update from LiNX, we will post accordingly.

Upon contacting the BSQ NOC we have received the following statement.

At approx 12:40pm today we experienced an issue with one of our four UPS systems whilst carrying our routine testing which resulted in this one UPS system powering off.

This caused a short brown out to approx 30% of customers equipment located in BlueSquare 2 & 3 as the unit powered off in an unusual way. The remaining three UPS systems took the load, and continue to provide power to the sites.

Full UPS protection is in place and we are running as normal and our UPS vendors are on-site investigating with senior management.

We aim to provide a further update within the next 3 hours.

We apologise for any inconvenience caused.

Our monitoring shows no servers offline at this time but if you are having problems connecting to your server - please raise a ticket via my.xilo.net.

We are already looking into the issue and awaiting a disk check to complete before we can access the box fully.

We will update this post during the disk check process which we hope will take no longer than 15 minutes.

Apologies for any inconvenience this may cause.

We are looking into this with urgency and will update this post as soon as possible.

There maybe a small 1-2 minute outage for some hosted customers.

Apologies for the short notice but immediate action is needed to ensure that a switch is replaced immediately if necessary.

The site concerned has been suspended and we’re attempting to block the attack at the edge network.

After the reboot, we identified the customer and have suspended their account until we can resolve their issues.

Unfortunately, due to the unclean reboot - the RAID array is having to initialise the data on the array. This will take a few hours and during this period, the server will respond slightly slower than normal.

Apologies for the two small outages and the slower than normal response times whilst the RAID initialises

Unfortuantely, after the issues relating to power, it has severely corrupted the storage array. We have attempted to restore from our back cluster but this has failed. We have contacted our backup software provider to ask them for assistance.

We’re also currently storing the data from the damaged array to another server so we can at least start a manual rebuild.

Please bookmark this page if possible instead of raising tickets or phone calls as we’re extremely busy and not able to answer all calls in a timely manner.

We’ll update this as much as possible and hope to have service restored soon.

Unfortunately, due to an apparent knock-on effect, this tripped a power bar in one of our racks which resulted in the fuse blowing causing half of the PDU (power distribution unit) to stop supplying power to some servers and equipment.

Engineers are onsite and are trying to get back on all servers as soon as possible. Some have blown PSU (power supply units) because of this which is delaying matters.

A full outage report will be sent as soon as possible.