XILO News and Status

We’re looking at an issue with the Echo reseller server. Currently services are responding very slowly.

Engineers are currently looking into the issue.

We’re currently reloading all servers with Fantastico. It appears that the cPanel update last night has removed it.

Please bear with us whilst we complete this work and repair the installations.

Once again, LiNX appear to be having network/routing issues.

This will be causing additional latency and route instability. This is out of the control of any one at XILO and will be affecting most of the UK Internet population.

The scale of the problem can be seen at https://www.linx.net/pubtools/trafficstats.html?stats=day

If we have any update from LiNX, we will post accordingly.

We just observed a small outage that affected some of our customers in BlueSquare 2.

Upon contacting the BSQ NOC we have received the following statement.

At approx 12:40pm today we experienced an issue with one of our four UPS systems whilst carrying our routine testing which resulted in this one UPS system powering off.

This caused a short brown out to approx 30% of customers equipment located in BlueSquare 2 & 3 as the unit powered off in an unusual way. The remaining three UPS systems took the load, and continue to provide power to the sites.

Full UPS protection is in place and we are running as normal and our UPS vendors are on-site investigating with senior management.

We aim to provide a further update within the next 3 hours.

We apologise for any inconvenience caused.

Our monitoring shows no servers offline at this time but if you are having problems connecting to your server - please raise a ticket via my.xilo.net.

The Foxtrot server has just suffered an unplanned reboot.

We are already looking into the issue and awaiting a disk check to complete before we can access the box fully.

We will update this post during the disk check process which we hope will take no longer than 15 minutes.

Apologies for any inconvenience this may cause.

Our monitoring has alerted us of some routing issues which may affect some customers into/out from the network.

We are looking into this with urgency and will update this post as soon as possible.

Internal monitoring has alerted our engineers of a possible problem with one of our core switches. We are conducting an emergency reboot of the device to see if the errors continue.

There maybe a small 1-2 minute outage for some hosted customers.

Apologies for the short notice but immediate action is needed to ensure that a switch is replaced immediately if necessary.

We’re currently looking into DoS (denial of service) attack against one site on Indigo. This has caused the web server to flood with connections and some pages may not currently load.

The site concerned has been suspended and we’re attempting to block the attack at the edge network.

Due to a customer script, the KILO server had two unplanned and necessary reboots today. The script in question saturated the CPU and as such, we were unable to login via SSH or console.

After the reboot, we identified the customer and have suspended their account until we can resolve their issues.

Unfortunately, due to the unclean reboot - the RAID array is having to initialise the data on the array. This will take a few hours and during this period, the server will respond slightly slower than normal.

Apologies for the two small outages and the slower than normal response times whilst the RAID initialises

We’re currently rebuilding the cluster platform for the servers above.

Unfortuantely, after the issues relating to power, it has severely corrupted the storage array. We have attempted to restore from our back cluster but this has failed. We have contacted our backup software provider to ask them for assistance.

We’re also currently storing the data from the damaged array to another server so we can at least start a manual rebuild.

Please bookmark this page if possible instead of raising tickets or phone calls as we’re extremely busy and not able to answer all calls in a timely manner.

We’ll update this as much as possible and hope to have service restored soon.