XILO News and Status

As many of you will be aware there are wide spread problems on the LINX foundry LAN.

This will be causing additional latency and route instability. This is out of the control of any at XILO and will be affecting most of the UK Internet population.

The scale of the problem can be seen at https://www.linx.net/pubtools/trafficstats.html?stats=day

Any problems that traceroute over 195.66.224.0/23 are directly connected to this.

We just observed a small outage that affected some of our customers in BlueSquare 2.

Upon contacting the BSQ NOC we have received the following statement.

At approx 12:40pm today we experienced an issue with one of our four UPS systems whilst carrying our routine testing which resulted in this one UPS system powering off.

This caused a short brown out to approx 30% of customers equipment located in BlueSquare 2 & 3 as the unit powered off in an unusual way. The remaining three UPS systems took the load, and continue to provide power to the sites.

Full UPS protection is in place and we are running as normal and our UPS vendors are on-site investigating with senior management.

We aim to provide a further update within the next 3 hours.

We apologise for any inconvenience caused.

Our monitoring shows no servers offline at this time but if you are having problems connecting to your server - please raise a ticket via my.xilo.net.

The Foxtrot server has just suffered an unplanned reboot.

We are already looking into the issue and awaiting a disk check to complete before we can access the box fully.

We will update this post during the disk check process which we hope will take no longer than 15 minutes.

Apologies for any inconvenience this may cause.

Our monitoring has alerted us of some routing issues which may affect some customers into/out from the network.

We are looking into this with urgency and will update this post as soon as possible.

We have received the following notification from one of our transit providers.

Impact: Temporary service breaks throughout maintenance period.
Service Impacted: Customers in Sovereign House, Meridian Gate, Harbour Exchange 6&7, Harbour Exchange 8&9, Global Switch 1, and Telehouse East and any customers with interconnects traversing this network segment.

March 2008
Maintenance Window: 22:00 28th March 2008 until 06:00am 29th March 2008
Expected service outage: maximum 10 minutes downtime per site, however each site connection shall be moved over in 2 stages so downtime is expected to only be a few seconds, additional 30 minutes downtime for router upgrade in HEX8&9.

During this time there maybe short bursts of network interruption which may result in slow or unresponsive connections for upto 10 minutes as above.

Internal monitoring has alerted our engineers of a possible problem with one of our core switches. We are conducting an emergency reboot of the device to see if the errors continue.

There maybe a small 1-2 minute outage for some hosted customers.

Apologies for the short notice but immediate action is needed to ensure that a switch is replaced immediately if necessary.

An arbitrary file inclusion vulnerability has been discovered in the Horde webmail application. At present, we can confirm that this security vulnerability in question affects Horde 3.1.6 and earlier. Based on incomplete information at this time, we also believe this affects Horde Groupware 1.0.4 and earlier as well (cPanel does not use Horde Groupware at this time).

cPanel customers should update their cPanel and WHM servers immediately to prevent any chance of compromise. The patch will be available in builds 11.18.2 and greater (or 11.19.2 and greater for EDGE systems). The updated builds will be available immediately to all fast update servers. The builds will be available to all other update servers within one hour of this posting.

To check which version of cPanel and WHM is on your server, simply log into WebHost Manager (WHM) and look in the top right corner, or execute the following command from the command line as root:

/usr/local/cpanel/cpanel -V

You can upgrade your server by navigating to  ‘cPanel’ -> ‘Upgrade to Latest Version’ in WebHost Manager or by executing the following from the command line as root:

/scripts/upcp

It is recommended that all use of Horde 3.1.6 and earlier be stopped (on cPanel and non-cPanel systems alike) until Horde updates can be applied.

You can disable Horde on your cPanel system by unchecking the box next to ‘Server Configuration’ -> ‘Tweak Settings’ -> ‘Mail’ -> ‘Horde Webmail’ within WHM, and saving the page with the new settings.

We’re currently looking into DoS (denial of service) attack against one site on Indigo. This has caused the web server to flood with connections and some pages may not currently load.

The site concerned has been suspended and we’re attempting to block the attack at the edge network.

After a systems upgrade by our card processor this weekend, some customers regular payments were not debited correctly resulting in a decline message being sent via email.

This issue was raised with the card processor who investigated the issue and re-configured our account correctly which appears to have resolved the problem.

Please contact billing via my.xilo.net if you are having continued problems in making payment.

After requests from customers, we have altered the way our blog sends out email notifications.

Whereas before it would send one single email, it will now send emails on every update (comment) that is posted on the topic in question.

If you have any questions regarding this, please contact support via My XILO.