We have located the Essex email server IP address in a couple of blacklisting services.  The email server IP address got listed primarily due to a couple of catch all accounts which received an influx of spam, this then got forwarded to various email providers resulting in a block.

We have located these accounts, disabled catch-all account feature and removed the emails from the email server queue.

It is suggested and strongly recommended that customers disable catch all accounts feature and switch it to :fail:

We apologize for the inconvenience caused.

Thank you,

Stonington has remained online since 6:02AM this morning and we have been fielding data corruption/restoration requests from clients on this chassis.  Approximately 25 Virtual Private Servers were impacted by this failure, a majority of which do not have corrupt data.

For those that are having us issue restores, please continue to utilize our account manager for interaction with our support staff (https://accounts.networkredux.com).

Aside from disaster recovery we are working with Dell TAM who will be assisting in aftermath forensics.  A brief summary of what is going on and who is impacted:

Stonington.networkredux.net (VPS Chassis)

8 Processor / 32GB Memory / 6×450GB SAS-15K RAID-10

25 Customer VPS Nodes

Approximately 400GB of data.

In April the stonington chassis experienced two significant disk array failures which prompted replacement of the raid controller card, the raid battery backup, and the cable connecting the battery to the controller card.  In the process of this maintenance firmware updates were applied to the controller card.

Early this week the system began to exhibit new symptons related to disk array failures, requiring the system to be taken offline and data to be verified.  The server was brought back online with ~15 minutes of this failure and virtual private servers were brought back online for the customers impacted.

Correspondence with our hardware vendor suggested a newer controller card update as well as controller driver that specifically dealt with similar RAID-10 configurations.  Before we were able to schedule this maintenance, the server began experiencing another failure last evening, which resulted in a ~12 hour file system integrity check due to the massive amount of data corruption which resulted from the failure.

Clients will be live migrated off of this chassis today as we verify their data is intact, and this server will be cycled out of production.

The stonington chassis is online as of 6:02AM PDT.  File system checks took a majority of the evening and we are still going to be dealing with client data that may need to be restored from backup.

All nodes on this chassis are online currently.  If you suspect any issues with your node please let us know immediately and we will flash restore your environment to last nights backups.

Stonington is currently working through a lengthy and extensive file system repair after a hardware failure triggered by the RAID controller card caused significant corruption.  Due to the size of the RAID-10 array (6×450GB SAS15K) and the amount of data stored this is taking a great deal of time.

We appreciate your patience and will continue to update this list as the stonington server works through the file system repairs.

The Stonington maintenance window has turned into an emergency maintenance and prematurely initiated at 6:15PM (PDT). We apologize regarding any inconvenience and thank you for your patience.

Estimated downtime: 30 minutes

Update: File system integrity check is taking longer than anticipated due to a large volume of disk errors. We will update this list once completed.

This evening we will be performing maintenance on the Stonington hardware node.

The maintenance window is scheduled for 8:00PM - 9:00PM (PDT) this evening with an estimated downtime of 30 minutes.

We apologize regarding any inconvenience and thank you for you understanding.

The Stonington Hardware Node has experienced a temporary disruption of services. Technicians have been dispatched for further investigation while we work toward a quick recovery.

Update: Stonington has quickly recovered and all virtual machines online while we run post-failure analysis.

If you do not know what HyperVM is, you can safely disregard this notice.

Security issues have been publicized regarding the HyperVM platform which a portion of our OpenVZ virtual private server customers utilize. As a temporary precaution we are disabling the access ports to HyperVM. This will not impact your virtual private server in any way shape or form other than temporarily losing the ability to rebuild your vps node with a new image on your own. If you need to rebuild your VPS image please issue a support request and one of our engineers on call will take care of this for you.

All other features remain intact include SSH access and out of band management (console) to your virtual private servers, as well as any other services and servers you may have installed or configured on your virtual private server.

We will keep this notification list updated as the security items are addressed by the upstream vendor.

Thank you.

Our facilities provider for DC2 and DC3 will be performing maintenance to the power grid for both locations.  Before performing the maintenance we will be transitioning to generator power, and anticipate running off of generator power for less than 1 hour.

The maintenance window is scheduled for 5:00AM Saturday June 6th and expected to last until 6:30AM.

Due to the redundant power distributions we have specifically built into DC2 and DC3 this operation is planned as non-intrusive.

Thank you.

The following maintenance window has been provided by one of our upstream carriers, tw telecom:

tw telecom Scheduled Maintenance Id . . SM000004815
Start Time  . . . . . . . . . . . . . . 05/28/2009 07:01 UTC/GMT
End Time  . . . . . . . . . . . . . . . 05/28/2009 10:00 UTC/GMT
Estimated Downtime  . . . . . . . . . . 20 to 30 minutes

DC1 users will be impacted as that facility is single homed with tw telecom.  DC2 and DC3 users will not experience disruption as traffic will reroute over our additional uplink connections.

Thank you.