Membee’s Data Center Back to Normal

Good Morning Everyone,

As promised, we wanted to post some detail regarding the issues with our hosting provider that affected the services of Membee on Friday, July 3.

We are including the incident report provided to us by our hosting company, Tenzing for you below:

“Investigation of this incident has determined a major hardware failure on the primary firewall occurred which prevented the device from being restored.

 

The initial fail over to the secondary firewall did not perform as expected and over the course of this incident the secondary firewall stopped serving traffic multiple times until it was ultimately removed from its role as primary.

 

It was later determined that the secondary firewall was experiencing an unrelated hardware malfunction causing the device to periodically drop its routes. Each time, a restart was initiated to restore service. Analysis has led us to believe that this condition was present prior to the incident but went undetected by the firewall’s monitoring and diagnostic systems.  It was not until traffic was routing through the device consistently over a period of time that the malfunction could be detected.

 

Both firewalls have been replaced and will be returned to our vendor for further analysis.  High-availability of the network has been restored and the network is performing as expected.”

We have been monitoring our systems since the incident and are happy to report that everything is up and running as expected.  We are always committed to ensuring that we can deliver worry-free performance to all our users and your members and we will continue to work with our partners to ensure that this is maintained.

You can find some great information on how we keep our systems up and running and your data safe by visiting us at http://membee.zendesk.com/entries/20982867

 

As always, we wanted to keep you in the “loop”.

Thank you,

Membee Customer Service Team