Recent Server Downtime
- Publisher Ted
- Date published
As many of our customers noticed, all web sites operated by Cracked Egg Studios, LLC, including CrackedEggStudios.com, were unavailable from December 18 - December 24, 2013.
Server Compromise
On December 12, our server began experiencing database corruption and by December 14, we learned that our IP address had been added to a number of email block lists. We spent the next few days investigating and our host found that a virus had hijacked our internal Tomcat instance and changed its security settings so that it could bypass our mail software to send SPAM.
On December 18, our web sites were replaced with a generic maintenance message suggesting we would be back in a few hours. Unfortunately, the infection was more persistent than initially thought, so our host's estimate of 6 hours quickly turned into the better part of a week.
On December 18, our web sites were replaced with a generic maintenance message suggesting we would be back in a few hours. Unfortunately, the infection was more persistent than initially thought, so our host's estimate of 6 hours quickly turned into the better part of a week.
How This Affects Users
While there was no evidence that the virus or an unauthorized user had accessed sensitive information (in fact we believe the virus simply propagated from an infected workstation that was connected via SSH), we recommend that any users who submitted any Install, Upgrade, or Import service requests during the month of December should change their site and FTP login information where possible. If you have ever purchased one of these services before December and want to be extra cautious, please do not hesitate to do the same.
We suggest that all users on our sites reset their passwords as soon as they are able.
We suggest that all users on our sites reset their passwords as soon as they are able.
Changes to Our Implementations
With this recent event, we noted some ways that our server security could be increased and have already done so.
In addition, we will be adding another level of separation to the services that were affected by this infection. Moving forward, we expect to move some internal software to a completely separate Amazon server that can achieve the same functionality without having Tomcat installed at all. Further, we intend to begin processing email traffic through an off-server service as well. These changes should begin rolling out during January.
In addition, we will be adding another level of separation to the services that were affected by this infection. Moving forward, we expect to move some internal software to a completely separate Amazon server that can achieve the same functionality without having Tomcat installed at all. Further, we intend to begin processing email traffic through an off-server service as well. These changes should begin rolling out during January.