Behind the scenes: How we responded to Heartbleed

First things first. As we noted in our first post about Heartbleed, we’ve fully implemented the recommended patch for the Heartbleed SSL issue across primary GoDaddy services. Heartbleed is a critical vulnerability that has affected nearly two-thirds of the Internet. Many Internet companies, large and small, have been working long hours to update their services to keep customers and visitors safe.

GoDaddy has over twelve million customers, and is one of the largest web hosts. How does an organization of scale turn on a dime and secure an operation of this magnitude against an industry-wide vulnerability such as Heartbleed?

Locking down the perimeter

In this type of incident, we always pursue multiple solutions in parallel. As our operations team was assessing the impact to our servers, our security team was hard at work developing filters to detect the attack. We continue to monitor the situation looking for any threat shifts and will adjust as appropriate.

Rolling out a patch to secure millions of websites

We have processes in place for “patch deployment at scale,” and our operations teams have a sophisticated set of configuration management and patch management tools in place to support both regularly scheduled patching and rapid response for critical events such as this one. This enables us to maintain a secure environment at all times, even during an incident.

In this instance, we used these tools to first assess the environments running vulnerable versions of OpenSSL, and then systematically pushed out the updates and initiated the required Web server restarts. We used a combination of tools and automation to manage this process.

Management of certificates

In addition to hosting millions of websites, GoDaddy is a large provider of SSL certificates. We quickly realized that this vulnerability would cause a spike in “rekeys,” a term used to describe the replacement of a certificate with a new one using a different private key. Because Heartbleed can potentially expose private keys to an attacker,  who can in turn use them to intercept encrypted traffic to that website, we’re recommending that any certificate used on a server that was vulnerable be rekeyed once the server is patched. We are not charging our customers to rekey their certificates.

Rekeying certificates causes the old ones to be revoked after 72 hours, which in turn places extra load on our CRL and OCSP validation services that browsers use to determine if a certificate should be trusted. We have a highly scalable system in place – we serve well over 1 billion OCSP responses every day – but nevertheless we’re keeping a close eye on these systems to ensure they remain healthy.

What you need to do

While we have rolled out the patch to all of our Shared Hosting and WordPress Managed Hosting customers, if you are running certain versions of OpenSSL on your Linux server using a Virtual Private Server (VPS) or Dedicated Server, you may need to take action.  Additionally, if you have shared hosting and an SSL with us, send an email with the common name of your SSL to heartbleed@godaddy.com, and we’ll take care of it.

If you purchased an SSL from us and are hosted elsewhere, ensure your service provider is patched before rekeying your certificate.

How can you check?

Run the test available at http://filippo.io/Heartbleed/ — this will tell you whether or not your website is impacted.

What if your server needs attention?

Update your server to the latest version of OpenSSL. Click here for the instructions to secure your server against the Heartbleed vulnerability.

1. Restart all services that use OpenSSL. (If you aren’t certain which services this includes, restart your server.)
2. Rekey any SSL certificates your server uses. We have instructions for rekeying certificates you purchased through us here. Again, we are not charging to rekey certificates.
3. Ensure you’re using your SSL properly by using an SSL configuration tool.
4. Test your mail server configuration using a tool such as CheckTLS.com.

Wow, that was intense – are we all good now?

Let’s make sure. Double-check your domain name at http://filippo.io/Heartbleed/ and make sure you get an “All good” response.

Hope this has been helpful. As always, reach out to us if we can answer any questions or if you have any concerns. We will keep you updated if any further action is required as the situation unfolds.