Split is a platform for configuring and analyzing experiments. At GoDaddy, we’re building an experimentation platform leveraging Split as the configuration GUI and bucketing engine while layering more advanced and GoDaddy-specific features on top of the experimentation capabilities already available within Split. One challenge with experiment bucketing is that the time spent to get the library ready, which involves downloading the experiment definitions at page load time, might impact the KPIs you’re trying to optimize. For example, Google has run experiments demonstrating that introducing 100 milliseconds to web search causes the daily number of searches per user to drop by 0.2%. Given this potential impact, we joined forces with Split to design and implement an open-source solution that would allow us to get the benefits of an experimentation platform with no performance penalty.
Some applications at GoDaddy store static HTML and other assets in a CDN, like Akamai, to optimize Time To Interactive (TTI). We wanted to integrate those applications with our experimentation platform, but since experiment cohort assignments are user-specific, the integration options we had available would have lost the benefit of caching. To maintain a low TTI, these apps should be able to continue serving their assets from a CDN and to fetch experiment cohort allocations with low latency.
Our requirements for a solution included:
The initial set of solutions we explored would all add latency to applications or overhead to our partners, limiting the kinds of experiments we could run and adding friction to broad adoption.
Ideas we considered for fetching experiment bucket assignments client-side, but then rejected:
We shared requirements with the team at Split and proposed an idea to separate raw configuration (data outlining experiments like cohorts, targeting rules, and split percentages) from decisioning (the point at which a user is assigned to a cohort for a particular experiment). This approach avoids any network call from the browser, works with edge caching because no user-specific information is sent to the browser, and decisions in microseconds. Split provided a design spec, and we both contributed to a set of tools that let us co-locate raw configuration alongside GoDaddy teams’ cached pages at the CDN layer.
On the server-side, to keep cache benefits, we needed a response that is not user-specific. One of the advantages of using Split is that experiment targeting happens on the client side, so the server can respond with a generic payload.
To do this, we built a small server-side tool in Node.js and Golang that periodically fetches split definitions and segments data from the Split API and serializes it into a set of strings to be uploaded to the CDN and cached alongside the page content. Those strings are subsequently consumed by the browser SDK when the page loads.
We then integrated this tool into our GoDaddy experimentation SDKs (Node.js and Golang at the moment):
On the client, we needed to handle decisioning.
configured to use
LOCALSTORAGE. When the local storage cache
support evaluation in this “ready from cache” state, thus avoiding an extra call
to Split’s CDN to download the raw configuration.
We first integrated this new set of experimentation tools into the domain purchase pathway - the flow GoDaddy customers go through to find and buy a domain name.
We ran A/A tests on each application in that flow to ensure that having the tooling in place would do no harm and that the pieces were working end-to-end.
Since then, we’ve launched the first A/B test using this solution on the cart page, resulting in an increased conversion rate.