I’m a huge believer in data driven decisions. So I was excited when my product team decided to adopt a well known feature experimentation tool. Experiments are great because they help the whole organization be more rigorous. Have a theory about your user? Test it. Disagree about product direction? Test it. Risky feature idea? Test it! Experiments can make the whole team move faster and with more certainty because you can start thinking about your product scientifically… if they work.
My excitement didn’t last for long. Adoption was frustrating and expensive for everyone - integrating took weeks longer than expected, introduced bugs to our applications, made us re-architect how we send events from the front end, and was difficult to communicate across the organization. By the end of it experimentation was a code word for pain in the ass and no one wanted to touch it. So much for democratizing data driven decisions.
So I started talking to other software engineers, product managers, and founders across the industry. Not just about experimentation, but about how they use data to make decisions. To my surprise experiments kept coming up as the one thing that everyone not only thought could be better, but hated:
Integrating into your codebase sucks - everyone we talked to has dedicated dozens, if not hundreds of engineering hours to integrate and maintain their tooling. Even more importantly it’s hard to integrate the tools into your process - terms are difficult to understand, and even more difficult to communicate. In the words of one interviewee “all the time I’ve saved with [this tool] has been put right back into making it work.”
The goal of experimentation is to understand your users better, but you end up only knowing how one variable changes. Everyone seems to have the same process. Run an experiment, get a result, spend hours trying to understand why the variable they tested changed. Usually that means digging through their data warehouse looking for patterns in other events, assuming they have data skills or a data team with bandwidth. If they don’t then it’s up to whoever can make the most plausible sounding argument.
Existing products are expensive. The issue isn’t the price, it’s that they are unpredictably expensive. I once ran a test that triggered 50x (!) the number of billable events we thought it would. Getting a bill like that doesn’t just cost you money - it slows you down by making you afraid to use a product that gives you business-critical information.
We think we can solve all these problems by adding Context (get it?).
Easily integrate with React and Segment (mobile integrations coming soon). Add a script tag to your client, use a React hook to split users into treatment groups, and add our API as a Segment destination.
Run an experiment from the dashboard. Get results for the primary variable you track and for all events across your entire application. If the goal is to understand how user behavior changes, it’s not enough to know that a user clicked “add item” more often. You should know that “completed order” went down and “average order size” went up. If you track an event in Segment, you’ll automatically see changes caused by every experiment.
That’s it! If you’re on a team that’s building with React and Segment you can have contextually aware experiments ready for production on your lunch break. Sound too good to be true? Well… we haven’t built it yet, but we’re very excited about getting started. Stay tuned for updates on how we’re adding Context to experimentation.