Defining experiments
Most of our products are always innovating and building things customers are asking for. This doc isn’t talking about those kinds of innovations we do every day.
For the purposes of this document, an experiment is a new product that is so different from what we do now it doesn’t naturally fit in one of our existing buckets. It’s a little bit of if it looks, smells and quacks like a duck, it’s probably a duck.
I’ve included little accents by Justina, who has led multiple experiments that have succeeded and failed.
How we choose
We look for experiments that:
- have a big market – needs to be able to impact our metrics, we want billion dollar opportunities if brand new or appeal to a wide spectrum of our customers. If a new product/business line, show me how this can be a 100m valuation company. If an add-on for customers, show me how this will benefit >30% of our customer base. If this seems hard, that’s on purpose – we have limited time and should maximise ROI on our time
- drive towards our goals/metrics, e.g. helps drive TPV by increasing engagement, increase product suite to deepen hooks
Experiment construct
Goal: must achieve 7% wow growth or 30% mom growth (+ bespoke metrics depending on line of business). In our case this typically looks like getting a few customers to buy before we build and then finding more and more to continue to justify investment into the idea.
Team:
- 2-5 ppl, typically 1 technical, 1 product/sales (start with sales)
- Must be willing to work 7 days a week until project succeeds or till I cancel (typically 3 months). I’ve found that within our org it takes this amount of sheer willpower and hustle to make something succeed from nothing. This also self-selects to see if you actually believe in the idea enough to devote many hours a week to it.
These guidelines are roughly based on YC’s flagship program
Each member of the team has to be willing to learn new things and adapt to make the project succeed, even if those things aren’t within their current skill set. Also, you need to be humble enough to take on any and all tasks required of you, even the really small and menial ones (and there’ll be a lot of these given how small the team is)
Justina (one of our original experimenters)
Time constrained to 3 months – longer if Moses approves (typically where regulation is a precursor). After 3 months, we look at the project and see if you’ve hit targets or why you haven’t. If you hit targets, we assign more resources ($, devs, ppl) to it. If not, we axe the project gracefully, e.g. pivot it into something else or let it slowly die. We often maintain experiment projects that have sold even after the project so make sure you’re so committed to an idea, even if it dies, you’re happy to keep serving the <5 customers who are still using it.
