Make Scrappy Experiments
"The real measure of success is the number of experiments that can be crowded into 24 hours."
- Thomas Edison
Marc Randolf (co-founder of Netflix) described the early days of new venture creation in a fascinating conversation with Tim Ferriss: “I had tons of ideas. The problem is that back then I was a bit of a perfectionist. And so the tests we put together would be these mini works of art: We’d do custom photography, or we would lovingly craft and argue all the copy. We would spell check it and have copy editors review it. And we would stress test the site and check every link. And as you can imagine, that would take two weeks to put together. And then the test would fail. And we’d look at each other and say, ‘We just wasted two weeks.’ And we’d say, ‘Okay, faster.’ And we’d cut some corners and do a test a week. And it would fail. And then we’d cut some more corners and we’d begin to do a test every other day. And then pretty soon a test every day. And soon we were doing four and five tests in the same day.”
Every innovator should know how to craft clever experiments; but one clever experiment is hardly sufficient. One of the things that both Edison and Randolf teach us is that quantity matters. And paradoxically, you can get to quantity in two ways: by dedicating more resources to experimentation at large, or by dedicating less resources to each individual experiment.
As you might imagine, I’m a big fan of the latter. I’m a firm believer that there’s always a way make each experiment cheaper. You’ve just got to get scrappy.
A variation on this theme is to make experiments slightly less accurate (statistically speaking), which requires significantly less resources than experiments that are geared towards statistical significance. This thread by the fantastic Wharton professor Ethan Mollick cites a fascinating piece of research conducted by Wharton’s own Eduardo M. Azevedo and NYU’s Jose Montiel Olea, alongside a team of folks from Microsoft. As Mollick recounts, they discovered that, in many scenarios, the better strategy is to “run more small (sample size) & (thus) less accurate tests.”
When you’re wondering how to reduce the cost of an experiment, ask yourself, “How would a scrappy start-up try to create this data, but at a fraction of the cost?”
Related: Make Experiments Cheaper
Related: Judge Experiments Before Ideas
Join over 11,147 creators & leaders who read Paint & Pipette each week
The fine folks over at Section recently interviewed me about some common mistakes I’m seeing folks make when they use AI. One of the biggest ones? The tendency to expect too little, and get just that.