Video: Data-driven testing gives startups ability to fail and succeed fasterAmy J. Born | October 13, 2021
Thomas Edison said, “I haven’t failed, I’ve just found ten thousand ways that don’t work.” Today’s tech culture has embraced the concept of failure as essential and added the element of speed. Failing fast is a trademark of Elon Musk, and Mark Zuckerberg has described this mindset as "move fast and break things."
To determine whether this philosophy results in success for new companies, Duke University researchers studied more than 35,000 high-tech startups founded between 2008 and 2013 and found that, on average, startups using A/B testing — randomized controlled trials to test product feature options — are more successful than those that don’t use a data-driven approach.
"To be honest, most startups and most small businesses have a pretty standard framework for making decisions, which is 'trust your gut,'" said strategy professor and researcher Sharique Hasan. "Maybe collect a little bit of data. But usually, entrepreneurs have a belief about something that [their] startup should do, and they pursue it and they follow through."
He added that more quantitative decision-making is possible with A/B testing, and that speeds up the discovery of what works and what doesn’t. "If an idea is getting traction, you can scale it faster, and if it's not going anywhere, you can kill it faster," he said.
The researchers determined whether and how testing helped start-ups scale products and grow themselves by examining the characteristics, web metrics and product innovations of the companies in their study that did and did not use A/B testing. Performance was measured by page views, which improved 30% to 50% after a year of A/B testing use. Those companies that used A/B testing also introduced new products at a higher rate (9% to 18%) than those that did not experiment, the research showed.
Despite a common idea that A/B testing is only effective for small-scale questions or incremental changes, that A/B testing was observed to be useful in guiding more significant decisions in connection with a company’s overall strategy. In fact, the chances of success for those decisions increased with testing. With such clear benefits, the researchers were surprised to learn that so few firms (less than one-fifth) used A/B testing. Firms in Silicon Valley used it about a quarter of the time, while only 19% of those outside the San Francisco Bay Area used it. Firms with 10 or more employees used it more often than smaller firms.
Small, early-stage startups with no financing were among the least likely to use testing even though the data showed that such companies could see the greatest benefit.
"This kind of supports this idea that decision-making may be an important bottleneck for startup success, and that using a more scientific, rigorous approach to start decision-making can really improve quality of a startup's performance," Hasan said.