One of the most important things I learned from running a startup is that on a macro scale the innovation market is efficient. If the market conditions allow for a startup to arise, it’s overwhelmingly probable that multiple startups already exist in that market. The converse is also true — if there are no startups in a given market, it’s overwhelmingly probable that market conditions are not hospitable and startups cannot arise.

In this sense startups are similar to biological life. Wherever the conditions are hospitable, life already exists. The difference is that startups live in an economic rather than a biological environment.

This has massive implications for picking startup ideas because technology companies tend to be winner-takes-all. The first technology company to solve a problem is often worth much more than all of its competitors combined. So if a company already exists in a market, it’s overwhelmingly likely you won’t be able to displace it.

Conventional wisdom states that startup founders should work on problems they or people they know have, but as a filter for startup ideas this model is far too permissive. Market efficiency implies that if a problem is valuable and solvable, it’s overwhelmingly likely that it already would have been solved. Conversely, if it isn’t solved, there must be structural reasons for why solving it isn’t possible.

Encouraging founders to work on these sorts of problems is a winning strategy for investors because the law of large numbers is on their side. It’s OK for most startups in their portfolio to be overwhelmingly unlikely to succeed. So long as the portfolio is large enough for one or two winner-takes-all companies to succeed in large markets, a fund will have good returns.

However, this is a terrible strategy for founders, because the law of large numbers isn’t on their side. A startup founder doesn’t get enough shots to get a good expected value if they simply look for valuable problems. So a winning founder strategy must include a much tighter filter for startup ideas.

If you can’t displace an existing technology company, and it’s overwhelmingly probable that the market environment prevents any given unsolved problem from being solved, how do successful new startups arise?

Paradoxically, the solution is to stop thinking of startup ideas in terms of solving problems in the early stages of startup germination, and instead start thinking in terms of changes in the economic environment and the opportunities these changes unlock.

Much like the biological environment, the economic environment isn’t static. Society’s moral compass changes over time. Political environments adapt to social norms and nation-states adopt more liberal laws. People in developing countries get richer. Small quantitative advances in technology adoption cross a threshold of critical mass and turn into qualitative changes.

Changes in the environment unlock startup opportunities. And since technology companies are often winner-takes-all, recognizing an environmental change first and getting to market with a viable solution dramatically increases the probability of long-term market dominance.

In the early stages ask yourself — how is the world changing and what can I do about it? Only once you identify an environmental change and make sure you’re not late to market, should you start thinking about your company in terms of solutions to problems.

Competition

Looking at environmental changes to generate and evaluate startup ideas is a good first pass, but it isn’t sufficient. By the time you discover a change, many other people will have discovered it too. For example, when a journalist publishes an article about a trend, they will have gotten their information from multiple existing companies working in that market.

When you’re first to market with a viable product, your customers will compare you to the status quo. You’ll snap up early adopters — folks who are on the bleeding edge of the trend, and if your product is good, the case for more conservative customers will get stronger and stronger over time.

When you’re second to market with a viable product, your customers will compare you to the company and product that got to market first. If that happens, your job will be orders of magnitude more difficult. It’s no longer sufficient to capitalize on the trend. You have to find a strong point of differentiation with the first company, and empirically this is dramatically more difficult than founders think.

First mover advantage is exacerbated by the fact that technology companies enjoy compounded growth. Building organizations takes time. Incorporating, fundraising, recruiting, and shipping and marketing a product can easily add up to many months of work. If you’re second to market, your competitor will have been scaling their user acquisition machine while you’re still scrambling to ship your product. They’ll acquire resources at an exponential rate that allow them to reach even more customers before you do, and you’ll be stuck trying to explain to users how your product is different from theirs. Most of the time, that means the death of your company.

Advantage

So you need to get to market with a viable product before anyone else does. Empirically, you can’t do that by out-executing existing companies. Even in the unlikely event that most of your competitors are incompetent, just one capable competitor will dramatically reduce your chances of success. You need to find a trend before others do, and get started while the market is wide open.

There are no formulas to learning about changes early, but you can put yourself in a position to get unique insight. For example, you can learn about changes early because of your job or hobbies. Speech recognition researchers probably knew that speech recognition was getting “good enough” a few years before I did, because I only learned about it when working speech recognition products started hitting the market.

Sometimes your unique background can make you aware of changes that other people can’t see. A plausible story about why most of the population is blind to something, that’s a reasonable edge. For example, maybe you think that people who haven’t lived in the third world don’t understand how important remittances are, or that the third world is just now getting sufficiently networked that you can do them digitally.

You might also be in a position to exploit changes because of specific advantages that are unique to you. Some changes can be exploited by just about anyone; other changes might require deep technical knowledge or a huge amount of effort that most entrepreneurs aren’t willing to put in. Self-driving cars might be an example of this: everyone was talking about how they were just on the edge of possible for years, but only Google managed to put together the expertise, capital, and determination to get to the last mile.

There are other ways to get an edge. For example, if you have the resources of a large company at your disposal, you might be late to market and still win over your competitors by having a larger marketing budget or existing distribution infrastructure. But this is rare in a startup world. Startups usually don’t win because of superior resources — they win because they recognize a trend and ship a viable product first.

Story

Thinking about your startup in terms of changes in the world doesn’t just make your startup dramatically more likely to succeed because it’s a better model for technology innovation. It has other surprising benefits.

In the initial stages of a startup you have to talk to a lot of people — potential users, advisors, investors, industry experts, early employees, and eventually journalists. The first question people ask is “what does your startup do?” So communicating what you do is the first tactical problem you have to solve.

Human beings are wired to respond to storytelling, and all stories have the same fundamental structure. Here is the simplest possible framework to tell a story, from Pixar’s rules of storytelling:

Once upon a time there was ___. Every day, ___. One day ___. Because of that, ___. Because of that, ___. Until finally ___.

The first part is the setup. The world used to work in a particular way, and here is what it meant for people’s daily lives. The next part is the change in the world. These parts are only tangentially related to your startup — the environmental change you’re describing should be inevitable regardless of whether you start your company. The market is efficient, remember? If you don’t start your company to respond to the change, many other teams will. Your goal is to get to market with a viable product first, which more often than not will help you establish market dominance.

Once you describe the change in the world, only then do you talk about your product and company. The change is occurring independent of your actions. You just happen to notice it first.

The last part shows how your company will transform people’s lives. When your startup grows large enough to own the majority of the market, you will be the cause of the next qualitative environmental change. In this way your startup will complete the circle of technology innovation.

This is how the best companies tell their stories. People buy cars driven by internal combustion engines. But battery technology has gotten cheap enough that we can build a pure electric vehicle. It’s still fairly expensive, so we’ll start with luxury vehicles. As battery technology gets even cheaper, we’ll go down market. Until finally, we’ll transform the auto industry so that every car will be an electric car.

This type of story telling is incredibly effective at propelling your startup. Early adopters, employees and advisors will be more likely to sign on because you’re inviting them on a journey. Investors will be more likely to respond because this matches their model of the world. And journalists will cover you because you’re essentially doing their job for them.

A good story gives you tremendous descriptive power to explain what you do to different parties in a consistent way. But even more importantly a good story is prescriptive — it informs decisions and forces you to make disciplined choices. If you’re about to make a decision that doesn’t fit your story, you have to explicitly choose whether to reevaluate the decision or to refine your story. Early stage startups get pulled in a thousand different directions. They’re much more likely to get killed by compromises than by committing to a wrong path. A good story keeps you honest, prevents you from trying to please everybody, and gives you the mental clarity to go in a concrete direction.

Exceptions

The notions that innovation markets are efficient and that first movers are overwhelmingly difficult to displace are controversial because it’s easy to think of counterexamples.

Beats Headphones broke every rule in this article and still succeeded because fashion companies are inherently fickle and unstable. Microsoft Surface shipped a poorly differentiated product and succeeded in a fairly mature market because of a huge marketing budget. Many opportunities remain undiscovered for years and many deeply technical problems remain open for decades until finally someone abruptly snaps them up. Even seemingly unstoppable hyper-growth companies make critical mistakes and effectively commit suicide.

In the startup world these cases are the exception rather than the rule. Most founders can’t swing a partnership with a successful hip-hop artist and can’t execute a billion dollar market campaign on the cheap (but think they can). Startups tend to feel exceptional because you’re going off the beaten path, but within the startup world you’re less exceptional than you think.

If you’re convinced you’ve found an exception, be careful. Most startups fail at inception and take years to find out because the founders didn’t understand efficient markets and first mover advantage. So think about this model deeply, and understand it first. Exceptions do exist, but make sure you have a deep understanding of what you’re doing. You need to learn the rules before you can invent your own.

Self-deception

One caveat to thinking of startup ideas in terms of environmental changes is that it’s so effective that it’s easy to deceive yourself with early success. Smart and sufficiently determined entrepreneurs can find evidence for environmental changes to support almost any startup idea. If you’re committed to building a food delivery service, you can always invent a hypothesis, weave a narrative about your product around it, find credible supporting data, and polish your pitch until a large number of people believe it. Unfortunately you will have created very compelling fiction.

This isn’t unlike the process of undermining the scientific method by believing in your hypothesis so strongly that you start to unconsciously reject discouraging evidence. Confirmation bias is deadly in science, and it’s equally deadly in business.

It’s relatively easy to attract a surprising amount of capital, talent, and press interest to your company this way, which makes the technique extremely dangerous. I’ve met people who burned through millions of dollars of other people’s money and wasted hundreds of man-years on projects because they’d inadvertently created very compelling fictions and were sufficiently taken by early success to believe in their own fiction themselves.

So it isn’t just important for the story to be compelling. It’s also extremely important for the story to be right.

Evaluating the correctness of a story is an extremely challenging art that lies at the intersection of many disciplines — economics, science fiction, history, anthropology, and sociology. You’ll need to think deeply about the world and invest years of concerted effort into building a good intuition for thinking about the future. Even then you’ll be right only a fraction of the time. But you can still get much better than random chance.

Intuition

Here are some things I’ve read that have helped my think about the future. I use them to learn how to put together stories, and to hone my intuition on how likely these stories are to be correct:

Reading The Economist for a few months is a great way to get on the same page with conventional wisdom. The Economist writers are very good at thinking about the future (but keep in mind that they have no strong pressure to be correct).
Peter Thiel’s book Zero to One is excellent. Most business books are terrible, but this one is the exception.
Reading Harry Potter and the Methods of Rationality is very instructive. What Harry spends the first eleven chapters doing is exactly what you want to be teaching yourself. “The rules are different, there is magic in the world. What happens now? Can I get rich from galleon/sickle arbitrage?”
Similarly, I browse the TV Tropes wiki from time to time to help me deconstruct my own fiction.
William Gibson’s essays in Distrust That Particular Flavor helped me think deeply about technological innovation in the real world.
Similarly Five Things We Need to Know About Technological Change by Neil Postman has a unique way of thinking about technology. He’s bearish on technological innovation, but still has a uniquely useful way of thinking about how technology propagates.

Once you get through these links, start going deeper.

Learn as much as you can about microeconomics to get a better model for how the world works. Not just the basics of supply and demand, but more advanced topics like complementary and substitute goods, elasticity, marginal utility, etc.

Study history. Don’t just learn dates and names, but ask deep questions. Why did the Ottoman Empire never develop feudalism despite being in similar economic and technological state of development as European nation-states? Why didn’t the inventors of the steam engine capture most of the value they created? Why didn’t the Chinese civilization end up conquering Europe during its golden ages, when it was far more developed than European nation-states?

Get a rudimentary understanding of anthropology and ethnography. Understand emic and etic approaches to cultural evaluation and learn to think deeply of cultural change.

Cover the basics, and start branching out on your own. Look carefully, and you’ll find lessons in the most surprising places. As you think deeply about the world, start acting. The best way to get an intuition for how to make good decisions is to make actual decisions and observe results, so don’t get wrapped up in analysis paralysis. Start a company, and start learning from your own mistakes.

Thanks to Michael Lucy for reviewing and contributing to this post.