If intelligence were all that users cared about, price competition should have closed this gap by now. It hasn't.

Open source's share of token volume on OpenRouter, the leading LLM aggregator, has been stable at 30% for some time, despite all the shifts in the broader market. They even go so far as to call this an "equilibrium."

Standard economic intuition says that, at the same quality level, the cheaper option should win. The LLM market seems to violate this. Something else must be going on.

This puzzle has a precise analogue in economic theory – latent demand. And rather than AI researchers, it's economists and finance researchers who have the most to say about it.

Priceless

A group of economists from MIT, Boston University, and Microsoft recently published an empirical study of LLM market dynamics, what they call "The Emerging Market for Intelligence.”

The irony of that title: the paper's results suggest consumers of LLM APIs care about far more than mere intelligence:

… open-source models are approximately 90% cheaper than closed-source models, conditional on the same level of intelligence… Nonetheless, the share of tokens consumed from open-source models remains consistently below 30%, suggesting meaningful differentiation between open and closed-source models not captured by intelligence measures – The Emerging Market for Intelligence: Pricing, Supply, and Demand for LLMs

Intelligence only explains so much. You can see it visually here, where open source models at any given level of intelligence are consistently cheaper:

Not just that, but there's incredibly wide dispersion in prices at every level of intelligence. In fact, prices vary by up to two orders of magnitude, and models that are much more expensive relative to their benchmark performance still see substantial use.

That dispersion of pricing has only gotten more severe over time despite prices falling overall. The price-to-intelligence ratio, which you can think of as representing a measure of quality-adjusted token pricing, has become more dispersed over time:

More price puzzles: the average price actually paid for tokens (as opposed to simply the average or median list price) increased over the course of 2025. This seems to have been completely missed in the discourse last year and is strong evidence that price is not a primary motivating factor for many users:

A reasonable hypothesis is that the observed run up in token-weighted prices for output tokens was driven by the release of the Claude 4 models in May 2025. This occurred during a period where, per the prior chart, intelligence-adjusted prices were falling through the ground. This is a powerful argument for revealed preference of users, who migrated toward premium models to pay for something benchmarks don’t fully capture.

The authors conduct a series of regression analyses across about 150 model-provider combinations to further dig into the drivers of pricing across models, controlling for intelligence, being open source, reasoning capabilities, context length, and other factors. Immediately we can see a few interesting phenomena:

• Intelligence alone doesn't determine pricing: the of a simple regression of token price on benchmarked intelligence is only 10%
• Age is just a number: adding in controls for the age of a given model doesn't explain much variation in pricing, only increasing the by ~4% over and above the baseline controls. Surprisingly, there's no guaranteed obsolescence here
• Brand matters, even more so than intelligence: Adding in controls for the model creator bumps up by 13%, even after including many other controls, explaining more pricing variance than intelligence alone

In total, intelligence clearly matters but it's far from being the only relevant pricing factor.

Volume control

That's pricing, what about quantities? Here too we find a complex picture with no simple answers.

Again, intelligence is not destiny – LLM customers are making decisions based on many factors. If you weight intelligence over time by the actual token usage of those models on OpenRouter, what you see is that typical usage concentrates far from the frontier of intelligence. The median user isn't using anything close to the smartest model, and even the 90% percentile token is coming from a model that is well below the best available:

Some more regressions, this time with token volume as the dependent variable.

• Pricing matters but only so much: Prices have a statistically significant relationship with token demand, but within a given model, prices explain only 18% of the variance in token volumes. The remaining 82% is driven by other factors
• Price competition is zero-sum: The price elasticity of demand is almost exactly one, meaning a 10% price cut generates roughly 10% more volume, leaving revenue (price times quantity) unchanged. To grow revenue, providers must shift the demand curve itself through alternative means
• HODL the model: Within a given model/provider combination (rightmost column), price is again statistically significant but explains almost nothing (within of 5%). Most of the action is in the discrete choice of which provider to use in the first place, which is driven largely by factors other than price

We saw a drastic example of this phenomenon during the release of Grok Code Fast 1, xAI’s extremely fast and cheap coding model, with heavily subsidized pricing (even as low as free). Grok Code Fast 1’s release had zero impact on the demand for other models, suggesting more nuanced user preferences not easily swayed by cost:

The pattern on the quantity side mirrors what we saw with pricing. Intelligence alone explains little price variation; price alone explains little quantity variation. In both cases, the observable characteristics we fixate on (benchmark scores, token prices) leave most of the picture unexplained.

Users don’t always pick the smartest model, and they aren't purely chasing the lowest price. Something else is guiding their choices.

Latent space demand

The asset pricing literature has developed precise tools for exactly this kind of problem. In “A Demand System Approach to Asset Pricing” Koijen and Yogo construct and estimate a “demand system” for public equities. They decompose demand for a given stock into two components, demand driven by observable characteristics (price, profitability, book equity, dividends, etc.) and demand driven by what they call latent demand -- investor preferences for attributes we cannot observe.

We refer to as latent demand, which captures investor 's demand for unobserved (by the econometrician) characteristics of asset – A Demand System Approach to Asset Pricing

When they apply this framework to the data on U.S. stock returns, they find something striking -- observable characteristics on both the supply and demand side only explain ~20% of the cross-sectional variance of returns. The remaining 80% is dark matter -- driven by unobservable, latent demand among investors:

Thus, stock returns are mostly explained by demand shocks that are unrelated to changes in observed characteristics. This finding is consistent with the fact that cross-sectional regressions of stock returns on characteristics have low explanatory power

Koijen and Yogo are careful to point out that they cannot directly identify what latent demand contains (by definition); they can only quantify how much it matters. Some of their hypothesized drivers: structural constraints (certain investors cannot invest in certain companies), heterogenous beliefs (a fancy way of saying that investors “agree to disagree”), and herding behavior (e.g. meme stocks).

So in the most liquid, most efficient, most studied market in the world, unmeasured factors dominate measured ones by 4:1. How much of AI demand could we realistically hope to explain?

Not much. A non-exhaustive list of factors that could drive one’s choice of LLM:

• Reliability and uptime guarantees (you know who you are)
• Safety policy and content moderation posture (you also know who you are)
• Tooling and ecosystem integration
• Support quality and SLAs
• Privacy and data retention policies
• Brand and reputational risk

These are dimensions that benchmarks like MMLU and SWE-Bench don’t capture. The ~10x price premium for equally intelligent closed source models hints that these unobserved factors really do matter. It's the LLM equivalent of Koijen and Yogo's finding: latent demand is the rule rather than the exception.

The AI economists acknowledge this:

There are two potential explanations for this gap. First, the Intelligence Index may not fully capture important differences in model capabilities. Second, users may systematically discount the value of open-source models—either because of actual or perceived differences such as weaker brand reputation, privacy concerns, or less extensive customer support – The Emerging Market for Intelligence: Pricing, Supply, and Demand for LLMs

Both explanations point to the same conclusion: latent demand is most of the demand.

In fact, the price regressions we looked at earlier show exactly that: intelligence only explains 10% of price variation across LLM providers. Adding in other characteristics like whether the model is open source, reasoning capabilities, context length, and age only gets you to ~45%.

Only by controlling for the model creator (e.g. Anthropic, OpenAI, DeepSeek, etc) are we able to explain more than half of LLM pricing variation. Further, controlling for whether the model is open source muddies the water a bit, since most model labs are either fully open or fully closed source, meaning some of what open source is “explaining” is actually a brand effect.

In total, this data suggests that the identity of the model creator is the biggest single explanatory factor in LLM pricing.

If it fits, it sticks

One way to explore this brand effect is to observe the degree to which users stubbornly “stick” to a particular model, i.e. the degree to which they maintain strong preferences irrespective of price or evolving alternative options.

Here again we can analogize to the stock market. In the decomposition of stock returns from before, the researchers broke latent demand into two parts: the “extensive margin” (certain investors only buying certain stocks) and the “intensive margin” (how much of each stock investors hold, assuming they hold the stock at all).

Applying this framework to the OpenRouter data, we see substantial evidence for both the extensive and intensive margins playing a role. For example, users concentrating all of their usage on a single model aligns with the extensive margin, while users using many models but varying in how they distribute their use cases between them points to the intensive margin.

The data? Just over 50% of OpenRouter users use only one model. Among those that use multiple models, a very large share of users allocate almost all their usage to only one of those models:

To quote the study:

Our interpretation of these patterns is that, for most firms, multi-homing reflects experimentation rather than intensive dual use of models tailored to specific tasks – The Emerging Market for Intelligence: Pricing, Supply, and Demand for LLMs

That is to say, to the extent users leverage multiple models, it’s mostly for experimental purposes rather than production use. When it’s time to get serious, most users make a singular choice and stick with it. The extensive margin dominates.

OpenRouter themselves recently published their own analysis which came to a similar conclusion, which they call the “Glass Slipper effect” in a nod to Cinderella:

The hypothesis posits that in a rapidly evolving AI ecosystem, there exists a latent distribution of high-value workloads that remain unsolved across successive model generations. Each new frontier model is effectively “tried on” against these open problems. When a newly released model happens to match a previously unmet technical and economic constraint, it achieves the precise fit — the metaphorical “glass slipper.”

For the developers or organizations whose workloads finally “fit,” this alignment creates strong lock-in effects. Their systems, data pipelines, and user experiences become anchored to the model that solved their problem first – State of AI: An Empirical 100 Trillion Token Study with OpenRouter

In the LLM market, it’s a competition to be someone’s #1 pick, and there’s no prize for 2nd place.

Genius is not homogenous

The LLM market presents a puzzle: massive price dispersion at equivalent measured intelligence, persistent market share gaps between closed and open source models despite premium pricing, concentrated usage patterns at the user level despite a panoply of options.

Old school asset pricing models failed because they assumed homogenous investor preferences. In the famous CAPM model, everyone holds (or at least should hold) the market portfolio. In reality, investors have wildly divergent preferences that show up as latent demand.

Similarly, the obsession with model benchmarks assumes homogenous user preference for intelligence over all else. But in fact users have varying preferences that benchmarks don’t and can’t reflect.

So what are the implications? Here’s a crack at a few:

• Models aren't commodities (yet): It seems many users are willing to pay a premium for quality, capabilities, or stability. Differentiation is possible, and the inordinate influence of latent demand reflects that
• Moats aren't about weights: Defensibility may be less about locking down model weights and more about trust, safety posture, and other intangibles
• Benchmarks measure what's measurable: We’re measuring what’s easy to measure, not always what matters. Most of what drives relative LLM adoption lives in dimensions that are hard to quantify. Benchmark-maxxing isn’t everything.

The benchmark obsession is understandable. Numbers are legible. Rankings are simple.

But the market is telling us something different: tokens aren’t fungible. Despite convergence in capabilities across models, in the eyes of the market, “all models are equal, but some models are more equal than others.”

• They are too stable over time for such a speculative asset
• They are impervious to shifting interest rates
• They don’t follow public tech valuations

In short, seed valuations are a bit of an enigma — it’s not at all obvious what drives them. However investors arrive at these numbers, they aren’t doing so based on typical Finance 101 factors like discount rates or comparable company analysis.

Seed companies don’t seem to be priced as businesses with intrinsic value derived from future cash flows. Rather than venture capital, they seem to be a proxy for the human capital of the founders and early team.

Seed valuations aren’t valuations.

Non-standard deviations

First: seed valuations aren’t volatile enough.

This really stood out when constructing my Venture Activity Index – the volatility of the late stage is much higher than the early stage, which is true even if you exclude the pandemic era:

This is strange. If seed stage investments are as speculative as they’re purported to be, we'd expect wild valuation fluctuations over time. Late-stage valuations should be more stable, since they have an existing business model, revenues, and potentially even cash flows (hard to believe, I know).

Imagine if large, blue chip stocks were more volatile than small caps. That wouldn't make much sense. The stability of seed valuations relative to more mature startups makes no sense either.

As I’ve previously covered, capital flows affect valuations, and this force is most extreme at the later stages where the so-called "crossover funds" (funds who've crossed over from public market to private market investing) are most active. Thus, capital flows drive excess volatility at the later stages. However, I don't think that can explain everything going on here.

And yes, there are many more seed deals getting done each quarter than later-stage rounds, so due to some fundamental laws of statistics we’d expect less variability with a larger sample. Even so, seed valuations seem too stable on a relative basis.

Discount rates don’t matter if there’s nothing to discount

Second: interest rates don’t affect seed valuations.

In the standard corporate finance logic, early stage companies should be the most sensitive to interest rates because they have the highest duration. Think of duration as a measure of distance, measured in years, to the average dollar of cash flow for a business. For early stage companies, all cash flow the business will ever produce is far into the future, so they definitionally have high duration. In a discounted cash flow model, seed valuations would be extremely sensitive to movements in the discount rate, similar to a long-dated bond:

Duration measures the sensitivity of the value of a bond to a change in interest rates, which is tied to the lifetime of the bond. Bonds with longer tenure or back-loaded cash flows are more sensitive to changes in interest rates

Due to the multiplicative nature of discounting, the present value of far-away payments is more sensitive to a change in interest rates than the value of soon-to-come payments – High Retention = High Volatility

That's not what we see. Some time ago I investigated the “average” impact of interest rate surprises on venture valuations across all stages. I've since revisited and refined that analysis, adding more granularity to the estimates. I've also stripped out the COVID era, where the impact of fund flows contaminates the estimates.

What I found was fascinating. Here's the impact of a surprise 1% increase in the one-year Treasury yield across seed, Series A / B, and Series C / D, along with a measure of uncertainty around these estimates, going twelve quarters out:

• Interest rates have no effect on seed valuations. Not only is the impact zero, it’s precisely zero. There isn't a ton of uncertainty around these estimates.
• For Series A / B, we start to see some effect, maxing out at a 16% decline seven quarters after impact before recovering.
• For proper growth stage rounds like Series C / D, the impact is even quicker and more severe, peaking at 19% six quarters out.

This is the most striking evidence — the clearest indication that investors value seed stage startups in a totally distinct way to the rest of the venture market. It’s hard to rationalize this evidence within the usual frameworks.

They not like us

Third: public tech valuations don’t influence seed valuations.

When the big tech (“FANG”, the “Magnificent 7, etc) valuations move around, private valuations typically follow, with a lag. This makes intuitive sense given venture investors use comparable public company valuations to decide how much they’re willing to pay for private companies. In a rational market we’d expect some correlation between public and private valuations, even if imperfect. This is “comps analysis” in a nutshell.

And that’s what we see — except for seed companies:

When the Nasdaq rises 1%:

• Series A through D valuations rise, matching the bump in the Nasdaq within about a year.
• Seed valuations don’t move at all, marching to the beat of their own, relatively quiet drummer

While there’s a very clear pass-through effect of public prices on private valuations for most stages, seed startups are the exception, seeing no pass-through at all. It turns out, investors don’t care about public comps when pricing seed stage companies.

It’s clear that seed valuations are not really valuations in the traditional sense. They don't behave like valuations in either their volatility over time or their sensitivity to interest rates. Something else must be going on here.

These prices are sticky too

It struck me that seed valuations were incredibly sticky (an early working title for this essay was "Why are Seed Valuations so Sticky?”). Seed valuations neither rise nor fall dramatically far from trend, whereas other stages see much strong gyrations:

This "stickiness" is unique to seed valuations and doesn't mirror the behavior of free-floating financial assets, which are typically much more volatile and difficult to forecast.

I stared at the seed valuation data for a long time as I contemplated this essay. As my eyes glazed over, I tried to come up with analogues, other phenomena that mimic the behavior of seed valuations.

Then it hit me – wages. Wages are often said to be “sticky”, and seed stage valuations look eerily similar to wages over time, which also tend to be quite stable around a long-term, upward trend.

The simplest thing to do is plot compensation against seed stage valuations. I found a wage series called the “Employment Cost Index” (ECI) that measures compensation growth over time, and I pulled out a version of this that’s specific to “professional, scientific, and technical services,” which I take to be a good proxy for tech workers:

• The first thing that stands out is the obvious difference in growth rates. Seed stage valuations have risen much faster than wages for the typical tech worker. For every 1% increase in tech worker wages, seed stage valuations grow 4-5%.
• We can predict seed stage valuations from the wage data. I regress seed valuations on the ECI using pre-2020 data. The fit is tight pre-2020 (which is frankly easy since they’re both roughly straight lines).
• Then I evaluate its forecasts on post-2020 data, “out of sample”. The model returns to a reasonably close fit once valuations settle down after the 2021-2022 bonanza.
• Notably, seed valuations bottomed out at exactly the level you would have predicted using the employment cost index.

This feels like more than coincidence. Regressions do have a high risk of being spurious, which is to say, total nonsense. I was skeptical the first time I ran these numbers. But after multiple sanity checks, this seems to be the real deal. There is a tight connection between seed valuations and wages for tech workers; the two follow the same trend, one an accelerated version of the other. Thus we have a scaling law between tech wages and seed valuations:

The inverse of stock-based compensation, seed companies appear to be compensation-based stocks, at least in how they’re valued.

Google is my BATNA

This analysis doesn't prove anything, but it does suggest an interesting link between tech wages and seed valuations.

Why would seed valuations be linked to the cost of tech labor? And why would those valuations grow so much faster than wages? I’m not even 100% convinced that it’s a direct, causal connection — perhaps there’s some third variable that drives both. Totally plausible. I plan to explore this in a future piece.

Regardless, it’s quite clear to me after crunching the numbers that seed valuations don’t behave anything like valuations, at least not valuations of companies or speculative assets. Their stability suggests investors have a precise sense of their worth, despite their riskiness. The value investors place on these companies does not fluctuate wildly over time.

This is odd only if your mental model values these fledging enterprises as… enterprises. What if seed stage valuations instead represent the value of the labor and human capital of founders and early employees? The behavior of seed valuations would make a lot more sense if we saw them as proxies for the opportunity cost of talented tech workers.

I’ll explore this hypothesis in a future essay.

Imagine a standardized test that works like this:

• The questions and answers are freely available on the public internet
• Cheating is not regulated, it’s a 100% honor system
• The exam never changes, it’s the same questions every time
• Scores on the exam seem to be getting better every year

In case you’re wondering, the correct reaction is “this is not a very good exam.” Cheating is rampant. The answers can all be memorized. The test isn’t really much of a “test.” The scores are meaningless.

This is the current state of AI benchmarking. Ironically, we hold AI benchmarks to a very low standard. We allow practices that would never fly in other serious domains. It’s happening in plain sight, with a nod and a wink every time an AI lab releases a new model that tops the leaderboards.

We must do better.

Something’s in the water

The first dirty secret of AI benchmarks is quite literally, dirty: training data contamination.

A common refrain among machine learning researchers and engineers — “don’t train on the test set.” In other words, don’t train a model on the same dataset that you will evaluate it on. If you do, whether on purpose or not, that particular test is useless for evaluating model performance.

Most open benchmarks are available on the public internet. In other words, the entire set of questions and answers exists on the open web, likely in multiple places. It is extremely easy for this data to inadvertently make its way into LLM training pipelines. The big AI labs are constantly hoovering up data from the internet via web crawlers, scraping their way across the web. While diligent researchers are careful about what ends up in the training data, applying all sorts of sophisticated filters to the raw data before feeding it to their models, invariably these filters are imperfect, and data from the benchmarks leaks in. It can happen during pre-training, it can happen during fine-tuning, either way, the bad data gets in. This renders invalid the downstream model performance on the benchmark.

The test data from a particular benchmark can appear in many places. It could be in plaintext on a webpage somewhere. It could be a file in a GitHub repo, alongside code which a model is being trained with. It could be in the comments of a file or script inside of a repo. It could be on Reddit or X. It could really be anywhere.

An increasingly important issue with evals is test set contamination. After the creation of a good eval, examples of the eval tend to get propagated into various places in the internet, like arxiv papers, ChatGPT examples, or reddit — Jason Wei, OpenAI

It’s been shown that a single training example can meaningfully impact the behavior of a model, so it only takes one mishap to contaminate a model. Model builders routinely train on trillions of tokens, so there’s no shortage of ways for accidents to occur.

You might wonder, if the entire benchmark makes its way into the training data, then why don’t the models score perfectly on the benchmarks? Remember, these models are complicated mathematical functions of the data — they are optimizing for all sorts of criteria that we don’t fully understand (though we’re making progress) and have been trained on vast sums of data, so any particular benchmark represents a very small share of the overall training corpus. A model trained on a particular dataset will fail to reproduce that data at times. This complex and non-deterministic mapping of inputs to outputs is arguably the distinguishing feature of LLMs relative to traditional machine learning models. So we shouldn’t necessarily expect to see perfect benchmark performance even in the most blatantly contaminated scenarios.

It’s not what you think

The next time you hear about how a new model crushes some well-established benchmark, take a moment to actually go to the primary source and review the tasks in the benchmark. Pick any of them, it doesn’t really matter which. I would bet that, even as a non-specialist, you will very quickly begin to question whether the benchmark is really testing what it claims to test.

Sometimes this is subtle; sometimes it’s not. Many AI coding benchmarks have fairly trivial solutions that don’t require meaningful code understanding or “intelligence.” Many “reasoning” benchmarks don’t require any reasoning.

Again, don’t take my word for it — I encourage you to, right now, read through a few examples from the many published benchmarks, and see what you think about the concepts they are purportedly testing. You will be less than impressed, and you will leave the exercise with a much reduced opinion of and trust in the performance of the latest and greatest models.

This also relates to my point earlier about contamination, which is especially pernicious in the software development domain. Many “brain twister” coding tasks are well-represented in the training data for these models, in part because software developers spend so much time preparing for such questions as part of job interviews. Coding competitions abound, which can serve as a signal to employers of an engineer’s talent and skills. Their competition code often ends up on the public internet, where it gets picked up by training data crawlers.

At test time, the model has already seen the question, perhaps in slightly modified form, and has memorized the answer. It’s not right to say a model regurgitating some code from some repo is “reasoning” about software development. It’s much more akin to an engineer getting stuck and looking up some code on Stack Overflow, pasting it into their editor, and moving on.

There’s nothing wrong with a “Stack Overflow bot”! I would and do pay for such things. I just think we need to be very careful and specific about what we call it.

Teaching to the test

Critics of standardized tests often complain they encourage teachers to “teach to the test”, focusing exclusively on material that will improve their students’ performance on the test but that perhaps misses other information valuable to their overall edification as human beings and citizens. I think this critique has bite to it in the context of AI models.

Here’s why. Kaggle is the most famous competition platform for machine learning practitioners. They hold frequent competitions where the goal is to build a machine learning model that can generate accurate predictions in a particular context based on provided (or otherwise procured) inputs. A key feature of Kaggle competitions is the public leaderboard — the real-time ranking of submitted models on a “validation” dataset that only Kaggle has access to. Thus, they avoid the problem of data leaking into the training set.

However, as is well-known among Kagglers, this “one weird trick” doesn’t solve all problems. After some time, the competition closes, at which point every model is re-evaluated on a previously closed “test set”, the scores for which make up the “private leaderboard”. It is extremely common for the “best” model from the public leaderboard to fall a few places in the rankings on the private leaderboard and for another model to rise to the top. In fact, this happens so often that being in 1st place going into the final evaluation can often be a source of stress rather than relief, since you know there’s some risk you’ve “overfit to the public leaderboard.”

Many users watch the public leaderboard closely, as breakthroughs in the competition are announced by score gains in the leaderboard. These jumps in turn motivate other teams working on the competition in search of those advancements. But it’s important to keep the public leaderboard in perspective. It’s very easy to overfit a model, creating something that performs very well on the public leaderboard, but very badly on the private one. This is called overfitting.

Even though your model never saw the validation data directly, you did see its score on the data. That score is a noisy proxy for the data itself, so anything you do to improve the score implicitly leverages features of validation data, even if you never saw data directly. Since that score influenced the techniques and tricks you employed during model training, there’s a sense in which the validation data did “leak” into training.

You’ve effectively “taught to the practice test”. And as anyone who’s experimented with the materials of various test prep providers, practice tests can differ meaningfully from the actual test.

The same risk applies to most AI benchmarks, which are the “public leaderboard” of AI evaluation. At any point during the training process, researchers can check their models against the benchmark to see how it’s doing and ensure that performance is moving in the right direction. Benchmark performance isn’t a “surprise” at the end of a training run. It’s the same with Kaggle contests — contestants can always check their performance via the public leaderboard. This creates perverse incentives to only do things that drive direct improvement on the benchmarks, leading to overfitting, the curse of all machine learning practitioners.

Again, if you want some intuition, think about the education context. Imagine if teachers knew in real-time how their students would perform on the test — a score every day of class rather than one a year. Whatever concerns you have about teaching to the test, you should be only more concerned if teachers had a real-time view of their students’ test performance. The dopamine rush of seeing those scores bob up and down every day would drive an even more myopic focus on test performance. Anything a teacher could do to drive up those scores, even if it meant rote memorization of facts, would be extremely tempting. This is the world we live in when it comes to AI benchmarking.

There’s a balance we must strike. To the degree the test actually tests what you think and hope it does, it makes sense to “teach to the test”, assuming it’s actual learning and not just coerced memorization. However, if the test isn’t perfectly correlated with the true objective, these efforts can drive perverse behaviors in both the teachers (trainers) and students (models).

Closed source benchmarks

I’m a big fan of open source when it comes to software. I’m a big fan of closed source when it comes to standardized testing.

Open benchmarks have an important role to play, and I don’t think we think we should dispose of them. A common basis of comparison that is open access, that we can all inspect and contribute to, is certainly valuable. But we also need to round out the ecosystem with closed, private benchmarks, a critical complement.

Just like in the world of standardized testing, private benchmarks would need to establish themselves as standard-bearers for quality, which requires building up trust with the broad community. In a competitive market where end users of these models really do care about quality and performance, benchmark providers will sprout up and compete for their business, jostling for position as the ultimate arbiter of model performance. In the limit, the benchmarks will be benchmarked, in the same way that colleges are ranked for quality in explicit and implicit ways. Trusted “brands” will emerge.

Ideally, these private evaluators would not train and market state-of-the-art foundation models themselves, as there would always be an incentive to either leak the benchmark data to the training pipeline or craft a benchmark that the researchers know their model will perform well on.

In a perfect world, you would have blind benchmarking, which is to say benchmarking that is obfuscated from the folks who trained the model, so that there would be no way for the training pipeline to be contaminated. Model trainers would not know in advance what the benchmark is or what the questions are. They would train models to the best of their abilities, and only know how it performs on the benchmark after training is done. This is analogous to pre-registration of scientific studies, where researchers pre-commit to a certain methodology before conducting a trial or analysis.

Even better, you would want one-time benchmarks: benchmarks crafted *once* and then only used *once* to test a model or a set of models. At that point, the benchmark would be “used up” and no longer valid. That’s how Kaggle works, and it’s exactly how most standardized tests work — each exam is only given once, on a certain day, and then it’s thrown out or farmed out to test prep providers and tutors who use it as part of their materials. This is fine — as long as the score students report to colleges is from a clean, never-before-seen exam.

Again, a college would never accept a student’s scores from an SAT practice exam as equally valid as the real thing. All I’m asking for here is a similar level of rigor.

Super serious

My point here is not — “this AI stuff is a big conspiracy, these models are just stochastic parrots with no real intellect at all”. I think we’re far enough along and the results of these models are way too compelling to short change them like that. We should take these models very seriously; they really are getting better and better all the time.

My point is exactly that — we should take these models seriously, which means taking their evaluation seriously. We aren’t right now, at least not in the public discourse around model performance.

No exam is perfect — as we know, standardized tests have their flaws. But we’re much better off having them than not. Right now, the broader AI community has no form of standardized testing for AI models. This is untenable.

It’s why I was so excited to lead Patronus AI’s seed round last year, and it’s why I’m even more pumped now that they’ve raised a $17M Series A.

As a reminder, the VAI is an index measuring current venture investment across stages relative to the long-run trend. It’s a useful and simple barometer of activity and sentiment in the venture market. The methodology is here.

Some small modifications since the last version of the index that I published in Q2 2023:

• Updated data for Q4 2023
• Removed seasonality from the data
• Removed super later stage rounds post-Series D. This was causing excess volatility that wasn’t representative of the overall venture market

With that out of the way, here’s where we’re at:

• Capital deployment remains depressed, sitting about 50% below trend
• We’ve hit the trough but remained there rather than recovered to trend. It's a bit of a dead cat bounce

Another interesting view of the data is to show the various components by stage:

• The later stages spiked the most in 2021, the earliest stages the least. This then reversed, with late stage capital falling somewhat more than early stage capital
• However, all stages are similarly depressed at this point relative to their respective trends

As I caveated in the first version of the index, if we continue to stay below trend I may need to update the definition of trend to recognize that we’re on a new trajectory. For now, I'll continue to use a long-term, constant growth rate to define the trend for each stage:

That's it! I think the data mostly speaks for itself here, so I'll keep my commentary light for this one. Will update this again next quarter.

A common explanation for the Series A winter is raised expectations – investors are demanding to see better metrics and traction. Most companies don't meet this new, higher bar, hence fewer deals get done.

While certainly true, this can't be the whole story. Only if we look at Series As in a vacuum does this explanation seem comprehensive. Incorporating what's happening in the rest of the market, however, there's a much richer narrative to explore.

The bigger picture? In addition to raising their expectations for Series A, investors have lost faith in Series A as a sign of product-market fit.

Series A investors expect more but believe less.

Relatively speaking

Valuations for both Seed and Series A stage companies rose dramatically over the last decade or so of venture:

The appreciation in Series A valuations seems extreme relative to that of Seed deals. Of course, the scales here are different, so we'd do better to baseline these to the same starting point and track the growth over time:

While the two grew roughly in line for the first few years, Series A valuations began to accelerate around 2017. That acceleration sustained itself until the recent fall off in venture valuations.

Notably, both Seed and Series A valuations are recovering. That's a small point in favor of the "raised expectations" story – investors are funding higher-quality businesses, deserving of stronger valuations.

Here's where the expectations story starts to lose its luster. Below I plot the valuation gap or "premium" for Series A relative to Seed, along with a flexible trend line to remove noise:

• From 2010 to 2014, the two grew roughly in line, with Series As being priced 200% higher or 3x Seed rounds
• Then, Series A valuations accelerate relative to Seeds to a 325% premium or 4.25x multiple
• In 2019, the Series A premium spikes again, peaking in late 2021 at 500% or 6x
• Things collapse after that, ending in Q3 2023 at 2010 levels

If investor expectations told the full story, Series A valuations relative to Seed deals would be higher today. Since Seed investments remain speculative and standards have risen for Series A investment, we'd expect the typical Series A to be even more valuable relative to Seeds, reflecting higher quality. Instead, the gap has compressed.

It's been quite the rollercoaster ride. Notably, this rollercoaster is unique to Series A / Seed. We don't see remotely the same behavior in the premium between later rounds:

The premium for Series B vs. A and the premium for Series C vs. B are both much more stable over time, with little trend once you remove the noise. The Series C / B premium saw a slight bump during the exuberant 2021 days, but that was short lived:

• The Series B / A premium has been roughly 200% for the entire period
• The Series C / B premium hovers around 125% with few exceptions

For whatever reason, the way investors value Series A companies relative to Seed stage companies has fluctuated substantially over time (slowly rising and then suddenly falling), while their views on relative value in later stages have held steady.

Why?

Series A University

What is different about graduating from Seed to Series A than going from Series A to B, Series B to C, etc?

There are many potential explanations, but the one that jumps out to me is some notion of product-market fit (PMF) and business de-risking. Series A companies tend to have much better PMF than Seed companies, in a way that's distinct from Series B relative to Series A, and so on.

Think about it like higher education – a college degree is a signal of one's skills to the labor market. Employers value them accordingly. Further, a college degree means more relative to not having one than having a graduate degree means relative to having only an undergraduate education. Again, employers value them accordingly:

While the probability of a company achieving PMF is certainly not 100% at the time of Series A (many companies raise Series A before that), it's definitely much higher at the A round than at the Seed round, since it's effectively 0% at that point.

Startup risk reduces materially once a company graduates to Series A. In a way, that's exactly why a startup successfully raises its Series A – the founders have de-risked the business, so much so that an investor is willing to invest substantial capital, take a board seat, etc.

If we posit that the "price" or "value" of PMF is well-proxied by the Series A / Seed premium, we can think of this premium moving over time due to fluctuations in the supply (among startups) and demand (among investors) for PMF:

• A rising price typically indicates either contracting supply – the relative supply of startups with PMF vs those without – or growing demand – the relative demand for post-PMF startups vs pre-PMF startups
• Importantly, these are relative concepts – the number of pre and post-PMF startups could be both growing over time, but if the number of pre-PMF startups is growing faster, that would imply a relative decline in PMF supply

So what explains the pre-2022 run up in the Series A / Seed premium: declining (relative) supply of PMF or rising (relative) demand for PMF?

My argument: rising demand for PMF.

First, as I've argued elsewhere, the growth and development of the venture ecosystem has been a mostly demand-side story. That should strongly bias us toward the demand explanation:

Investors are the primary driver of fluctuations in venture activity and equity prices around their long-run trend – We Don't Have Nearly Enough Startups

The venture ecosystem is supply-constrained – there isn't nearly enough startup equity out there to satisfy investor demand.
Additional capital drives opportunistic company formation at the Seed stage. However, the additional capital doesn't improve survival to the later stages – it simply drives prices up for the remaining companies – It's Valuations (Almost) All the Way Down

Second, the ratio of Series A to Seed transactions has been reasonably stable since about 2014. It's declining somewhat (Seed rounds are growing slightly faster than Series As) but not nearly as rapidly or vigorously as relative prices have moved. In fact, it's remarkably consistent, even through the ups and downs of the last few years:

• Seed activity (the denominator) exploded through 2014, bringing down the ratio (A long time ago, Series As were more common than Seeds)
• The Series A vs Seed ratio stabilized thereafter, only slightly falling over the years

My read: this is evidence of stable relative supply of pre vs. post PMF companies. More often than not, when quantities are steady (as in the chart) and prices are rising, it's due to constrained supply rather than constrained demand.

So, investor demand for de-risked companies rose substantially, driving higher relative prices for Series A companies. Meanwhile, the supply of such opportunities didn't grow to match that, so we didn't see a dramatic change in Series A relative to Seed activity.

Busted

Ok, so we have a plausible story for everything that happened up until the 2021 market peak. But things obviously turned after that. For my theory to be credible, it has to offer some explanation for the recent downturn that fits the data.

Speaking of data – everything we've looked at thus far is from the perspective of the overall market. But individual companies and founders don't experience the aggregate, they live their own particular trajectory, raising subsequent rounds spaced apart rather than at the same time. A founder looking to raise Series A cares about how the market looks 18-24 months after they raise their Seed:

Let's take the founder perspective and compare Series A deal activity to Seed activity 21 months prior, a reasonable (median) estimate for the time it takes to raise that next round. Think about it as a rough proxy for the Seed to Series A "graduation" or "survival" rate. Not every Series A was preceded by a Seed financing, so this is more of an upper bound for the graduation rate:

• From 2014-2020 it's a very similar picture to what we looked at before, a stable "graduation rate"
• Conditions improve substantially in 2020, i.e. more Seed companies survive to the next round
• However, post 2021 the graduation rate collapses to the lowest levels ever seen, ~20%

So, while the ratio of Seed to Series A deals at any particular point in time has been stable, due to the timing discrepancy (founders raise their Series A some time after the Seed), the numerous startups that raised Seed funding during the boom are now facing a massive bust. Ironically, though 2021 was the "best" time to raise a Seed based purely on venture activity and valuations, it was the "worst" time to raise Seed funding once you account for today's tough Series A environment.

This simple analysis doesn't accurately identify the "level" of the graduation rate, but it proxies the "change" over time. It's safe to say the graduation or survival rate from Seed to A has fallen by more than half. Other sources like Charles Hudson, Jamesin Seidel, and Carta corroborate this:

Decline in Graduation Rate from Seed to Series A - Historically, we've seen a strong pipeline of companies moving from seed to Series A. Recent numbers, however, indicate a significant decline in this graduation rate. Measured graduation rates will continue to fall for several quarters as companies go out for and fail to raise Series A rounds. Graduation rates from seed to Series A could drop to 25%, or one-third or one-half of what they were at the peak. – The Big Reset in Seed to Series A Graduation Rates is Real and Permanent

Back in 2020, approximately 23% of Seed-stage startups made it to Series-A within two years. Fast forward to 2022, and the market looks different. Over the past 20 months since the start of 2022, the graduation rates have decreased to 5%. – A Deep Dive into Q3 2023's Funding Landscape

The percentage of companies who make it from seed to Series A within two years fell by a lot for the 2021 seed cohort.

27.5% of companies that raised a seed round in 2019 made it to Series A within 2 years.

Only 17.6% of companies who raised their seed in 2021 have "graduated" to the next round. – Peter Walker on LinkedIn

How does my theory explain this data? Put simply, venture investors re-assessed their beliefs about Series A companies. They no longer see Series A as a substantial indicator of product-market fit, given so much risk still remains. That's why their relative price has fallen off.

This is different in subtle ways from the common "Series A investors have raised their expectations" narrative:

• If it was only about loftier expectations, the Series A premium would have risen, since only the best companies would be getting funded, and those companies would fetch a high price.
• That's not what we see – relative prices have fallen, not risen.

Investors haven't only raised their expectations; they just don't see Series A as indicative of PMF to begin with. If they did, the relative price of Series A companies would still be elevated. Instead, they no longer see the point in paying up for them.

Again, return to the education analogy:

• If the college wage premium plunged, we'd say employers no longer see the college-educated as having "labor market fit". In other words, they lost faith in the college degree as a signal of quality in the labor market
• You would not intuitively connect this to rising employer expectations (though that could certainly be true too)

So sure, investors raised their expectations – you need more revenue to raise your Series A, more users, etc. But the bigger factor is that investors' beliefs about Series A changed.

Investors today expect more but believe less.

If the product doesn't fit, you must acquit

I have abused the expression "post-PMF". The reality is most companies do not achieve product-market fit by their Series A, and that's always been true. But that's my entire point – investors have wizened up to the fact that Series A companies still have material risk.

PMF is more fleeting than we all had appreciated. Series A valuations corrected to reflect this.

Balance has returned to the Force. Maybe that's a good thing. But it's dreadfully volatile if you're living it in first-person as a founder. The Series A market got the wind knocked out of it just as the large 2021 Seed cohort came up for air.

While it's a tough time for founders, I think it's important to be clear-eyed about the reality of the situation. The bar has been raised, but further, investors are questioning the bar as a meaningful indicator of success in the first place.

So yes, your first priority is still to raise that next round, but there's still much work to be done thereafter. More so than ever, Series A does not mean you have product-market fit. Investors don't believe it and neither should you.

One, the developer experience for the engine itself is poor. Streaming engines tend to be difficult to use, difficult to spin up, and difficult to operate on an ongoing basis, as they have a ton of moving parts and complexity to manage. This is especially painful when trying to self-host, develop locally, or integrate into a modern continuous integration/deployment (CI/CD) pipeline. As a result, only the most sophisticated and experienced engineers can make proper use of these systems, leaving behind most software developers and narrowing the market for event-based architectures.

Second, most streaming data systems don’t integrate well into the rest of the developer workflow, and they lack many of the “creature comforts” developers are used to seeing in their other tools. The modern developer workflow extends much further out and is much broader in scope than the streaming engine itself. For streaming engines to cement their place in the hearts and minds of developers, they must offer solutions to the rest of the puzzle.

We’ve talked before about why developers love Redpanda’s core developer experience, which is leaps and bounds above what’s come before in the streaming space. With native Raft, no JVM, and full Kafka compatibility, Redpanda enables developers to do their best work, reducing unnecessary complexity and toil while doing it all at the highest throughput, lowest latency, and, importantly, lowest cost.

Comparison of infrastructure and admin costs between Redpanda and Kafka

But it’s also important to address that second point — how Redpanda creates a holistic developer experience that services all aspects of the modern developer’s workflow, from deployment to monitoring and Day 2 operations to long-term storage. These remain unresolved, “open loops” that have prevented a broader audience of developers and organizations from taking advantage of real-time capabilities.

At Lightspeed, it’s our core belief that there’s a potentially massive market expansion opportunity for real-time, streaming data systems. Opening up this opportunity will require thinking much bigger about what a streaming engine can be and do.

That's where Redpanda comes in.

Spinning up without spinning your wheels

Too often, developers looking to deploy streaming data systems are left to their own devices when it comes to deployment. Deployment is often a mess, involving multiple separate binaries, each with its own peculiarities, requiring engineers to develop expertise in systems they couldn't care less about and in which they have no comparative advantage. Streaming systems can easily become one of the most painful pieces of infrastructure they’ll deploy.

Further, developers increasingly see the cloud as their default deployment platform. In the same way that SaaS relieves some of the operational burden of running software by offloading that to the vendor, developers would love to be able to consume cloud-based streaming services with as few wrinkles as possible. Developers want a cloud experience but their organizations don’t want to lose data sovereignty. They need a security model that works within the constraints of the typical enterprise while also getting the ease of use of the cloud — the best of both worlds.

Lastly, developers on the bleeding edge don’t want to think about infrastructure at all, cloud or otherwise. They don’t want to think about servers, machines, nodes, availability zones, or regions. They want access to true streaming as a service rather than streaming as a server. In other words, they want serverless, and they want it now.

The modern developer experience requires a modern platform. Redpanda is that platform.

Here’s how Redpanda makes deployment a breeze.

One binary to rule them all

The core of the Redpanda deployment model is the way its nodes are architected. Redpanda nodes are fully-contained processes that ship with everything needed in a modern streaming system, including an HTTP proxy, a Raft-based consensus mechanism, and a schema registry. Every node runs the same, single binary, leading to significant operational simplification along with a more efficient overall operating model.

Close up of a self-contained Redpanda node

It’s hard to overstate how game-changing this is. Due to this architecture, running a single Redpanda cluster doesn’t require operating a massive fleet of varying services. This is a massive boon for developer productivity and also enables Redpanda to go where previous streaming systems have struggled, including edge / IoT deployments and CI/CD pipelines with tight performance requirements and resource constraints.

“The latency and the throughput we got with almost no configuration was incredible. Redpanda is fast, reliable, friction-free and has very low operational overhead.” - Alpaca

Hold my cloud

Redpanda’s cloud turns the table on traditional cloud infrastructure. While the company does offer standard, dedicated, managed clusters in the cloud, they didn’t want to stop there. Redpanda went further, pioneering a deployment model they call BYOC, or, “bring your own cloud.” BYOC lets customers deploy Redpanda clusters within their own cloud environment, while still being fully managed by Redpanda.

“BYOC’s privacy-first architecture drives compliance for streaming data, and allows you to scale on your own infrastructure while maintaining data sovereignty requirements.” - Bring Your Own Cloud (BYOC): best of both worlds

Redpanda does this by cleanly splitting the control and the data plane. The data plane stays within the customer’s cloud account, while the control plane stays on Redpanda’s side. This creates an incredibly slick and elegant setup, with proper separation of concerns between Redpanda’s and the customer’s cloud environment. With BYOC, organizations get all the benefits of the cloud while retaining data sovereignty and maintaining privacy.

Diagram of how BYOC keeps the customer cloud separate for data sovereignty and privacy

Redpanda takes care of everything you’d expect in a modern, competent cloud service, like provisioning, monitoring, and maintenance, while sensitive data and credentials never leave the customer environment. Rolling upgrades that the customer can control ensure zero application downtime. For obvious reasons, BYOC has become an incredibly popular deployment paradigm among Redpanda users.

“Redpanda BYOC gives us a fully managed Kafka service running on our own cloud servers, balancing our internal compliance requirements with ease of use, and without compromising performance and compatibility.” - LiveRamp

Fewer servers, better service

Lastly, Redpanda is working on a number of exciting features around serverless, a relatively new paradigm within the context of streaming systems. This is their focus on developer experience taken to the logical extreme — what a streaming data system could look like from the perspective of a developer who doesn’t want to worry about systems or deployment at all.

Importantly, serverless isn’t some esoteric technology that is far ahead of where developers are today. Serverless is here now. According to Datadog’s 2022 State of Serverless report, over half of the organizations surveyed in each of the major public clouds have already adopted serverless in some fashion. For AWS in particular that number is over 70%.

And yet streaming data systems haven’t kept pace. Developers today who want to pair streaming data with a serverless approach are largely figuring it out on their own.

We believe that serverless, in addition to Redpanda’s planned WebAssembly capabilities, is the last major step toward unlocking accessible streaming and event-based systems for the great majority of developers out there. Developers will be able to talk to a streaming data system in their native tongue and perform data transformations on the fly, almost like a universal Google Translate for streaming data.

With JavaScript and Python being the world’s most popular programming languages broadly and most used specifically within serverless functions, Redpanda’s upcoming serverless offerings will expand the relevant, addressable audience for streaming data by an order of magnitude or more.

Monitoring that even Franz Kafka would approve of

Streaming systems can be daunting, intimidating systems that are difficult to manage for the average developer. These difficulties are made no easier by the fact that the command line (i.e. a dark screen with some white text scrawled over it) is often the default way to manage these systems. Ad hoc inspection and analysis is much harder than it should be.

This is especially frustrating during crisis situations, such as production infrastructure going down. In such situations, time is of the essence, and engineers don’t have time to wrestle information out of their tools, searching for obscure command line invocations that they didn’t even know existed. Once the information is located, it often appears as a torrential downpour of logs and metrics, with little to no organization or formatting. We’ve all been there.

This just won’t do. While many developers swear by their terminals, many would love to have a more visual and intuitive representation of their infrastructure. This is especially true of younger, more junior developers who didn’t grow up writing MS-DOS applications and who have a higher standard for developer experience.

Redpanda Console is a single pane of glass for managing the entire Kafka ecosystem. All of your streaming infrastructure in one place, with all the admin capabilities organizations expect:

• Observability over clusters, topics, brokers, and partitions
• Ability to easily change consumer group offsets
• Management of schema registry, connectors, and Kafka Connect clusters

Preview of how Redpanda Console serves as a dev-friendly pane of glass.

But the most impressive thing about Redpanda Console is that it’s not merely a “dashboard” with a slew of charts and numbers — it goes so much further than that.

Redpanda Console gives developers data observability superpowers they aren’t used to. This includes capabilities like push-down JavaScript filter, allowing users to filter their cluster’s data at any level of complexity by writing programmable data filters in JavaScript, with encoding support for JSON, Protobuf, Avro, and more.

The console also simplifies and strengthens access control. The console presents an easy interface for configuring access control lists (ACLs), setting up fine-grained role-based access control (RBAC), and reviewing comprehensive audit logs. It also integrates tightly with all the identity providers you’d expect, like GitHub, Okta, and Google, enabling Single Sign On (SSO) access.

Screenshot of the Kafka Access Control interface in Redpanda Console.

“Redpanda Console delivers a tremendous improvement in the productivity, effectiveness and quality of life of developers and operators who work with Redpanda or Kafka” - Redpanda Console: Putting the “fun” back into Kafka

Store more

Okay, so a developer is running Redpanda. They’ve spun it up within their own cloud account as enabled by the BYOC model, and they have the beautiful Redpanda Console setup giving them full observability and control over their cluster. Everything is hooked up and data is flowing through Redpanda.

But all that data has to go somewhere. It’s time to talk storage.

Storage is quite important in the context of event-based architectures:

• Systems like Redpanda sit between data producers and consumers, allowing each to go about their business without having to worry about what the other side is doing.
• However, due to severing this tie between producers and consumers, the responsibility of confirming successful writes falls on the streaming system.
• Data can be produced well in advance of being consumed, so streaming systems must guarantee that no data gets lost in the shuffle.

This is why Redpanda went out of its way to submit itself for testing by trusted third parties like Jepsen.

“How did Redpanda fare in its Jepsen testing? Redpanda is a safe system without known consistency problems. The consensus layer is solid. The idempotency and transactional layers had issues that we have already fixed. The only consistency findings we haven’t addressed reflect unusual properties of the Apache Kafka protocol itself, rather than of a particular implementation.” - Redpanda’s official Jepsen report

But it’s not just about the CAP theorem and distributed systems. Putting on our investor “cap” for a second — data gravity is how nearly every mission-critical data system of the past generated significant value for its creators and its customers. For a data infrastructure company to build a large and valuable business, it must hold and retain customer data for extended periods of time. Ephemerality won’t cut it:

• Such a data system cannot merely be a place where data passes through on its way to its final destination. Such architectures and usage patterns have rarely generated multi-billion dollar outcomes. Instead, the system in question must become a trusted place for developers and organizations to store data on an ongoing basis
• Users must be able to trust that once ingested, the data will always be there (or at least as long as their data retention policies demand).
• Users must also trust that, once the data is there, it will be easily queryable and accessible.

A sane default: the cloud

Here too, Redpanda has it covered. Presciently, the team built Redpanda with the cloud as its default storage tier. This is a stronger statement than merely being “cloud-native”: Redpanda intelligently manages reads and writes behind the scenes to provide the lowest friction and highest performance, effectively becoming an infinite storage destination:

“Making cloud the default storage tier unlocks new streaming data use cases that were once considered out of reach. The long-term data retention capability encourages businesses to treat Redpanda as the “single source of truth” for their historical records.” - How Redpanda’s cloud-first storage model reduces TCO

In addition to unlimited data retention, Redpanda’s cloud-first architecture enables:

• Remote read replicas — a “CDN” for your streaming data, enabling developers to spin up and hydrate a new Redpanda cluster with mirrored data from an existing cluster, even across zones and regions
• Inexpensive disaster recovery — smooth failovers, with no need to maintain a separate data replication mechanism
• Reduced data management, admin, and toil — Continuous data balancing, self-healing clusters, and unified retention controls

Redpanda accomplishes this with its Shadowing Indexing architecture, which was built from scratch to support cloud-native Tiered Storage, enabling Redpanda to seamlessly move data between brokers and reliable, cheap cloud stores like Amazon S3 or Google Cloud Storage (GCS). Users can access their data using the same Redpanda/Kafka APIs they’re used to, while getting infinite data retention and scalability for free.

Diagram of how Redpanda’s Tiered Storage works with Amazon S3.

The upshot of all this innovation is amazing ease of use and performance at an incredibly low cost.

Data unchained

We’ve already talked a bit about data sovereignty in the context of the cloud, where security and retaining control of enterprise data are key concerns. That’s a very literal interpretation of sovereignty. But there’s another, potentially equally important notion of control that developers and organizations care about, but often give up as a side effect of adopting cloud data stores. Often enough, one doesn’t even realize something has been lost until you’re in too deep.

“Data sovereignty is much harder to achieve than data privacy. Privacy can be achieved with policy: delete this, mask that, obfuscate here, index like so. Sovereignty can only be achieved if you, the user, control the hard drive lifecycle where data resides. There are no two ways about it. Data either lives inside the hard drives that you control or it does not.” - Data sovereignty is the future of cloud

Redpanda believes developers should control the data they produce. This goes without saying in the world of on-prem. However, we cannot take this for granted in the era of the cloud:

• That hard drive holding your data is no longer running in your data center — it’s running in someone else’s.
• Further, once it ends up in that external store, many vendors erect all sorts of barriers to accessing and manipulating the data: esoteric and proprietary data formats, domain-specific query languages, egregious egress fees, and the like.

These aren’t (pleasant sounding) data lakes or even (somewhat less enticing) data warehouses, they’re data jails, and the vendor holds the only key. As far as Redpanda is concerned, that isn’t true data sovereignty.

Redpanda wants to fix this. Leveraging popular open data formats like Apache Iceberg and the unique BYOC architecture we discussed earlier, Redpanda will enable developers to finally regain ownership over their data. Developers will be able to pick their preferred storage vendor while maintaining ownership, queryability, indexing, and portability no matter where the data ends up. Today it’s S3, tomorrow it’s Snowflake and Databricks, and in the future, it’ll be any vendor they’d like. Organizations will also be able to bring their own query engines to the data, no matter the underlying data format.

Developers will once again own their data. All enabled by Redpanda.

Conclusion

Redpanda meets developers where they are and then enables them to go even further. That reach will push Redpanda to places no streaming system has gone before, driving a level of mission-criticality beyond most other developer infrastructure. The ease of use and incredible flexibility of Redpanda will generate developer love and appreciation for the product. And, with infinite retention and a strong commitment to data sovereignty, Redpanda’s “data gravity” will only grow.

Diagram of the Redpanda compatibility ecosystem

It's why we were so excited to lead their recent $100M Series C financing. The real-time revolution has only just begun, and Redpanda is leading the charge.

After a euphoric 2021, funding took a nosedive, falling to the lowest pace we've seen in some time. While economists and politicos debate whether we're in an economic recession, there's no debate in the venture economy – it ain't pretty out there for founders trying to raise capital right now.

But that sentiment is somewhat anecdotal, backed by the gut feeling of venture market participants. It'd be great to have a view of the venture cycle that was backed up by the data, some sort of indicator of the phase of the cycle we're currently living through.

So I came up with a methodology for measuring the state of the venture "business cycle" – how the venture market is performing relative to some underlying notion of "trend". Its construction is actually quite simple, and the result is something I think could serve as a useful barometer for the ecosystem as we chart a course from here.

It's called the Venture Activity Index, or "VAI". Let me walk you through how I got there.

Quick stats

To measure the venture cycle, we need a notion of "trend" – the thing around which the venture market is fluctuating. If we plot funding growth on a logarithmic scale across each stage of venture investment, a clear linear trend emerges. (Note: linear trends in logarithms implies constant percentage growth over time):

I could have come up with a fancier notion of trend, but I'll proceed with the linear one as it's nice and simple.

Venture activity, as defined by aggregate funding, fluctuates up and down around this trend. The slope varies by stage, but it's a fairly consistent story across stages – on average, funding grows ~5% each quarter.

One quick note: I could have focused instead on the number of investments rather than the scale of investment activity measured in dollars. However, working in dollar terms is nice because it implicitly accounts for valuations, which is critical for holistically characterizing the state of the venture market.

Next, let's remove that trend and focus on the gyrations around it:

• The cycles were only moderately correlated across stages before 2020. The average cross-stage correlation hovered around ~0.25 pre-2020.
• However, since 2020, they've moved in lockstep, following one common cycle up and then down.

This is typical in economic data: in volatile times, different parts of the market often become much more tightly correlated. It's interesting to see the same phenomenon in the venture data. Further, it's important that the various stages of venture correlate with one another, as otherwise there'd be no sense in talking about a singular "cycle" for the whole ecosystem.

Overall, the correlations are around ~0.65 for the whole period, with the seed stage being the most dissimilar to the other stages of investment:

The amplitudes of the cycles, however, differ across stages, both historically and more recently:

• The seed stage is relatively steady, never more than 50% off trend, with a quarterly standard deviation across time of about 20 percentage points.
• Series A is a bit more volatile but not significantly so.
• Growth and later stages are the most volatile, ranging from a standard deviation of ~30 p.p. for Series B all the way to ~50 p.p. for Series D+.

In other words, booms and busts are substantially larger at the later stages:

And in case you're wondering, this was true before 2020 as well. It's not just a recent phenomenon: the late-stage market has always required a bit of a strong stomach.

One metric to rule them all

Now let's throw it all together. Again, in the spirit of simplicity and robustness, let's take the simple average of the individual trends to arrive at a blended view of the venture ecosystem.

I present to you… the Venture Activity Index (VAI):

The last three years of the VAI are the obvious anomaly:

• Capital deployment rose to 90% above trend, exploding over only a few quarters.
• The correction was just as vigorous, bottoming out at ~60% below trend and stable for the last two quarters.

We haven't seen anything like this in the prior ten years of activity. Previous peaks and valleys were at most +/-25%. The volatility of the last three years is unheard of in recent memory.

When knocked off trend, venture activity returns to steady state after ~1.5 years. Only in that sense was the exuberance of 2021 relatively normal – it lasted roughly as long as such booms tend to last. We'll see if that behavior continues.

Again, I want to emphasize this is a measure of the venture business cycle, not the trend itself, which we first removed. A rise or decline in the VAI implies funding grew faster or slower than trend, respectively, not that funding rose or fell in absolute terms.

Due to its cyclic nature, the VAI forecasts future changes in venture activity. When it's high, future venture activity slows. When it's low, venture activity accelerates in the next few quarters. This is true across stages, though it's most accurate for the later stages:

Overall, a 10% increase in the VAI forecasts anywhere from a 6.5% decline in seed stage venture funding over the next year to a 21.5% decrease at the Series D+ stage (and vice versa for a decrease in the VAI).

Conclusion

The Venture Activity Index is a simple and informative indicator of the state of the venture "business cycle". In a single number, it quantifies the "gut feelings" we all have about the venture climate today, in the past, and in the future.

To wrap up, a few caveats:

• The backward-looking trend updates every quarter, which can slightly change the historical numbers as new data comes in.
• A linear trend may not make sense in the long-run. For example, if the rate of capital deployment never returns to its prior trend, at some point we'd have to accept that a new trend has been established, making the old one irrelevant. I may later change my calculation of trend to address this.
• My simple average of the various stage-specific trends could be improved upon with some sort of factor analysis technique. Again, I may switch to this at a later date.

I plan to update the VAI quarterly as new data comes in. For the latest data, look here.

Valuations rise and fall with the volume of dollars invested in startups according to a stable ratio: for every one percent change in funding, valuations move two-thirds of a percent.

But since peaking in late 2021, valuations have only fallen 0.4% for every 1% drop in funding. This pricing "error" has accumulated: today's valuations are 60% higher than you’d expect for the amount of capital invested.

In other words, the "shadow price" of venture capital is a lot lower than what we're seeing in announced transactions these days. Never before have valuations and funding diverged so meaningfully from their long-run equilibrium, for so long.

What does it all mean?

A 3-for-2 special

Valuations and capital invested in venture have both grown substantially since 2010:

Interestingly, they trend together. As one rises, the other does too. Same in the other direction.

It's not a perfect relationship, but there's clearly something tying them together.

In fact, you can predict average valuations from the capital invested each quarter. The below chart plots actual valuations over time against what we'd predict based on capital deployment – a simple regression of valuations on funding:

Capital invested predicts the average valuation of deals done each quarter.

Most of the time, the "errors" of this capital-based prediction are small. Further, these deviations typically correct themselves within a quarter or two.

In this sense, venture valuations are a function of capital flows. More capital drives valuations ever higher, in a fairly predictable fashion. There's an equilibrium, a balance between the two.

They don't grow at the same rate, however. For every 1% increase in funding, valuations rise about 0.66% or two-thirds of a percent. If you like clean whole numbers, the ratio is about 3:2.

Most of the increased funding goes to rising prices rather than more individual investments. This mirrors points I've made previously:

"the valuation inflation we've seen "comes from" the incredible growth in demand and lack of supply of startup equity"
…
"Additional capital drives opportunistic company formation at the Seed stage. However, the additional capital doesn't improve survival to the later stages – it simply drives prices up for the remaining companies" – It's Valuations (Almost) All the Way Down

Price check, please?

Economists have a term I think is relevant – shadow price – or, the price of something that either isn't typically traded in the market or for which accurate pricing is hard to come by. For example:

• The price of an illicit good (drugs, contraband of various sorts, etc)
• In certain economies like Argentina, the shadow or "black market" price for exchanging the local currency for a foreign currency like the U.S. dollar, which differs from the "official" rate.

Why is this relevant to venture capital?

Well, because so many venture transactions don't happen. Most startups fundraise only once in a while. Founders delay fundraising if they can't fetch an attractive price from investors. Down rounds are verboten. Those deals are missing from the quarterly venture activity data:

• Accounting for these phantom fundraises would lower the average venture valuation, since low prices are the whole reason those deals aren't happening. In other words, there's massive selection bias.
• Companies also tend to be more public about their valuations the higher they are. Journalists love reporting on high valuations. As a result, the data we have on valuations is also likely skewed too high due to reporting bias.

The prices we see investments getting done at are misleading. Thus, the concept of a shadow price applies to venture – it's the price that would prevail if companies were forced to transact at current prices and we had perfect data on all fundraises.

I think of the dashed line in the chart above as something akin to the "shadow valuation" of venture capital investment.

Wow, that's a high price!

The reliable relationship between prices and funding has broken down in recent quarters.

While valuations and funding have both declined since late 2021, funding declined by much more. Valuations have not fallen by nearly as much as the 3:2 ratio would imply. Instead, since the market peaked the ratio has been more like 5:2, or, said differently, valuations have only fallen 0.4% for every 1% drop in funding.

In other words, even after collapsing, valuations are still too high relative to their historical relationship with funding.

We can visualize this growing "error" – the percentage difference between actual and predicted valuations – shown below. Above the zero line means valuations are too high, below means valuations are too low:

Current valuations are ~60% too high relative to the volume of capital being deployed. There just aren't enough dollars sloshing around to support these prices, at least based on the 10 years of venture history prior to the go-go days of 2021.

Since 2010 valuations have rarely been "off" by more than 20%. 60% is unprecedented.

Again, notice how in the past any deviation from equilibrium quickly reverts, nearly always within a quarter or two. Something has changed – "error correction" is totally absent in recent quarters.

This is worrying. Never in the last decade-plus of venture activity have valuations and funding flows diverged so meaningfully from their long-run equilibrium, for so long. Could this mean there's a lot more pain ahead?

These deals won't last

Now, it's totally possible we have it all backwards.

Till now I've taken for granted the idea that capital flows drive valuations.

Perhaps valuations have risen for separate, independent reasons and funding volume grew to meet these new prices. Thus, it could be funding that corrects itself, rather than valuations. Perhaps the current gap signals funding is too low rather than warning valuations are too high.

Let's check what happened in the past. That is, how have valuations and funding historically reacted to past deviations from their long-run ratio? That would be a strong clue as to which is the driver and which is the passenger here.

That's straightforward enough – just run a regression of valuation and funding growth on deviations. If valuations react more, then we know valuations correct for past deviations from equilibrium. If invested capital reacts more, then its funding that corrects.

Turns out, valuations correct past deviations (left chart), funding does not (right chart):

When valuations and funding volume drift apart, it's valuations that come running back:

• If valuations are high relative to funding, valuations tend to fall the following quarter
• If valuations are too low relative to funding, valuations rise next quarter

Funding doesn't respond at all.

Deviations reflect unusual valuations rather than abnormal funding. Valuations are out of whack, not funding volume.

Pricing "errors" forecast future valuations, since valuations reliably and quickly respond to those deviations in later periods. The implication? Valuations should fall dramatically from here. That they haven't yet can be chalked up to a combination of market inefficiency and reluctance.

Just like in a Keynesian model of the economy, where sticky, slow to adjust prices exacerbate economic downturns, "sticky valuations" lengthen and worsen venture recessions. Deals that could get done at more reasonable valuations don't happen, as founders and existing investors don't want to take the hit:

Private valuations lag public valuations, often by a substantial amount and for a long time – Old Valuations Die Hard

We should prefer swift, definitive corrections over slow moving train wrecks. Instead, we drag things out. Everyone suffers as a result.

Tourist prices

That venture prices are tied so strongly to capital flows is striking:

• This contradicts standard corporate finance which values companies based on fundamentals. The amount of capital in the market shouldn't impact valuations if the intrinsic value hasn't changed.
• However, that's not what we see – there doesn't seem to be anything "fundamental" anchoring the price of startups. Startups are priced based on the amount of capital in the private markets at the time.

More capital, higher prices. Again, it's not one-for-one, but it's clear most of the capital goes toward higher prices rather than more deals getting done.

In my last essay, I was struck by how sensitive private valuations were to interest rates:

… seven quarters out valuations are up ~25%, falling back to their original level after three years – Don't Discount Interest Rates

As readers pointed out, a pure discounted cash flow (DCF) analysis would never suggest such a large impact. This is a fair point, and one I too pondered.

However, these magnitudes make much more sense in the context of capital flows. Low interest rates attract capital to risky assets like venture capital as investors reach for yield unavailable elsewhere. If capital flowing into startups raises prices, then we really have two simultaneous effects:

• The first is traditional, static, corporate finance, which says interest rates (or rather, the discount rate) influence valuations.
• The second is the dynamic effect of interest rates on venture capital funding, which then impacts valuations via the 3:2 relationship I studied above.

This double whammy is how you end up with such severe valuation swings.

Ironically, it's the tourists who "price" venture investments on the margin – those much-derided investors who, like birds, cyclically flock to and flee from venture investing with the changing economic winds. Us "locals" have no choice but to live with it.

But the Fed's meds have a curious side effect – they're a shot of adrenaline for the venture market.

Low interest rates jolt the heart rate of venture capital, driving a manic frenzy of transactions and funding for startups.

On the other hand, high interest rates kill the vibe, causing deals to dwindle and prices to plummet.

You don't need a PhD to understand that. What's less obvious is exactly how large of an effect we're talking about here.

Turns out – it's huge. For a 25 basis point or 0.25% change in the one-year Treasury yield:

• Deal activity adjusts ~10%
• Valuations move ~25%
• Capital invested shifts by ~30%

These effects are persistent, meaning we're always dealing with the aftermath of past interest rate shocks. It's the (highly oxygenated) air we breathe.

It's Jay Powell's world. We're just living in it.

Interest rates are interesting

The Federal Reserve's zero interest rate policy (ZIRP) has come to an end:

It's no secret interest rates affect tech valuations:

During the last few years low interest rates and money printing led to a funding bubble in private technology – Elad Gil, Startup Decoupling & Reckoning

So the corporate finance logic goes, companies are worth the present value of their cash flows, and the "discount rate" one applies to those cash flows is the key input – lower rates mean higher valuations, and vice versa.

As much as investors like to ignore it, there's an obvious connection between valuations and interest rates:

Source: Redpoint Ventures

Source: Bessemer Venture Partners

These charts reference publicly traded companies. However, I've never seen anyone quantify the sensitivity of private tech valuations. In other words, "if interest rates fall by X percentage points, venture valuations rise by Y%," and vice versa:

• Merely knowing Y is a positive number is only marginally helpful; we should really want to know exactly how large Y is!
• Also, to what extent are tech valuations driven by interest rates and to what extent by other factors?
• Oh, and while we're at it – is the effect instantaneous? Or does the effect take time to percolate given the inefficiencies of the venture market?

Meanwhile, interest rates could affect more than just valuations. Recall a few essays ago I introduced the following framework for thinking about the individual "components" of a dollar of venture capital funding – deals, valuation, and dilution:

Interest rates could affect all these variables, and we should want to know how:

• Deals: Low interest rates drive speculation, encouraging "betting" on startups. High rates send the gamblers home.
• Dilution: Low rates create a more founder-friendly environment, reducing the ownership founders give up. Vice versa for high rates.
• Funding: To the degree investors "search for yield," startups attract capital when other investment opportunities are scarce.

So I set out to find some answers.

A quick note before we proceed. It turns out, the Federal Reserve is mostly predictable, so most of its moves are already "priced in" by the market. I instead focus on unexpected and unanticipated shifts in interest rates that market actors haven't yet reacted to. To account for these expectations, I control for other macro variables like U.S. GDP, inflation, and the Nasdaq index.

For example, if the one-year Treasury rate declines by 0.5%, the market may have only expected a 0.25% decline based on current economic conditions, leaving 0.25% unexpected:

OK – that's the most complicated concept you need to understand. With that out of the way, let's jump to the results.

Long and variable lags

The following charts trace the effect of a 0.25 percentage point or 25 basis point (bps) cut in the one year U.S. Treasury yield on various measures of venture activity, up to twelve quarters / three years out:

• 25 bps is the typical increment the Federal Reserve uses.
• I focus solely on the "surprise" component. The "total" change in rates would have been even larger.

The effects are quite strong, especially on deal flow and valuations:

• Deal activity rises for six quarters before receding back down to zero after ten quarters. Peak impact is substantial – about 10% more deals are getting done six quarters out
• Valuations take longer to peak but the size of the effect is more extreme; seven quarters out valuations are up ~25%, falling back to their original level after three years
• Dilution is more muted, remaining largely flat initially but then dipping ~5% after a few quarters without ever recovering with the three year window

Critically, interest rates affect deal flow, not just valuations:

• It's not simply the same set of companies raising money at higher prices. More companies get funded in the first place as well.
• As I found in Old Valuations Die Hard, deal flow reacts quickly, while valuations take longer to adjust but see a larger impact overall.

This has pros and cons:

• More deals means more entrepreneurs get to take a swing.
• Conversely, however, when interest rates rise, founders not only accept lower valuations – some founders don't get funded at all.

Let's drive that point home by visualizing the same analysis but for an increase in interest rates rather than a decrease:

Rising rates squeeze the life out of venture capital. Per the logic I outlined in my last essay, we know this reflects reduced investor demand, since quantities and prices drift together:

Demand can shift… which causes prices and quantities to move in the same direction (up when demand increases, down when demand decreases – We Don't Have Nearly Enough Startups

The (multiplicative) aggregation of these individual sub-effects yields the overall effect on venture funding:

• Funding builds up for almost two years, peaks at nearly 30% above baseline, then falls back to zero by about 11 quarters out.

With rising rates, funding falls just over 20%:

Perhaps it's obvious, but these are extremely large effects!

Thankfully, interest rates don't move up or down enough to regularly generate these sorts of reactions. A 10 bps / 0.1% surprise is much more common than a 25 bps / 0.25% one.

Notably, the effects aren't permanent; As interest rates reset, so does the venture market.

A brief history of time value

The current "era" of venture capital has been deeply influenced by this strange interest rate regime, having evolved entirely within it.

With our previous estimates, we can run a backwards-looking "attribution analysis", explaining the ups and downs of venture in terms of interest rates. In other words, we can break down the recent history of venture activity into the portions influenced by interest rates vs. other factors.

First up: deal activity. In red I plot an index of the overall growth in venture deal activity since early 2014, averaged across funding stages, and in blue I plot the portion attributable to unexpected interest rate shocks:

• Interest rates explain nearly the entire explosion of venture deal activity over the last few years.

Of all the charts, this is the one I probably spent the most time staring at and double-checking the numbers, as it's just so striking.

Let's do the same for valuations (notice the bigger scale here):

• The higher rates of the 2018 era pushed down valuations, but the effect reversed as rates fell heading into 2020. Interest rates on their own doubled valuations.
• However, the valuation inflation was so extreme that interest rates can't explain it all, implying more had to be going on.

This is important: that interest rates explain nearly all deal flow but only part of the rise in valuations implies that demand for startups has outstripped supply of startups. With nowhere else to go, that excess demand spills over into prices. You can't invest in startups that don't yet exist, so you compete for the few that do, bidding up prices. This is the same conclusion I reached in a prior essay:

… the valuation inflation we've seen "comes from" the incredible growth in demand and lack of supply of startup equity – It's Valuations (Almost) All the Way Down

Here's dilution:

• Dilution is noisier, complicating interpretation. The way to read this chart is that interest rates increased dilution through 2019 but afterward exerted downward pressure on dilution.
• From 2019 through 2022, interest rates drove dilution down 20%, which is exactly how much they declined in total, therefore accounting for nearly all of the fall.

Putting it all together, here's what our little attribution methodology has to say about the effect of interest rates on overall VC funding:

• From 2014 through the top of the market in 2021, interest rates accounted for a 200% (!!!) expansion in venture funding.
• As with valuations, the run up in venture funding was too extreme to be completely explained by interest rates.

Don't discount interest rates

Monetary actions affect economic conditions only after a lag that is both long and variable – Milton Friedman

Before concluding, I should mention some caveats to this analysis:

• First, there aren't many examples of big interest rate moves over this period. I may have overfit to the few meaningful changes in rates since the 2008 Financial Crisis.
• Second, it's possible I haven't controlled for all the relevant variables. For example, a lot happened during the pandemic that isn't fully captured by my choice of controls.
• Third, I use the U.S. Treasury yield, i.e. the risk free rate, as my measure of interest rates, which doesn't fit a high-risk asset class like venture capital. Unfortunately, no venture-specific analog exists.
• Fourth, and I'm repeating myself because it's important, these are the effects of an unexpected shift in rates, whereas most interest rate movement is expected by the market. If interest rates change by 50 basis points tomorrow, no more than half of that was in fact a "surprise."

Caveats aside, Friedman's proclamation appears to hold for venture capital – it takes multiple years for interest rate effects to fully play out.

Even in the private markets, we're all Fed watchers now: these interest rate effects are too large to ignore. In fact, interest rates are so impactful that they explain most of the mass hysteria of the last few years, both on the upside and the down.

Importantly, interest rates cannot forever move in one direction. Over the years, they mean revert, making their impact temporary at best. Don't get too accustomed to any particular regime – the new normal will always eventually look like the "old normal."

Don't discount interest rates.

Two possible drivers – investor exuberance and founder fervor:

• Did investors become substantially more favorable toward private startups, and founders merely reacted to that increased interest?
• Or, did we all become much more entrepreneurial, and investors simply provided the capital in response?

In other words, was it demand, supply, or some combination of the two?

Here's a hint: we don't have nearly enough startups.

Narrative violation

Measured in either financing activity or equity valuations, the venture-backed startup ecosystem has grown a lot:

There's a common narrative out there that the bonanza in venture-backed startups over the last decade reflects growing entrepreneurialism, the ease of starting a company, etc.

I question this story.

Only Seed rounds grew significantly faster than real GDP over the last decade. Series A's also outpaced GDP but not by much. The rest are on or even below trend, especially after the recent slowdown:

Note: Q4 2022 US GDP not yet available as of publication

The long-run trend of real GDP reflects supply-side factors, i.e. the fundamental productive capacity of the economy. The similar trends at the later stages imply the "fundamental productive capacity" of the venture ecosystem, i.e. its ability to generate real, meaningful businesses, is growing no faster than the rest of the economy.

The U.S. economy isn't growing quickly by the way – 2% real growth year-over-year is a good benchmark. So if the trends line up, the venture world can't be growing much faster than 2% in real terms, that is, excluding the impact of valuation inflation that I discussed a few essays ago.

At the later-stage, valuation inflation explains nearly all growth in venture funding over the last eleven years – It's Valuations (Almost) All the Way Down

An equally if not more plausible story is that demand-side factors drove the explosion in venture activity over the last decade. Growing investor interest and appetite for private startups could explain much of the growth.

However, merely staring at trend lines doesn't yield much insight. It's like a tough marketing attribution exercise – the growth attributable to either demand or supply is ambiguous.

We should resolve this ambiguity:

• As participants in this ecosystem, we should want to see it grow in real terms rather than merely nominal ones
• In other words, we'd be much better off if venture activity reflects growing capacity and ability of entrepreneurs to start successful new businesses
• If in fact the growth in venture activity is merely a reflection of investors needing increasingly obscure and speculative places to park their money, we have a big problem on our hands. The shell game will implode:

Valuations can't rise forever, so over the long-run venture capital can't grow much faster than the number of ventures themselves – It's Valuations (Almost) All the Way Down

Economics 101: Supply and Demand

First, let's quickly review Economics 101.

Here's the classic supply and demand chart. Supply and demand together determine equilibrium prices (Y-axis) and quantities (X-axis). Importantly, demand is downward sloping (you want more stuff the cheaper it is) and supply is upward sloping (you produce more stuff when you can sell it for more):

In this model, two forces can move markets. Demand can shift (for reasons that don't have to do with prices themselves), which causes prices and quantities to move in the same direction (up when demand increases, down when demand decreases):

Or, supply can change (again, for reasons other than price), in which case prices and quantities move in opposite directions (price move against supply, quantities move with supply):

In simple terms, when everyone wants the same thing and wants it really badly, the price tends to go up and more of that item gets bought and sold. When everyone wants to produce and sell the same thing and wants to do so really badly, they compete against one another, driving prices down to accommodate the increased activity.

In venture terms, when investor demand for startups rises, valuations increase and more deals get inked. When entrepreneurial supply expands, valuations decline and more deals get done.

VC is trendy

Venture deals and valuations have both grown over time, so it's all demand-driven right?

Not so fast.

While it's tempting to end the analysis here, for reasons I'll skip over, you can't simply compare two trending metrics and assume they're correlated. The relationship could easily be spurious, and we don't want that. By the magical transitive property, anything we concluded based on that relationship would be spurious as well. Cool math.

The key is to "de-trend" the data first. We need to remove the long-run trend and then compare deviations from that trend.

Ignoring the details, here's what that looks like (data normalized and smoothed slightly to remove noise):

Lo' and behold: deal flow and valuations fluctuate together around their respective trends.

OK, maybe that's not always true, but it's nearly always true. Early stage is the exception to the rule.

Seed financings were negatively correlated with valuations before roughly 2017:

• When deal flow increased, valuations plummeted, and vice versa
• This smells like supply to me – there was only so much investor demand, so when more startups came to market, they competed, and prices tended to fall, benefitting investors. When the supply of startups contracted, prices rose as investor battled over the few remaining deals

Series A rounds used to be positively linked to valuations, but they've de-correlated over the last few years:

• Pre-2019, Series A deal flow and valuations tended to move in the same direction
• Sounds like demand to me – startup supply was constrained, so investor sentiment drove the market, moving prices up as they grew more eager and down as they soured on the venture ecosystem

Across growth and later stages, deal flow and valuations are unambiguously positively related, moving almost in perfect unison for the last decade:

• Demand clearly wins it – late stage supply is badly constrained, so investor demand is the prime mover. Their manic and depressive episodes move the market accordingly

Some of these relationships have shifted over time, so let's visualize that with rolling three-year correlations for each stage:

This is my qualitative narrative in quantitative terms:

• The Seed stage flipped being negatively correlated to positively (supply → demand)
• Series A de-correlated to effectively zero (demand → ❓)
• Series B and later deal flow and valuations have always been strongly positively related (demand all day baby)

Channel attribution

This is the best evidence I've seen to date for the demand-side hypothesis.

The scorecard so far suggests demand reigns:

• Investors are the primary driver of fluctuations in venture activity and equity prices around their long-run trend
• Yes, more deals are getting done (so by definition more startups are getting funded), but that appears to be a function of increasingly desperate investors rather than increasingly bold and enabled founders

In other words, the supply of startup equity is badly constrained:

The venture ecosystem is supply-constrained – there isn't nearly enough startup equity out there to satisfy investor demand.

Additional capital drives opportunistic company formation at the Seed stage. However, the additional capital doesn't improve survival to the later stages – it simply drives prices up for the remaining companies – It's Valuations (Almost) All the Way Down

As a reminder, our evidence for this is the positive link between de-trended deal activity and valuations. That's a nice trick, but it only tells us at each point in time whether movements in demand or supply dominated. I want to explain the entire last decade or so of venture history in terms of supply and demand channels.

Yes, yes, history is not bi-causal, hammer looking for a nail, etc, but how about one more magic trick to close things out?

Warning: armchair econometrics ahead!

Let's stretch our simple supply and demand model to the absolute extreme:

• Remember, a positive relationship between deal flow and valuations suggests a change in demand, while a negative or inverse relationship suggests shifting supply
• So, we could simply attribute each quarter of venture activity to either demand or supply demand based on whether deal flow and valuations move in similar or opposing directions during the quarter
• We can then cumulate the respective contributions of demand and supply to the growth of the venture ecosystem over time

Here's what that looks like for venture deal flow since early 2010. Demand is red, supply is blue, sugar is sweet, and so are you:

That's a lot of red out there:

• The demand channel drove most of the growth in venture activity in nearly every stage other than Seed, where its relative contribution is closer to 50/50
• Early stage supply contributed positively to deal flow from 2010 to about 2015 but then stagnated
• Supply has never been a meaningful contributor at the growth and late stage and even seems to have contracted in certain cases

Et tu, valuations?

The story isn't much different for valuations, except perhaps with some signs flipped:

• Here again, investor demand was the main driver, pushing prices higher in every stage
• The increase in early stage supply in the early 2010s relieved some price pressure, but this eventually receded
• The effect of late stage supply on valuations is somewhat noisy, but by the end of the sample those supply constraints appear to have driven prices higher, on balance

In case I haven't sufficiently caveated already: this is extremely unscientific. No Nobels will be awarded for this work (your subscription is enough reward for me, awwww), but it does serve as coarse, suggestive evidence that demand is, or at least has been, king in venture over the last decade.

There's only so many startups to go around

So, what have we learned?

To the degree founders are starting more venture-backable companies, it's largely driven by the cold, rational calculus that investors are much more eager to buy up equity in private companies today than they used to be.

This explains much of the growth in the venture ecosystem over the last decade: more deals get done and those deals are more expensive because of surging investor demand. This has been great for founders.

However, this will be painful for investors, since companies rarely grow into their valuations:

Most often, companies don't grow fast enough to compensate for rising valuation multiples. Instead, high valuations today imply slower value appreciation in the future, i.e. lower returns.
…
Investors like to think companies will grow into their valuations, but more often than not, the stock simply underperforms – Companies Rarely Grow Into Their Valuations

Meanwhile, the supply of startup equity remains constrained. Rather than potential founders becoming more eager to start companies for "fundamental" reasons, entrepreneurs are reacting to investor sentiment. While there's been some growth in supply at the earliest stages, the fundamentals haven't necessarily improved much, which is why late stage deal flow hasn't grown any faster than U.S. GDP.

I'll repeat what I said earlier – we'll all be much better off if more people start companies for good, wholesome reasons that don't have anything to do with valuations.

There was a time where the notion of handing millions of dollars to an extremely young company sounded crazy, and anyone willing to do so extracted a significant ownership stake for taking on that risk. Founders got diluted, badly.

We're well past that now. Dilution has fallen in every stage since 2010:

We have more startups, but:

• It's largely been a reaction to attractive valuations and reduced dilution that founders must endure to raise capital
• Past the seed stage, it's not at all obvious that founders are fundamentally better equipped to build successful companies today vs. a decade ago

The demand side got a bit ahead of itself. It's time for the supply side to catch up.

We don't have nearly enough startups.

A worsening economic outlook and tight monetary policy sent tech stocks through the floor.

However, private valuations are slower to adjust.

Private valuations lag public valuations, often by a substantial amount and for a long time.

As we all lick our wounds during this venture downturn, many are asking: when will it all end?

The answer? About three years.

Public-private partnership

Private valuations rose dramatically over the last decade. But so did the Nasdaq:

It's remarkable how well the two track one another.

In a sense, the growth of public tech valuations seems to pin down the rise of private valuations, defining a trend to which private valuations always tend to return. It's clear that venture investors use public valuations to justify private ones.

One feature immediately stands out – private valuations are much more volatile than public valuations, particularly at the later stages:

• This is to be expected – it's not the same companies that are fundraising each quarter, so we'd naturally expect the prices to jump around a bit
• Meanwhile, the composition of the Nasdaq doesn't change much quarter to quarter, so as an aggregate it's much more stable

If we throw each one on its own axis, the relationship is even more obvious:

Across stages, private valuations move one-for-one with public valuations.

• Seed valuations have grown somewhat slower than the Nasdaq, moving 0.5% for every 1% move in the tech index
• Series A, B, and C valuations have moved one-for-one
• Series D+ valuations have grown 1.3% each point of growth in the Nasdaq

We can replicate the same analysis for private venture activity. The relationship is not as tight:

Deal activity is more or less related to public tech valuations, depending on stage. Here too we see much more volatility in the private realm than the public, again, for understandable reasons.

In general the slopes here are flatter than for valuations:

• Seed deals have grown at roughly the same pace as public valuations
• For other stages, growth has lagged public tech valuations – growing at about half the pace

Level with me

Though intuitive, these comparisons are not rigorous. It's too easy to find spurious relationships among variables that are all trending in the same direction. There's less risk of that in our case since public and private technology markets are closely linked, but we should be wary regardless.

We can do better by focusing less on the relationship between level of the Nasdaq and some other variable and more on the correlation between changes in the respective metrics.

The key question: how does a change (up or down) in public technology valuations affect the prices of private financings, and how quickly?

Some good ol' regression analysis can answer this.

Glossing over a bunch of detail:

• We can forecast future changes in private valuations and financings based on current changes in public tech valuations.
• We'll run one regression for each forecast horizon: concurrent (time zero) impact, one quarter out, two quarters out, etc.
• The coefficients of these regressions trace out the impact of the original movement in the Nasdaq on private prices and volume in later periods.

The path to recovery

Hope you're still with me. Let's run those regressions and see what we get.

We'll focus on downward movements in the Nasdaq. This is how a 1% decline in the Nasdaq affects private valuations and deal flow over future quarters:

Private valuations gradually decline after a drop in the Nasdaq:

• Prices drop continuously for four quarters
• At the trough, private valuations fall ~2.25% lower for every 1% loss in the Nasdaq
• Venture valuations take 10 quarters to recover

Movements in the Nasdaq reliably forecast venture deal activity too, which drops and rebounds faster than prices:

• Deal activity bottoms in the third quarter after impact
• Overall deal count rises back to its original level by the 8th quarter

Down for a down round?

This masks a lot of underlying variance between the different segments of the venture ecosystem. Starting with valuations, let's explore how the shape of the recovery varies by stage:

Valuation dynamics depend on stage: Early stage valuations drop 1-1.5% for every 1% decline in the Nasdaq, while growth and late stage valuations decline 2-2.25%.

Note: these aren't exact estimates. Uncertainty grows as we look further out from initial impact, so don't pay much attention to the numbers at the 12-quarter mark. It's the shape and length of the recovery that matters most.

With some back-of-the-envelope math, we can estimate both the magnitude and the length of the current downturn:

• The peak to trough decline in the Nasdaq this past year was about 25%, implying a ~25-35% haircut to early stage valuations and a ~50-55% cut in growth and later stage valuations
• That might sound extreme, but the sudden explosion of late stage valuations these past few years was itself quite unusual. It's not implausible that prices could fall as dramatically as they rose

In fact, the correction is already well underway:

As predicted, late stage valuations drop much more than the Nasdaq, while early stage valuations move roughly in line. Prices have fallen faster than my forecast, but the magnitudes are on point.

Timing the bottom based on my regression here is a bit tricky, as the market began its decline in Q1 2022 and mostly ran its course by Q3 2022. If we use the midpoint of Q2 2022 as the "start" of the venture recession:

• Private valuations should bottom in Q2 2023
• I'm hesitant to forecast the heydays of 2021 ever returning, but if they did, it'd happen around Q4 2024. I wouldn't hold your breath though

Coming to terms

Again, let's focus on a 1% decline in the Nasdaq:

Financings follow an interesting "sine wave" recovery in the aftermath of a plunge in the Nasdaq. Again we see more severe impact in the later stages: The quarterly volume of Seeds and As drops by 0.75-1% for each 1% decline in the Nasdaq. Late stage deals sink 2%.

One ray of hope – financing activity seems to come back stronger after a venture recession.

Pulling out that scribbled envelope again, these numbers suggest early stage deal activity will drop 20-25%; growth and late stage deal volumes should slide about 50%.

Again, the data bears this out:

Remember, the venture deals should only take about a year to hit their low, so activity should stabilize in or around Q1 2023, notably ahead of valuations.

As I've said, these are rough estimates. All the typical caveats apply: "this time could be different," etc.

Conclusion

The bubble burst, but the pain won't last forever.

The historical relationship between the public and private tech markets lets us trace out the likely path of the venture recovery.

The first year is the hardest. Prices drop precipitously, in many cases, much further than their public tech comparables.

Things get better from there, with deal activity back on track after another year or so. Prices, however, remain depressed a little while longer.

All-in-all, it's a three year odyssey from start to finish.

Soaring valuations over the last decade inflated financing volumes.

As a result, "real", price-adjusted funding growth looks quite different from unadjusted growth, similar to traditional economic measures like GDP.

Adjusting for rising valuations, real venture funding at the early stage is only growing at half the unadjusted pace.

At the later stages, valuation-adjusted venture funding isn't growing *at all*.

It's valuations (almost) all the way down.

Pre-Algebra

First, a quick math refresher:

• Total funding volume equals the number of financing rounds multiplied by the average round size
• Assuming a standard equity round, average round size further separates into valuation multiplied by dilution (the proportion of shares sold to investors)

That gives us three components – deals, valuations, and dilution:

This framework is simple but powerful.

With it, we can explain the individual sources of growth in the venture funding ecosystem, cumulating into overall funding growth.

Algebra II

Let's walk through each component one-by-one.

I'll plot the growth over time of each component since 2010, leveraging data from Pitchbook for all venture deals completed in Western markets through 2021.

Importantly, I'll plot the cumulative log growth:

• This makes the components additive rather than multiplicative, which will help the analysis later
• This also means you can read compound annual growth rates (CAGRs) off of the y-axis. Just take the value on the y-axis and divide by 11, which is the number of years this analysis covers

Let's start with growth in funding volume by stage:

• Across stages, annual venture funding grew by 2.5 to 3 log points, with CAGRs ranging from 23-27% CAGR (again, just divide the log growth by 11)
• The earliest (Seed) and latest (Series D+) stages grew fastest

Next, let's dive into each of the components.

Deal counts have ballooned across all stages, but the Seed stage stands out:

• Seed deals grew the most by far over this period, 2.1 log points or 19% on an annual basis
• On the other hand, Series C deals and later only grew ~0.75 log points, a 7% CAGR
• As and Bs were somewhere in the middle, growing 10% annually over this time

It should be noted: that Seed deals grew so much faster than the rest implies the "graduation" or "survival" rate of startups fell materially, as they've had more than enough time to mature.

More early-stage startups has not led to many more late-stage startups.

We'll revisit this.

Next up, valuations:

• Unlike deal counts, which grew the most at the early stage, valuations grew the most at the later stages
• Series C valuations grew at a ~20% CAGR, while Series D+ grew at ~24%
• Series As and Bs again formed the middle of the pack, growing 1.75 log points or 16% year-over-year
• Seed valuations "only" grew 10% annually, which is still exceptional if you think about it

Lastly, let's look at dilution. Note again these numbers are in log points (not percentage points):

• Dilution has fallen in every stage since 2010
• Dilution at the Series A and C has fallen the most, while dilution at the Series B has fallen the least
• Dilution evolved similarly across stages through 2019, but for whatever reason, VCs over the last few years were much less desperate to "get their ownership" in Series A, C, and D+ rounds

Note that falling dilution negatively impacts funding volume since less equity gets sold.

Multivariable calculus

And now, the main event.

For every financing stage, let's aggregate the change in deals, valuations, and dilution to explain the cumulative growth of funding:

• At the Seed stage, most funding growth came from more deals getting done. Valuations rose too, but deal count explains most of the overall funding growth

This seems healthy – growth in funding should ideally come from growth in the total number of deals.

• It's a completely different story at every other stage: valuations explain most of the growth in funding at Series A and later.
• At the later-stage, valuation inflation explains nearly *all* growth in venture funding over the last eleven years

In contrast to the Seed stage, this seems unhealthy: growth in the majority of the venture ecosystem (at least as measured in dollars) over the last decade was primarily driven by rising valuations.

Now, if you've spent a lot of time staring at data like I have, you'll know that these sorts of analyses can be very dependent on the starting year (in this case, 2010).

Just to be safe, we can avoid privileging any particular year as the starting point by instead calculating the variance in funding levels over time rather than the growth. Variance decomposes the same way as growth, but this time there won't be a "base year" affecting the results.

Do this, and the conclusion is the same – year-to-year variation in valuations accounts for most of the variance in funding volume over time at the later stages:

• At the seed stage, two-thirds of funding variance over time is accounted for by deal volume and one-third by valuations
• At Series A and B, 65-70% of funding variance is explained by valuations
• At Series C and later, valuations account for more than 80% of the annual variance in venture capital funding

In economics, when quantities and prices rise together, you can be fairly sure it's driven by growth of demand, or equivalently, you know the market in question is supply constrained:

• The supply of startup equity at the later stages is constrained; there's only so much of it to go around
• On the other hand, investor demand for venture assets has exploded, especially at the later stages where companies have been significantly de-risked

In this sense, the valuation inflation we've seen "comes from" the incredible growth in demand and lack of supply of startup equity.

Real analysis

Here's another trick – if we go back to our decomposition and divide both sides by valuation, we get the following:

In other words, we can adjust funding volume by the average valuation at which these financings occurred to get a "valuation-adjusted" funding metric. This is analogous to how economists "deflate" nominal GDP by inflation (e.g. CPI or similar) to arrive at "real" GDP.

It's also the result of multiplying deal count each year by average dilution, i.e. the total equity bought and sold in the venture market each year.

This lets us measure "real" growth in venture funding rather than dollar-based growth, which conflates deal activity with valuation movements.

Compare this "real" funding metric to the "nominal" one I showed earlier:

Wow.

Real venture funding is growing much, much slower than you think:

• At the Seed stage, real funding growth has a 17% CAGR vs. 27% for raw funding dollars
• At the Series A and B, real funding grew at a 7% CAGR vs. 23% for nominal funding, one-third the rate
• Real funding was flat for a decade at the later stages before finally ticking up in the heydays of 2021. Real annual growth is only 2% and 4% at the Series C and D+ respectively, vs. 23 and 27% without adjustment

I want to state this as directly as possible: there was no growth in real late-stage funding activity for a decade until the bonanza of 2021.

Said differently, no more late-stage equity traded hands in 2020 than in 2010, measured in terms of points of cap table ownership.

Now you're ready for Economics 101

This data contradicts the common narratives of the past decade of growth in the venture ecosystem.

Venture capital is about two things, ventures and capital.

Capital has been in excess supply the last decade, but ventures haven't.

Valuations can't rise forever, so over the long-run venture capital can't grow much faster than the number of ventures themselves.

In a way it all makes sense:

• By the time startups graduate to the later stages, they are real companies, typically with meaningful revenue, operations, and headcount
• Accordingly, their total count should grow roughly in line with the real growth rate of modern, industrialized economies. We wouldn't expect an order of magnitude difference, otherwise late-stage startups would overtake the whole economy

In other ways, it's highly concerning:

• The supply of late-stage startup equity hasn't risen to match investor demand
• Perhaps this explains why late-stage funding seems so anemic right now - activity has reverted back to the zero-growth trend, which feels quite slow relative to the flurry of 2021 deal-making

The venture ecosystem is supply-constrained – there isn't nearly enough startup equity out there to satisfy investor demand.

Additional capital drives opportunistic company formation at the Seed stage. However, the additional capital doesn't improve survival to the later stages – it simply drives prices up for the remaining companies.

I've long felt this but never had the data to back it up.

Now I do, and I think it's a big problem.

• It's predictable, making the business much easier to forecast
• It's persistent – over or underperformance today reverberates far into the future

Contrary to popular belief, SaaS companies do not in fact "pull forward revenue." A beat today puts the company on a permanently higher trajectory. Revenue doesn't mean revert later.

Likewise, deals might "slip," but there's no "catch up growth" – a missed quarter doesn't get made up for next quarter. A miss today predicts lower revenue for years to come.

Diamonds, beats, and misses are forever 💎.

Long vs. short

Public market investors are often criticized for being too "short-term focused."

Politicians and CEOs alike love to hate on investors for overreacting to current business results, bidding up the stock or kicking off a fire sale depending on how quarterly earnings play out

A widely-held view among Washington policymakers, corporate executives, the media, and the public is that frenzied, short-term stock market trading has coupled with Wall Street’s unquenchable thirst for immediate results to disrupt US firms and badly hurt the economy – How Big Is the Problem of Stock-Market-Driven Short-Termism?

Baked into this criticism is a view that the long-run is hard to influence – fluctuations in the short-run tend to mean revert over time. One's view of a company should change only slowly and in the face of extraordinary evidence.

In defiance of this heavy-handed finger-wagging, there's an alternate view that short-run performance matters and is informative about the long-run. The future is bound to the present, and we should decisively update our forecasts in light of new information.

In summary:

• Long-termists believe revenue will eventually revert back to some long-run trend based on the fundamentals of the business, its market opportunity, etc. Short-term fluctuations are noise and should be ignored. Quarterly beats and misses should have little, if any, impact on fundamental valuation analysis.
• Short-termists (though they would never refer to themselves as such) believe the present moment matters and says much about the future. Short-term performance is a valuable signal that should factor into valuations. Mean revision dominates mean reversion.

Who is right? And how would we know?

A random or deterministic walk down Wall Street?

There's a simple way to resolve this dispute. We only need to rephrase the positions of the long/short-termists, and the resolution will be clear.

Long-termists think today's revenue fluctuations poorly predict future revenue. They don't often phrase it that way, but that's the core underlying logic behind the belief in a stable, long-run trend.

This has multiple important implications.

For one – revenue can be "pulled forward," but this will tend not to influence the long-run trajectory of the company:

Adobe was downgraded to neutral from buy at UBS... Analyst Karl Keirstead said after speaking with 14 large enterprise IT executives and services partners of Adobe, he's worried that spending was pulled forward in 2020 and 2021, which will pressure its growth rate this year. (Link)

Further, a missed quarter today can be made up for next quarter. Missing the target leads to revenue in the future quarters to be slightly higher, as revenue reverts back to trend. You could call this "catch up growth."

Meanwhile, short-termists think today's revenue movements predict future revenue. One's long-term revenue forecast should be quite sensitive to beats and misses in the present day.

So, pulling forward revenue doesn't come at the cost of future revenue. Similarly, missing your target today doesn't mean you're any more likely to hit tomorrow's target. In fact, you're less likely, since you're now on a lower revenue path.

The clean and simple test?

Simply run a regression of future revenue on current over/underperformance! The coefficient tells us to what extent our estimates of future revenue should shift in response to strong or weak results today:

• A coefficient near or greater than one tells us future revenue rises/falls by at least $1 for a $1 beat/miss today (the short-termists win)
• A coefficient well below one means forward revenue is insensitive to today's performance and tends to mean revert (the long-termists win)

Back to the future

The chart below plots revenue growth over four quarters against revenue growth today for 35 public software companies, demeaned by the respective average rate for each company. Positive values mean revenue came in higher than normal, and vice versa:

Future and current revenue over/underperformance are positively related. The coefficient is 1.16, implying a $1 beat today forecasts a $1.16 beat four quarters from now. The exact reverse is true of a miss – a dollar of missed revenue this quarter lowers our expected revenue in four quarters by $1.16.

Additionally, the R^2 is decently high – 0.49, so about half of the variation in expected revenue four quarters out can be explained by revenue performance in the current quarter.

One quick aside since I know what certain folks are thinking here: I am not merely saying that current revenue predicts future revenue. I'm saying changes in revenue predict changes in future revenue.

So far, the short-termists seem to be winning. But one year is not a long-time – perhaps mean reversion takes longer? Let's check by extending the analysis to eight and twelve quarters out. That's too crowded for a single scatterplot, so I'll summarize the results and only show the coefficients at each horizon:

If anything, the case for short-termism only gets stronger. The two-year revenue forecast shifts by $1.25 for a $1 beat/miss today. The three-year forecast changes by $1.43.

Even three years out, we see no evidence of mean-reversion among the typical public software company. Beats and misses permanently shift the trajectory of the company.

Now, dollars are a nice unit of account, but they're admittedly hard to contextualize. A $1 beat/miss that turns into a $1.43 beat/miss three years out could be more or less meaningful depending on the scale of the company and its growth rate. Many of these companies were likely to be much larger in three years anyway.

Let's do the same analysis with percentages instead. What impact does a 1% beat/miss have on future revenue, also in percentage terms?

Persistence persists, but this tells a slightly different story. Revenue three years from now is still 0.85% higher/lower than it would have otherwise been.

My interpretation: the additional (or lost) revenue from a beat (or miss) grows at a somewhat slower pace than the remaining revenue base, so we see some convergence. In other words, "surprise" revenue doesn't grow as fast as "expected" revenue.

You could frame this as slight mean reversion. Personally, I'd say the short-termists still have it.

Working as intended

Why does this happen? Why is the future so sensitive to the present?

Rather than a surprising phenomenon, I see this as the defining characteristic of subscription business models.

To say revenue "recurs" is merely to say revenue today generates revenue tomorrow. Said differently, a good test of subscription revenue quality is the degree to which it persists and predicts future revenue.

In that light, these results are expected. If a change in revenue today didn't predict a change in revenue tomorrow, it'd be hard to call it recurring.

For some intuition, look at those cohort revenue charts that have become so popular among public software companies (at least among those with good retention dynamics to show off):

Missing a quarter means losing a slice of the cohort stack. Assuming positive net dollar retention, that slice would have grown over time; the opportunity cost of weak performance grows over time.

Ironically, a miss for a company with high retention hurts more than one for a company with low net retention, since the high retention company has more (future revenue) to lose. I examined this phenomenon from a slightly different angle a few years back:

Both in theory and in practice, better retention drives higher volatility – "High Retention = High Volatility"

This is why SaaS beats and misses are so consequential.

If investors think a miss or beat is likely to stick, that will meaningfully impact their valuation views, as it must. On the other hand, the stock shouldn't move much if short-term performance reflects merely temporary dynamics in a company's go-to-market engine.

That SaaS valuations tend to react so strongly implies investors do believe these GTM gyrations are permanent and must be reflected in future projections. It's possible there's some mean reversion happening under the hood, but this effect is totally swamped by the magnitude and persistence of the beat/miss itself.

Revision of the mean

A bad reaction to this analysis would be: your sales leader says the quarter came in light because some deals slipped, you look them in the eyes and confidently tell them "there's no such thing as a deal slipping," and point them to this essay.

No. That's not what I'm saying at all. (Please do send them this essay though!)

At the level of individual sales, deals slip from one quarter to another all the time. But at the level of aggregate revenue that doesn't seem to matter. Revenue won't be higher next quarter simply because revenue came in low this quarter. Every quarter is, for the most part, a blank slate.

Likewise with pulling forward revenue – it happens, but on average you can't find it in the data. COVID is a good exception to this – companies that benefitted from the transition to remote work like Zoom or DocuSign are now slowing, reverting back to their long-run trend line:

While DocuSign beat revenue expectations last quarter, full-year guidance came in far lower than expected. DocuSign… was perceived to be a "COVID winner." There are now fears that the company merely pulled forward years of sales over the course of the past two years, and that its revenue growth trajectory will be lower going forward. (Link)

While Coupa's upside certainly isn't at risk long term, it does appear some of its future growth was pulled forward during the COVID-19 crisis. (Link)

Unfortunately for Twilio, it belongs to the “high beta growth” club, which began a secular decline early last year. The San Francisco-based Cloud Communications company boomed from the work-from-home trend... But, like so many other tech companies, it pulled forward too many gains, setting itself up for a nasty re-pricing. (Link)

Note: this analysis only includes software companies. Businesses with more transactional business models probably see less persistence and more mean reversion. Think about companies like Peloton:

• Despite having a subscription service, the vast majority of revenue was in physical bike sales that are one-time in nature
• Peloton really did pull forward demand, which left little market left to grow into by the time things returned to normal

However, if you have significant recurring revenue, mean reversion is much less relevant.

This analysis continues a line of thinking I touched on in a prior piece exploring COVID's impact on software companies:

… mean reversion is a strong force, but COVID was stronger, especially on the downside. Once knocked down, the typical software business never got back up…

Mean revision is at least as important as mean reversion. Not only are software companies below their pre-COVID trend, the trend itself has changed for the worse – COVID Hurt Most Software Companies

For software companies, every day is Day 1 or, at least, dependent on it.

Sites like layoffs.fyi track layoffs in gory detail, as do journalists and tech writers:

Large layoff rounds in tech, sadly, are still on.

Olive AI let go ~30% of staff. They were one of the biggest unicorns in the Midwest (valued $4B)

Capsule (digital pharmacy startup, valued >$1B) had large layoffs. I'm still not sure of the exact % here (feel free to DM)

— Gergely Orosz (@GergelyOrosz) July 20, 2022

However, it's slow hiring, not layoffs, that most drives unemployment.

Layoffs don't tell the whole story.

Stocking up

Employment and unemployment are both "stocks".

Stocks are quantities measured at a point in time that change over time. For example:

• The volume of water in a lake
• The stock (hint hint) of inventory in a retail store
• Cash on a company's balance sheet

On the other hand, "flows" are rates of change of a particular stock. For example:

• Water from various rivers flowing (hint hint) into a lake
• Shipments of new inventory
• A startup's monthly burn rate

A single stock can have multiple associated flows, and the sum total of these flows determines how the stock changes over time.

In the labor market, workers flow between employment and unemployment (stocks) at various rates (flows):

In equilibrium, the various stocks of a closed system are stable. However, if the individual flows change, the related stocks change too:

• During recessions, flows from employment to unemployment (job separation) increase, while the flows out of unemployment into employment (job finding) decrease
• Vice versa for recoveries – people find jobs more quickly and fewer people get laid off in the first place

In this way, variation in job separation and job finding "explain" changes in unemployment.

As the labor market cools, hiring freezes

We commonly associate unemployment with layoffs. During recessions, we imagine heartless employers suddenly laying off droves of workers:

While this story feels intuitive, the data doesn't bear this out.

The below chart plots the U.S. "employment exit rate" (a euphemism for layoffs or voluntary quitting that ends in unemployment) along with the unemployment rate itself. Think about the exit rate as the share of all employed workers who leave employment each quarter:

Employment exits jiggle up and down at a much higher frequency than unemployment, and the timing of those short term fluctuations doesn't align well with granular movements in unemployment. Over longer periods though, they do seem loosely connected.

Now let's look at the other piece of the puzzle. Here, I've swapped out employment exit for the "job finding rate," i.e. the proportion of unemployed workers who find a job each quarter:

The lines perfectly mirror each other.

The fluctuations closely track the state of the economy. When job finding plummets, unemployment rises at the same time. As the economist Robert Shimer put it:

… a decline in the job finding rate… contributed to every increase in the unemployment rate during the post-war period – Reassessing the Ins and Outs of Unemployment

As the job finding rate slowly recovers, so does unemployment.

So, rather than massive layoffs defining recessions and spikes in unemployment, the more powerful explanatory story is one of hiring freezes – employers simply stop hiring during recessions.

Per our stocks-and-flows model, the stock of unemployed workers grows and shrinks primarily due to variation in the outflow from unemployment to employment rather than the inflow of employment to unemployment.

In other words, hiring freezes are the real story, not layoffs.

Hiring or firing?

What's the relative importance of hiring freezes vs layoffs?

Here's how we can estimate this:

• Let's say we hold variable A constant and allow variable B to move around. If unemployment still moves around a lot then we know most of its variation is driven by variable B
• Similar logic applies in the opposite scenario (allowing A to fluctuate while holding B constant)

This table quantifies the relative contributions of job entry and exit rates to overall unemployment variability by calculating counterfactual scenarios in which either variable is held constant while the other is allowed to fluctuate. The first row holds job separation constant and the second holds job finding constant (Source):

The result?

... since 1987, including the recessions in 1990–1991 and 2001, and 2008–2009, the job finding rate accounted for virtually all fluctuations in the unemployment rate – Reassessing the Ins and Outs of Unemployment

The job finding rate explains the vast majority of variation in unemployment, and the proportion has only increased over time.

From 1987 to 2010, variation in the job finding rate explained 90% of the variance in unemployment, while employment exit only explained 10%.

Here's Shimer again:

The bulk of the reason unemployment rises and stays high throughout the recovery is because the unemployed workers stay unemployed for longer…

This is precise, quantitative evidence for the overwhelming importance of hiring over firing in explaining unemployment.

Hyper-(de)growth

If the prior data seemed counterintuitive, the following example shouldn't.

Imagine you're a high-growth startup with big ambitions to grow the employee base in the coming year:

• You are currently 100 employees
• You want to hire an additional 100 people over the next year
• We ignore voluntary attrition just to keep the numbers simple

Suddenly, the environment changes:

• Economic conditions worsen, capital gets tighter, you start to miss plan, etc
• You need to make cuts, both in the current employee base and the future one
• So, you lay off 20% of the team

This is extremely painful, demoralizing, and controversial, only made easier by the fact that everyone else is doing it too:

My sense is that it's partly - "if we layoff people at the same time everyone else does, it'll look more like bad market conditions were to blame and less like we've ran this thing poorly"

i.e. if everyone stampedes at the same time, no one is to blame for trampling anyone

— Nnamdi Iregbulem (@whoisnnamdi) May 5, 2022

You also institute a hiring freeze, which only seems fair after laying off so many team members:

3. Immediately re-hiring after layoffs paints the picture of leadership of what it is:

Incompetent.

Couldn't even figure out what % of people to lay off.

— Gergely Orosz (@GergelyOrosz) July 8, 2022

It doesn't take an HR savant to realize you aren't going to hit your original 200 headcount target for next year.

Now do the gap analysis – between layoffs and slowed hiring, which is more responsible for missing the original headcount target?

It's the hiring freeze of course!

• You laid off 100 x 20% = 20 people
• Meanwhile, you cut the hiring plan from 100 to zero

In effect, you "fired" or "laid off" 100 future employees while only laying off 20 current ones:

As one founder friend whose recent layoff was leaked on social media told me:

Sure, the layoff news leaked, but when we modeled this stuff out we realized it's the reduced hiring that would save us the most cash.

A hiring freeze is a much bigger story, even if the media only reports layoffs:

• Internally, management knows the hiring freeze will have a larger impact on team velocity (and, frankly, cash burn) than the layoffs.
• Yet externally, everyone will focus on the layoffs.

This misses the raging forest fire for the merely singed trees.

A new perspective

Recognizing the importance of hiring freezes should change your focus regardless of whether you're a founder or an employee.

As a tech worker:

• The biggest risk you face in an economic downturn is not so much losing your current job but instead being unable to find a new one (due to hiring freezes)
• In addition, as you assess the viability of various startups, pay equal, if not more, attention to slowed hiring at the company than highly visible layoffs
• When interviewing, ask about the hiring plan and how it's changed in recent quarters

If you're a founder:

• Slowed hiring is the real sign of weakness and vulnerability in your competition
• Even with no layoffs, your competitors could be in a dire financial straits, necessitating a complete freeze in hiring
• If you're well-capitalized, your biggest opportunity during a downturn isn't picking up people who were laid off (though you should definitely consider this) but rather hiring the people your competitors would have wanted but now can't due to hiring constraints

Lay off the layoffs

Layoffs are a noisy estimate of company performance:

• Layoffs are tainted with worries about public perception and internal morale, making otherwise troubled companies hesitant to do them
• Companies conduct layoffs during downturns in part because everyone else is doing it, and they know they won’t look as bad if they move with the herd

Hiring freezes are a clean signal of a viability and confidence of management:

• Few companies would cut back on their hiring plans simply to fit in
• If anything, knowing others are pulling back makes companies more aggressive in hiring

That said, hiring freezes are opaque from the outside:

• Outside of large organizations, most never publicly announce hiring freezes
• In many cases, hiring freezes aren't even openly discussed internally outside of the core executive team and the board of directors

Keep in mind both the seen and the unseen. It's easy to focus on the shock and awe of layoffs, but you miss so much in doing so.

Layoffs aren't even half the story – they're 10% of it.

While many claim COVID accelerated technology adoption and digital transformation, this didn't show up in the revenue of most software vendors.

In fact, the typical public software company remains 10-12% below its pre-COVID revenue trend, and the gap is only widening.

COVID put most software companies on a permanently lower growth trajectory.

A few case studies

The COVID shock generated divergent outcomes across software companies.

Let's focus on three examples to see what I mean: Zoom, Cloudflare, and Splunk.

The solid lines below represent the (log-transformed) quarterly revenue of each company. The dashed line represents a constant-growth linear trend based on their two-year pre-COVID revenue trajectory. Since the scale is in logs, straight lines imply constant quarter-over-quarter growth:

We see some interesting behavior as we cross the dashed line, which represents Q4 2019, the last quarter before COVID struck:

• Zoom immediately takes off, "zooming" above its pre-COVID trend as millions move to online communication. The trend eventually catches up
• Cloudflare is totally in the clouds, floating along like nothing happened, perfectly matching its pre-COVID trend of constant, exponential growth
• Splunk exhibits strong seasonality, so the initial decline is no surprise. However, it never recovers to trend, even two years later. Splunk got dunked

These symptoms run the gamut from manic to lethargic. Clearly, COVID was not a tide that raised all boats.

Mean reversion, or mean revision?

We can repeat the above analysis for all public software companies with enough data to establish a two-year pre-COVID trend. Given the different revenue scales, for visualization purposes let's normalize by dividing each company's actual performance by its extrapolated trend. This gives us the relative revenue performance of each vs. trend:

Each line represents performance vs. trend for one of fifty different public software companies.

With this we can begin to see the rich variety of post-COVID outcomes. Let's highlight our three case studies once more:

• Notice how dramatic and unrepresentative Zoom is among all software companies. Zoom peaks at ~150% above trend before gliding back down to Earth
• On the other hand, Splunk comes out as one of the worst performers, among others like Alteryx and Benefitfocus

There's a lot going on here, making it difficult to get a sense of the underlying distribution. Let's flip the axes and look at it a different way:

The above chart plots histograms of performance relative to trend across public software companies from Q1 2020 onward:

• Upon impact, performance is normally distributed, centered somewhat below zero
• Over the subsequent quarters, the distribution shifts left and widens, implying worse outcomes and greater dispersion between companies
• By eight quarters out, performance is unambiguously below trend, while variance continues to grow

Notice the skew: a handful of winners among a majority of losers.

Another way to look at the data is to calculate the proportion of software companies that climbed above their pre-COVID trend over time:

• Initially, only a quarter of software vendors were above trend
• One year out, one-third of companies have returned to trend
• The proportion stabilizes thereafter. Mean reversion is over

This is interesting. As dispersion grew in the first year of the pandemic, some companies returned to their pre-COVID trend. After these initial quick recoveries, however, mean reversion completely shuts down. The last lifeboats cast off, and if you aren't onboard, you're ngmi.

A key point I want to drive home: mean reversion is a strong force, but COVID was stronger, especially on the downside. Once knocked down, the typical software business never got back up.

Weakness, in numbers

Finally, let's summarize those performance distributions by their means and medians:

This is what I've been building up to – COVID materially impaired the typical software company, knocking it off trend:

• On average, software companies were down ~5% relative to trend 4 quarters out and down ~10% 8 quarters after impact
• The median dampens the impact of outliers like Zoom. Notice, median performance is even worse, ~7.5% at 4 quarters and ~12.5% at 8 quarters
• Meanwhile, underperformance relative to trend is still increasing. The gap between actual revenue and trend is growing ~4% per year

Thus, COVID's impact was not transitory: COVID permanently reduced the annual growth rate of public software companies by 4 percentage points.

Mean revision is at least as important as mean reversion. Not only are software companies below their pre-COVID trend, the trend itself has changed for the worse.

Conclusion: Conflation

I think there's been a massive conflation between:

1. the equity returns of software companies during the first year and half of COVID (exceptional, but temporary),
2. the financial performance of a select few COVID beneficiaries like Zoom (again, exceptional, but transitory), and
3. the business performance of the broader software market (poor, and durably so)

The temporarily exuberant stock prices of most software companies distracted from the toll COVID took on business performance. While investors got high on the Fed's supply, businesses themselves were silently suffocating.

This corroborates the findings of my last essay, where I explored how the mysterious "GDP factor" explains some the the "dark matter" in software valuations:

Despite some claims to the contrary, investors believe software companies are quite sensitive to the broader macroeconomic environment. In fact, it's the variable that correlates most highly with valuations after accounting for the individual financial performance of each company – The Dark Matter of Software Valuations

Here again, I find that software companies are sensitive to economic conditions, even in a best case scenario that supposedly favored them. "Digital transformation" can't come quickly enough it seems.

So let's set the record straight: COVID left a nasty scar on us all, software companies included.

Investors tend to focus on revenue growth and profitability as significant variables for valuing software companies, but those metrics only explain about half of the overall variation in valuations.

This leaves a vast amount of "dark matter" in the software universe – variation in valuations that goes unexplained.

I want to explore this vast darkness.

Who knew there was such skew?

Note: this analysis only includes public software companies that traded continuously from January 2020 to May 2022. Companies that went public or got acquired during that period are excluded.

It's been said before, and it's worth repeating: averages don't tell the whole story.

You can begin to see why by plotting average revenue multiples against median multiples. They aren't the same.

In fact, the distance between them tends to fluctuate over time.

• In the lead up to COVID they were quite close
• Then, they grew apart
• Only recently, they've gravitated closer again

A single line at best summarizes the data and invariably misses important details.

Let's zoom in:

Like software monetization, software valuations are fat-tailed:

• Most companies have unremarkable valuation multiples
• A few, however, skew well to the right, creating a long tail of multiples

This tail skews the distribution of multiples and increases the variance between public software companies. I'd imagine the situation is even more extreme among private software companies.

That variance changes over time, rising and falling:

Note: for statistical reasons I won't explain here, the remainder of this essay will refer to log-transformed software multiples instead of their raw values.

I'd love to explain this dynamic.

It's at this point that any good software investor turns to ol' reliable: regressions of valuation multiples on revenue growth and profitability:

We see a similar, if subdued, pattern in the variance that can be explained by financial performance. But notice how much variance our standard regression leaves on the table, unexplained.

Growth and profitability do not perfectly predict valuations.

I call this unexplained variance, the "dark matter" of software valuations:

Growth and profitability never explain much more than half of the variation in multiples:

What's hiding down in the deep, dark depths of the software universe?

Let's grab our telescope and zoom in once again.

Stranger things (and valuations)

We know that growth and profitability are imperfect predictors of valuation.

Let's remove their effect and focus now on the remaining, residual, variance in valuations. This means that we'll focus on relative rather than absolute valuations – relative to what growth and margins would predict, that is.

Here's how it looks:

The fat tail is less severe but still persists, even after accounting for financial performance.

Many companies hover near zero, which is to say their valuations are perfectly predicted by growth and profitability, but a number are quite "overvalued" on the basis of their financials alone.

This gap between expectations and reality is quite persistent. Over/undervaluation does not meaningfully mean-revert, even over multiple years:

• A 50% overvalued company in January 2020 by May 2022 still traded ~40% above the valuation implied by its growth and profitability
• A 50% undervalued company continued to sag ~35% below its predicted valuation by the end of this period

Some did see major revisions:

• Shopify, Okta, Coupa, DocuSign, and RingCentral all fell back down to Earth after flying high for some time
• Cloudflare, Appian, Alteryx, MongoDB, Atlassian, Anaplan, and Guidewire all ended up much higher than where they were pre-COVID

In addition to individual companies, the overall distribution of unexplained "valuation inequality" held in place the past few years, with some fluctuations:

• Investors value the 90th percentile company ~75% more than you'd expected from its growth and profitability
• Meanwhile, they peg the 10th percentile company ~50% lower than you'd predict.
• In terms of ratios, 90th percentile software vendors are worth 3-5 times their 10th percentile peers.

Remember, we've already accounted for revenue growth and cash flow, yet we still see massive variability:

• On the revenue side, among companies growing ~50% annually we see valuation gaps as large as ~70% (i.e., same growth, yet one company is worth 70% more).
• It's even worse for software companies growing closer to 20% year-over-year, where the gaps are even bigger: the most valuable company is worth ~2.5 times the least valuable.
• On the free cash flow side, if we look at companies within +/- 10 percentage points of breakeven, we see companies worth five times as much as the least valuable

Needless to say, the variation in software valuations is extreme, and that's after factoring out what we'd already expect based on financial profiles.

Darkness into light

I hope I've driven the point home that growth and profitability aren't everything.

OK, but what is this dark matter stuff?

Clearly the most obvious metrics fail to account for a sizable share of the variation in valuations among software stocks. Some sort of unobserved or latent characteristics drive the remaining differences.

Principal Component Analysis (PCA) is a great way to identify such latent factors. It's a statistical technique for summarizing the key informational content contained within a larger data set.

A full explanation of PCA is beyond the scope of this essay. Just know that, like alchemy, PCA lets us put dark matter in and get some nuggets of gold out.

When I run PCA, here's what I find:

PCA spits out two latent "factors" that highly explain our up-till-now unexplained valuation puzzle.

Unfortunately, these factors don't come out of the box with much explanation. We must interpret them manually.

After staring at "Factor 1" for a while, it hit me – the line looks eerily similar to the path of the U.S. economy over that period!

Folks, we have a winner:

Beautiful.

Our first hidden factor is not a particularly mysterious one.

Despite some claims to the contrary, investors believe software companies are quite sensitive to the broader macroeconomic environment. In fact, it's the variable that correlates most highly with valuations after accounting for the individual financial performance of each company.

The latent factor tracks the COVID recovery quite well, in some ways even better than the official GDP statistic, which is only released quarterly – a significant delay.

The factor provides a daily, real-time estimate of economic performance, at least that portion which is relevant for software companies (also called nowcasting).

I had less success figuring out the second factor. If you have any ideas, please send them my way!

With PCA, it's customary to plot the data against the first two factors, with the X and Y axes representing each company's correlation with the factor:

• Companies on the right side correlative positively with the GDP factor, suggesting they've benefitted disproportionately from the COVID bounce-back
• Conversely, stocks hovering to the left had less to gain from the rebound

I could spend (and may have already spent) hours staring at this plot. Rather than continuing to play "Where's Waldo," here are a few clusters I noticed (Q: What did I miss?):

• Financial engineering: Developer-centric tooling and infrastructure like Datadog, Elastic, MongoDB, Atlassian, PagerDuty
• Easy storage, easy life: Storage companies Dropbox and Box
• Trust, but verify: Zero-trust and identity solutions like Okta, Ping Identity, Zscaler
• Call me maybe: Cloud-based contact center and communications companies Five9, RingCentral, 8x8

A step in the right direction

Now let's come full circle and incorporate the GDP factor into our previous regression while also allowing each stock to have different sensitivity to the factor.

Here's what we get:

The GDP factor helps explain some of the mysterious "dark matter" but not always the same amount:

• In the weeks before COVID turned everything upside-down, the GDP factor didn't explain much variation in valuations
• Then, as the pandemic began, the GDP factor rose in importance, granting significant predictive power
• As the first positive results of the mRNA vaccines come out, it recedes again, and for about three quarters the GDP factor doesn't explain much over and above individual financial performance
• Finally, as the COVID recovery finally stalls out, the GDP factor rises in importance again

Here's how it looks in R-squared terms:

Still a lot of dark matter out there, but we've made great progress:

• At it's highest, the combination of (1) individual financial performance and (2) sensitivity to economic conditions explains ~80% of the cross-sectional variance in software valuations
• That said, there remain times when it explains no more than 50% of valuation variance

I'm fascinated by these dynamics, especially the middle period where, for whatever reason, economic sensitivity temporarily stopped mattering for how software companies were being valued, at least relative to one another.

Conclusion

So we've managed to chip away at some of the dark matter hiding among software valuations.

It's often said that the long-term growth trajectory of cloud companies is relatively secular, which is to say it's independent of general economic fluctuations.

It turns out software companies *are* sensitive to broader economic conditions in the short-run, at least in eyes of public investors.

The relevance of this sensitivity rises and falls over time:

• In periods of economic uncertainty (e.g. early pandemic, Q2 2022, etc), the GDP factor begins to matter quite a bit. Public investors care who gains and loses from a weakening economy.
• Meanwhile, when times are good, investors throw caution to the wind and ignore the economic sensitivity of software companies.

More sophisticated factor modeling could yield even more insights. No one pays me to do that though, so I think I'll move on.

If you do have thoughts on that second mystery factor, shoot me a note!

Suddenly, a PowerPoint deck isn't enough. Traction and monetization matter.

Your startup might have more of one than the other, and that's OK.

But less of one means you need more of the other. That's just math.

How much more? It turns out – more than you think.

Low monetization requires extraordinary traction, and vice versa.

2 + 2 = 4

In addition to factors like team quality and market size (which also matter at the pre-seed/seed stage), investors evaluate Series A startups on their traction and monetization.

By traction, I mean some rough mix of:

• bottom-up adoption
• free usage / free trials
• organic, inbound interest in the product
• proliferation of an open source project
• downloads
• a vibrant, growing community
• product-led growth

By monetization, I'm loosely referring to:

• revenue (duh!)
• customers
• top-down / direct sales
• production deployments
• pilots / POCs (proofs of concept)
• qualified pipeline

Note: monetization is a catch-all for activities and metrics that tend to correlate with revenue, so a company with zero revenue can still have a decent monetization "score."

If traction and monetization are collectively high enough, you will successfully raise your Series A round.

You can even make a little equation out of this:

There's a threshold, a sort of activation energy, required to successfully raise a round at the early-stage. Hit this bar and you're in the clear. Miss it, and you're not:

Let's make up some numbers.

Say the threshold is "4". You'd need some combination of traction and monetization to get to a sum of four:

• A score of two on both traction and monetization would work. We could call that "moderate traction and moderate monetization", or (2, 2)
• But a (3, 1) combo would also work – perhaps "high traction, low monetization", and so on.

As long as you can get to a total of four, you're good.

2 x 2 = 4

What if our model is wrong though?

Imagine investors evaluate your startup not on the sum of traction and monetization, but rather the product. In other words, traction and monetization multiply, rather than add:

Through this lens, the marginal value of one factor depends on the other. Traction makes monetization more valuable, and vice versa. Monetization is more valuable when you have more traction.

This is a much better model for how venture investors think about startups.

Think about it:

• Strong traction makes additional monetization more valuable, since there's a preexisting, large group of interested, engaged users to sell into.
• Strong monetization increases the value of additional traction, since you know that traction will be monetizable in the future.

The multiplicative model has an important implication however – it disproportionately penalizes low traction and/or low monetization.

To successfully fundraise, low monetization requires extraordinary traction, and vice versa:

Here's an analogy – a stock that drops by 50% (OK, maybe more reality than analogy) must rise by much more than 50% to reach its original price level, since returns are multiplicative (50% x 150% is only 75%). Likewise, low monetization requires more than proportionally higher traction to make up for it.

(2, 2) still works, since 2 x 2 = 4. But the (3, 1) combination is no longer viable, since 3 x 1 only equals 3. If your startup is merely a "1" on the monetization scale, you now need to be a "4" on traction. 0.5 on monetization? You need to be an 8 on traction!

You can see where this is going.

3 x 1 = ☠

Unfortunately, founders fall into the trap of thinking low monetization or traction can be made up for with merely moderate amounts of the other factor. This is wrong.

Many founders think investors will evaluate them with the linear, additive model. While many investors are quite linear in their thinking (sorry, had to do it), the multiplicative perspective is the more accurate one.

This clash of reality vs. expectations creates a "no man's land" between the additive and multiplicative perspectives: founders think they'll be fine when they go out to fundraise, only to be rudely mugged by the valley reality:

I've noticed many startups in the space between the linear and convex curves attempting to raise Series A rounds lately. This invariably goes badly. The founders are surprised by the lack of investor interest and end up taking much longer to raise their round, at a lower valuation than expected.

Some never raise at all.

Quick maths

Avoid this needless suffering. Be honest with yourself – if you're lacking in either traction or monetization, be sure you're killing it on the other dimension.

I've made the point that low monetization requires extraordinary traction (and vice versa), but I haven't defined what "extraordinary" means. Frankly, it's subjective at best, and I'd be lying if I said there was an objective standard or rubric.

When you know, you know:

you can always feel product/market fit when it's happening – Marc Andreessen

… once you have product-market fit… [business] performance accelerates so meaningfully that any ambiguity goes out the window. You get it "in your gut." It becomes obvious, as nearly everything about the business gets better – Product-Market Fit is Lindy

If it is true that product-market fit yields much better business performance, then product-market fit should be relatively obvious, even with little data – Product-Market Fit is Lindy

Further, I've simplified the framework here and taken other factors like team, market, etc as given. Realistically, investors will evaluate those things too, complicating our math a bit:

That's a lot harder to visualize, but the same ideas apply. You need to really spike on one or more factors to make up for lower scores on the others. Get this math wrong, and you'll be in for a nasty surprise.

Rather than cram for the exam, delay your Series A until you pass the simple math test I've outlined here. Until then, keep practicing those multiplication tables!

For the best companies, raising large war chests along with the branding effect that comes with them can serve as an "unfair advantage" against competitors, compounding gains and maintaining pole position.

This unfair advantage often comes through the hiring channel, which is high-octane leverage for fast-growing companies:

The “10x” in “10x engineer” conveys leverage just as the “3x” in “3x S&P” does. As an investor, leverage can multiply the returns you receive from a great investment thesis. And similarly, if you’re a founder with a great vision, your superstar employees will help you realize it faster and better – Talent as leverage

For most however, a frothy funding environment means more competition for talent. If everyone is raising tons of cheap capital and plowing those funds into hiring, your job as a founder hasn't necessarily gotten easier, since presumably you'll be working that much harder to recruit against every other startup doing the same.

The capital "bottleneck" hasn't disappeared – it's moved to another part of the production chain. Funding gets easier; hiring gets harder.

Cheap money, expensive people

He deleted the tweet, but Mike Speiser once said:

When money is cheap, people are expensive. When money is expensive, people are cheap. #dilution

The idea that hiring gets harder when funding gets easier (and vice versa) has two foundational components. The first one is easy, so let's get that out of the way first.

When companies are cash rich, they tend to spend more (*shocker*). Startups raise funding in large part to expand their teams. When the entire startup ecosystem simultaneously raises tons of capital, the additional money sloshing around inevitably shows up in the W2s of startup employees and the "voluntary churn" column of board of directors presentations. Recruiters spam the LinkedIn inboxes of every warm body in sight, and the hiring market heats up.

That's fairly intuitive. The second reason is much less so – discount rates.

Go back to the Mike Speiser quote. What does it mean for money to be "cheap?" Financial theory introduced us to the concept of the "discount rate" or equivalently the required rate of return or cost of capital. Money is "cheap" when discount rates are low, implying that investors and lenders require only a small rate of return on their investments in order to be satisfied.

Typically, low discount rates are associated with high valuations, since the present value of a series of cash flows rises as discount rates falls. Thus we get the frequent gyrations of the stock market, which summarize this dynamic in a single number:

the large movements in the value of the stock market arise mainly from changes in discount rates and only secondarily from changes in the dividend or profit flow capitalized in the stock market – High Discounts and High Unemployment

It's quite natural to apply the notion of discount rates to the value of financial assets, like company equity. But why limit our thinking to financial assets?

The present value of talent

Discount rates apply to human capital too.

If you don't believe me, check out the following chart. It plots the unemployment rate over time along with the inverse of the detrended (i.e. accounting for the size of the U.S. economy) S&P 500 index (so down is up and vice versa). Notice the tight relationship between the two:

Figure 2 suggests that the stock market and unemployment respond to the same underlying forces, especially in the past few decades – High Discounts and High Unemployment

You might ascribe this correlation to general economic conditions rather than discount rates, but unemployment and output per worker don't actually track each other very well:

To see how discount rates apply to workers in the same way they do to financial assets, imagine the net value of a worker to a company being their productivity minus their cost, i.e. wage. This "job value" takes into account all future value the worker generates along with their cost of employment, in present value terms:

Job value tends to increase during booms, for two reasons:

1. Discount rates are low, which inflates present value calculations,
2. Productivity is more variable than wages, which tend to be sticky

Thus, the present value of productivity moves up and down by more than wages, driving job value up or down with the market cycle:

A higher discount lowers the job value… The present value of the wage moves in the same direction as productivity, but less than in proportion. In this sense, the wage is sticky. Because the wage falls less than productivity, the job value… falls more than in proportion to the fall in productivity – High Discounts and High Unemployment

Naturally, the higher the "job value," the more eager employers are to hire workers. We get cutthroat competition among employers during good economic times and workers being dropped like hot potatoes during bad times. When the present value of hiring an additional worker is high, employers clamor to grow their ranks.

It's possible to use economic data to estimate job value across the entire U.S. economy. Job value and the stock market move together in lockstep:

The similarity of the job value and the stock market value is remarkable. The figure strongly confirms the hypothesis that similar forces govern the market values of claims on jobs and claims on corporations – High Discounts and High Unemployment

The up and down movement of job value is incredibly consistent across industries too:

The strikingly similar responses in diverse industries strongly supports the hypothesis that an aggregate driving force dominates the movements of the job value at the industry level. This evidence points in the direction of aggregate forces such as rising discounts in recessions – High Discounts and High Unemployment

When hiring falls apart

When interest rates are low, leverage becomes more attractive. Although the venture capital world likes to think of itself as distinct from debt-laden private equity land, in fact technology startups engage in all sorts of leverage, especially during boom times. I talked about one form of it, preferred equity, in a prior essay:

In this way, startups become synthetically levered. No debt appears on the balance sheet, yet the returns of the common equity (any really anyone below the most senior preferred stock) get juiced by the lower cost of capital achieved via preferred equity – Schrodinger's Balance Sheet: When Equity Becomes a Liability

I referred earlier to the notion that exceptional employees are a form of leverage. Just like financial leverage, talent leverage allows a company to do things they wouldn't otherwise be capable of with existing resources.

To attract such special individuals, startups have to promise the world to these ambitious folks, a sort of "debt." These superstar employees wouldn't join otherwise – the cash compensation alone is rarely worth it. They must be promised quickly-appreciating equity, as well as the Silicon Valley cache and prestige associated with a successful startup. This is related to Keith Rabois' idea of "talent arbitrage."

This Silicon Valley "spin" peaks during market highs – frothy periods where slick storytellers paint increasingly fantastical and implausible visions for what their company will achieve. It's a competition to see who can lever up their fragile enterprise the most.

If things start to go south, the bubble bursts, and these 10x employees are often the first to abandon ship:

The leverage you get from hiring really talented people is a huge risk during rough times, because these people have lots of other options and the ambition to pursue them.
…
If any of of the people in this chain stop believing the hype around which their project is organized, then the hype becomes unjustified… And once the bubble starts to wobble, 10x employees will move on to the next compelling tech vision, causing the leveraged death spiral mentioned in the last section. Leveraging your company with talent increases your volatility - either you orchestrate a revolution, or you implode – Talent as leverage

It's never pretty when leverage blows up, and talent leverage is no different. Troubled startups face double jeopardy – preferred equity implodes, as does the talent stack.

Easy funding =/= easy life

I'm increasingly skeptical of the notion that easy funding makes the lives of founders easier. Funding merely shifts the bottleneck.

Difficulty hiring is a recurring theme I've seen among founders during red-hot markets. The financial capital comes easy; the human capital doesn't.

In fact, hiring can be tougher after raising at a big valuation because candidates might think all the upside is gone already, making equity compensation less attractive:

At Retool, we decided to raise the smallest Series C we could, at a valuation substantially lower than most offers we received. Why? Because even though maximizing our valuation and money raised might be optically good (the press loves to report on large financing rounds at large valuations), it hurts the core constituency that got us here: our team… Reaching peak valuations too early results in substantially less upside for employees. – Raising less money at lower valuations

Between the increased effort that hiring requires during boom periods and the prospect of employee exodus after hitting choppy waters, founders should be wary of getting caught up in the irrational exuberance of the broader market.

The Rule of 40 is a popular heuristic for evaluating financial performance and predicting valuation multiples.

But it's got 99 3 problems:

• It's too rigid
• It's too short-term
• It's too narrowly focused

In this essay, I break apart the Rule of 40, fix it, and put it back together again.

Laying down the rules

After flying high for some time, software valuation multiples have come back down to Earth (Note: LTM multiples used throughout this article):

While there are all sorts of explanations for the recent tech correction, the Rule of 40 concerns itself not so much with the ebbs and flows of software valuations but rather their determinants at a single point-in-time. It provides a simple rule of thumb for evaluating the performance of software companies and estimating their likely valuation multiples:

The Rule of 40 states that the sum of annual revenue growth and profitability (either EBITDA or free cash flow margin) should equal 40 or more for the best performing companies, as in the following slide from Battery Ventures' Software 2021 deck:

In general though, the "40" isn't particularly important. Higher is better; that's it.

"Houston, we have three problems"

I see three problems with standard Rule of 40 analysis.

Problem #1: Revenue and margin may not be equally important

Because the Rule of 40 adds revenue and margin together with no particular weighting of each (it's analogous to a simple average), in a subtle way it implies they are of equal importance.

This isn't necessarily true of course. An additional point of revenue growth could be worth more, the same, or less than an additional point of profitability.

Fix #1: include revenue and margin separately in our regressions.

Problem #2: The long-run may look quite different

Even if revenue and margin were equally important on average, their relative importance could change over time.

Folks tend to run Rule of 40 regressions at a particular point in time, revealing how the market values growth and margin today, and today alone, saying nothing about how those relationships change over the long-run.

In a bull market, where growth is highly valued, our regressions fool us into thinking growth will be forever important. When the market subsequently turns, growth gets crushed, needlessly surprising many analysts and investors.

Fix #2: pool data from multiple time periods.

Problem #3: Growth and margin aren't the only factors

Simple regressions inflate the importance of the Rule of 40 by misattributing the influence of other factors to growth and margin.

For example: Snowflake is a high flying company, both in terms of it's growth rate and revenue multiple. But Snowflake really is a special snowflake – it has a lot of other things going for it too, like the massive market for data warehouse solutions.

Thus, naively comparing Snowflake and other companies with smaller markets in the same regression will tend to overstate the value of revenue growth, leading to the false impression that, "if only we grow as fast as Snowflake, we'll be valued just as highly." This is wrong.

Fix #3: control for the average influence of unobserved factors.

If it's broken, fix it

Let's implement each of these fixes one-by-one to see how they influence the results.

Here's the impact of revenue growth and free cash flow margin on valuation multiples with each of the three fixes implemented sequentially:

First, let's split up growth and margin to see how they separately impact valuations. To start, we pick a random date, June 30th 2021:

• According to this simple analysis, a single additional percentage point of revenue growth increases valuation multiples by 3%, while an additional percentage point of free cash flow margin only adds 0.5%.
• By this measure, growth is six times as important as margin.

Second, we'll pool together two years of trading data, excluding any companies that went public or got acquired during this time so we maintain a balanced sample throughout. The regression reveals the average importance of growth and profitability over the entire period:

• An additional point of revenue growth leads to a 2.7% higher revenue multiple (down from 3%) while an additional point of free cash flow margin yields 1.1% higher valuation (up from 0.5%).
• Growth matters only ~2.5x as much as margin.

Lastly, the results still suffer from the fact that we're attempting to explain all the complexities of valuation with only two variables (the "Snowflake problem"), giving credit to growth and margin where credit isn't due. The final regression controls for the average impact of these unobserved factors (fixed effects for the econometrics nerds):

• 1 point of either revenue growth or free cash flow margin increases valuation multiples by 1%.
• Growth and margin matter equally.

So it turns out growth and margin are equally important after all!

Note: it may be much easier to increase growth or margin by one percentage point, so we have to be a little careful in saying they "matter equally."

Still, we've much more rigorously confirmed that growth and margin have similar effects on valuation. Very cool.

How the rules have changed

Here's how the market value of growth and profitability have evolved over time:

We can see a number of interesting features:

• The market value of profitability temporarily spiked in the early days of the pandemic but then fell significantly, bottoming out at almost zero in Q1 2021
• Meanwhile, the market's thirst for growth surged from only 0.5% per percentage point of growth to 1.5% at the peak in late 2020
• Beginning in early 2021, margin began a steady comeback, and profitability is now as important as it was in the most uncertain days of the pandemic
• The value of growth fell in early 2021, recovered somewhat, and crashed even more spectacularly later in the year, returning growth to its pre-pandemic value

We can also plot the "spread" between the value of growth vs. margin, an interesting chart in its own right:

• Growth and margin start off fairly close, with a slight market preference toward margin
• After a brief period of uncertainty, the value of software growth relative to profitability roars back, peaking at a ~1% premium at the end of 2020
• Again, we see the first blow to growth in the first half of 2021 and a second blow landing in late 2021
• Relative to profitability, growth is even less valuable than it was pre-COVID

Rewriting the rules

A few conclusions:

• On average, growth and profitability are equally impactful to valuation, but one may win out over the other at various periods
• Growth matters much less than simple cross-sectional regressions imply
• As of early 2022, the market has turned decisively against growth in favor of profitability

In a way, these results affirm the utility of the Rule of 40: growth and margin are similarly important. However it adds a wrinkle: the degree of similarity varies over the market cycle.

Though the days of "growth at all costs" are clearly over, it's still fair to ask: growth at what cost? In other words, how should companies trade off growth and margin?

I'll leave that to a future post. 👀

It combines an equity-like security with a debt-like security. Which one it is depends on the success of the business:

• When things go well, preferred stock converts to standard common equity.
• When things go poorly, preferred stock turns into debt.

In this way, preferred equity exists in a constant state of quantum superposition. It's neither equity nor debt, until it is.

Schrödinger's balance sheet

Erwin Schrödinger was a renowned physicist and one of the first to study the strange world of quantum theory. Among us mortals, he is most known for his famous thought experiment, eponymously called "Schrödinger's cat." In it, he posited that a cat sitting in a box along with a flask of poison could be considered to be both dead *and* alive until we open the box to find out if the flask has shattered, killing the poor kitty.

Preferred equity is like Schrödinger's cat. Open the box: if a startup is very valuable, preferred stock all becomes normal equity. If the box contains a dead or slow growing startup that hasn't created much value, preferred stock becomes debt instead.

You can think about preferred equity as debt combined with a deeply out of the money call option. "Out of the money" is just a fancy way of saying the investor won't make much of a return unless the company gains significant value. If the company creates no value from that point forward, the option component is worthless, and the equity merely becomes debt. If the company creates significant value, the importance of the debt element wanes and the option component dominates the return profile.

Strangely enough, this is the exact reverse of what you'd want to have happen as a founder.

This point has been made by John Cochrane and others in the context of bank capitalization:

Ensuring that banks are funded with significantly more equity should be a key element of effective bank regulatory reform. Much more equity funding would permit banks to perform all their useful functions and support growth without endangering the financial system by systemic fragility.

Things go very wrong when banks are overly-funded with debt, as debt tends not to "fail gracefully" the way equity does. In financial crises, banks "break" in a very hard way due to the significant debt they carry, which must be paid back, necessitating government bailouts in the worst case scenarios. On the other hand, equity tends to break in a soft way: if the company isn't worth anything, nothing is owed. Thus a sudden decline in the value of the company doesn't cause a severe panic.

Cochrane goes so far as to advocate that in times of financial distress, debt on the balance sheet of struggling banks should simply convert to equity rather than be bailed out by the government. Notice what he's proposing – debt on the upside and equity on the downside.

Likewise, as a founder, you'd want your capital structure to be equity when things go badly, making it a variable cost that declines gracefully as your business success declines, and you'd want it to be debt in the upside case, effectively acting as a fixed cost over which you get leverage.

In startup land, however, we do the opposite – we convert equity to debt in times of distress.

Spin up

Preferred equity is more valuable to investors than common equity, so they're willing to pay more for it, necessitating the issuance of fewer shares and leading to less dilution for the company. Thus, relative to raising common equity, preferred equity has a similar benefit as debt, in that it allows you to raise more capital with less dilution.

In this way, startups become synthetically levered. No debt appears on the balance sheet, yet the returns of the common equity (really anyone below the most senior preferred stock) get juiced by the lower cost of capital achieved via preferred equity.

The value of preferred stock can be inferred from its cost, i.e. its premium relative to common equity. The preferred equity premium is rarely laid out explicitly, but researchers at Stanford and the University of British Columbia estimate that these days preferred equity is 50% more valuable than common equity and was even more valuable in the past:

This paper develops the first option pricing model of venture capital-backed companies and their security values that incorporates the dilutive future financing rounds prevalent in the industry. Applying our model to 19,000 companies raising 37,000 rounds shows that preferred contractual features make the most recently issued preferred shares worth on average 56% more than common shares – A Valuation Model of Venture Capital-Backed Companies with Multiple Financing Rounds

A quick illustration: you're a founder, and you know you need to raise roughly $60M to achieve a meaningful outcome. Because preferred equity is worth 50% more to investors than common, you could either raise $60M in preferred stock with some level of dilution, or raise it all in common stock for 50% more dilution. Let's say the common approach would dilute you by 60%. Then raising the same amount via preferred would only dilute you by 40%:

This obviously makes a massive difference in an upside scenario. Simply put, you get to keep much, much more of your company. When things work out, preferred equity has a higher cost to investors, and a lower cost to startups.

This is obviously great in a bull market. Cheap, bountiful leverage, available oftentimes for a fraction of the painful diligence associated with raising real debt.

The availability of preferred equity drives companies to raise more than they otherwise would. We'll look at such a scenario next.

Spin down

When rates of return are sufficiently low, preferred equity begins to look eerily like debt. When are rates of return lowest? When valuations are highest:

High valuation multiples corresponds to lower returns, and vice versa. After periods of frothy valuations, returns end up lower than expected, bringing lofty valuations multiples back down to reality – Companies Rarely Grow Into Their Valuations

The mountains of preferred equity being layered into private companies follows the same behavioral vein typically seen during the peak of a market cycle, that is, massive overuse and abuse of debt that juices returns. This has predictably disastrous consequences when things go south.

As is common in finance, the problems with debt reveal themselves only once it blows up. In a downside scenario, the returns of common equity holders get crushed by the weight of the liquidation preference attached to preferred equity.

Remember, venture capital investors are willing to pay $1.50 for preferred equity granting the same ownership they could get for $1 of common equity. In an upside scenario, they effectively pay $1.50 for $1's worth of equity. What happens in a downside scenario?

Let's return to our previous example. This time, the company chooses a certain, fixed level of dilution that it's willing to endure, 60%, and raises money up to that point. Since preferred stock can be issued at a premium, the company can raise $90M of it for the same dilution as $60M of common:

So far so good. The preferred approach gives you $30M more to put to work in the business. But let's say you never quite achieve product-market fit (not uncommon, even among companies that have raised tens of millions of dollars) and you woefully underperform expectations. You never become a unicorn, and the business is sold for only $100M:

• If you only raised common equity, the founders and employees walk away with $40M. Not as much as hoped for, but not bad!
• If you raised preferred however, founders and employees walk away with only $10M, since the equity is now debt, and the investors must receive their "1x liquidation preference" (their "principal" effectively). Brutal.

This may offer a clue into why the hedge funds of the world have moved so swiftly into venture. As sophisticated, multi-asset investors, they are quite used to lower-returning, low-risk credit investing combined with potentially high-returning, high-risk derivatives. Rather than "fish out of water", as they are often portrayed by traditional venture investors, could they in fact be quite shrewd (tiger)sharks?

The rabbit hole goes further. Define "overvaluation" as the premium at which preferred stock is valued relative to common equity. Analysis shows overvaluation predicts poor exit outcomes, measured in terms of (1) exit value and (2) the probability of going public or being bought out (note: low R^2 implies exit outcomes are difficult to predict, not that the relationships aren't statistically significant):

… overvaluation predicts low exit outcomes… The extent of this overstatement predicts poor subsequent performance – A Valuation Model of Venture Capital-Backed Companies with Multiple Financing Rounds

Abuse of preferred equity predicts worse outcomes, fewer exits, and lower returns.

Synthetic leverage, as with all forms of financial leverage, makes bad situations even worse AND more likely.

Superposition

Now, let's finally explore the quirky quantum world of superposition.

Preferred equity starts as neither debt nor equity: you have to "open the box" to see if the company is successful. Unfortunately, actions that make your upside better (raising a ton of money and putting it to work in the business) also make your downside worse. Things that look smart in the upside scenario (raising a "war chest") look disastrous in retrospect on the downside. Gain and pain are entangled; success and failure are superimposed on one another.

Perversely, the Silicon Valley ecosystem creates more synthetic debt (i.e. preferred equity) exactly as returns fall:

1. Valuations rise
2. Companies take advantage by issuing more preferred stock
3. Returns fall due to inflated valuations
4. Preferred stock converges toward debt

Let's return to financial crises. The capital stack of a company is very similar to that of a home. From the perspective of the founder, common stock is the "equity" in the home and preferred stock is the debt or mortgage. If value of home falls below the principal on the mortgage, the homeowner is "underwater", destroying the value of their equity.

Likewise, when a startup's valuation falls too low, the value comes out of the common shareholders; they get squeezed first. This is most severe in a down round where anti-dilution provisions come into play. This is the Silicon Valley version of a good ol' Wall Street bailout: the debt (preferred) holders get saved, and the equity holders get wrecked.

What tends to happen when homeowners go underwater? We know this from history – they abandon the home:

Now I ask: what might founders and employees do once they are underwater on the preferred equity they've raised?

Equity value motivates founders and employees. If there's no hope for their equity to be worth anything, could we see discouraged founders and employees abandon ship in droves? Even if they don't leave the company entirely, could they effectively go "delinquent" and "check out," significantly reducing their company-focused effort and output?

In startups, we see acceleration of value impairment: as things get worse, they get even worse. Not a great underwriting scenario for a debt-like instrument huh?

Poor performance alone doesn't create a "zombie startup." Zombification requires the total evisceration of team vitality and morale. Thus, Schrödinger's cat becomes Schrödinger's zombie:

"… it's very easy to get into this 'zombie mode,' where your startup is neither truly succeeding nor dying… This is actually worse than failing." – Drew Houston, Founder of Dropbox

Schrodinger's Pandora's box

Imagine you're selling your home and the buyer offers to pay you 50% above the asking price as long as they can get "preferred equity" in your home. You'd be ecstatic about the richer sale price, but you'd also be really curious to know what in the world "preferred equity" is and why it's so valuable to them!

Likewise, as a founder or startup employee, even if corporate finance isn't your favorite topic, you should wonder why investors value preferred equity so highly.

What could investors be so concerned about that they would willingly overpay by 50%? And if they're so concerned, do you think, maybe, you should be too?

Though it's easy to forget, ACV is a customer-weighted metric: it tells us something about the customers of the business. However, despite its name, it doesn't tell us much about the revenue profile of the business, which is arguably more important.

Further, the standard ACV metric obscures valuable information about the skewness of monetization / revenue. Thus, we can't use it to compare companies with differing customer concentration.

We need a different SaaS monetization metric.

I present to you: weighted ACV (WACV).

Customer-centric or dollar-centric?

In Enterprise Software Monetization is Fat-Tailed, I talked about the follies of focusing on simple averages, especially in enterprise software:

In enterprise software, the "average" customer is a meaningless concept.

Paying too much attention to the "average" customer leads many founders and investors astray.

Average/annual contract value, or ACV, is one such misleading metric.

ACV tells you something about the typical customer. Products with small ACVs tend to serve smaller, more fickle customers with a high probability of churning, like small businesses and startups. High ACV products address the needs of larger companies with more mission critical use cases, leading to less churn.

However, while ACV tells you something about how the typical customer might behave, it doesn't tell you much about how the typical dollar of revenue behaves. If 70% of the revenue comes from 20% of the customers, then for the purposes of analyzing revenues at a point in time, it's really only that subset of customers that matter. Such extreme concentration is not unusual for public software companies:

The implied top-20% and top-1% revenue concentration are quite large for most companies (blue = top 20%, red = top 1%)

So, the top 20% typically represent ~70% of revenue, while the top 1% represent ~40%. Not quite Pareto 80/20, but pretty close!

This distinction between customer behavior and revenue behavior is critical, as the behavior of the typical dollar of revenue is much more informative about the quality of revenue and the likelihood that revenue will be retained or expanded. For companies with high revenue concentration, simple averages are largely useless.

This is largely true to this day - for many public enterprise SAAS, a very small # of customers contributes a huge chunk of total rev.
e.g. ~0.95% of $NET paying customers rep >50% of rev in Q3/21

So why do some analysts still use metrics like ARPU/AARPU for enterprise saas? pic.twitter.com/WXSpWJYfNA

— Masterly Inactive (@masterly_in) December 30, 2021

Rather than the exception, high concentration is the norm in certain verticals (e.g. cloud infrastructure) or pricing models (consumption/pay-as-you-go) where a "customer" can be as small as tens of dollars per month:

Interestingly, many companies tied to infrastructure in some way like Datadog, Fastly, and Twilio... have 80/20 monetization distributions, at least in some years.

Unfortunately, the standard ACV metric is incomparable across companies with vastly different customer concentration. A company with many small or "free tier" customers will have an artificially low ACV metric, even if contract sizes are reasonably high among its paying customers. You can't naively compare such a skewed company to another business where every customer is small.

This is why you see public software companies with tons of bottom up but low-paying usage report ACV or other customer metrics including only the customers larger than some nominal spending level, like $5,000 (screenshot from the GitLab S-1):

Using arbitrary thresholds is a clunky way of dealing with this problem, as companies can pick any value they want. ACV calculations with different thresholds are effectively incomparable.

Weighing in

A customer-weighed metric helps you understand customers. ACV is a customer or equal-weighted metric, which is to say every customer is weighted equally. So if there are 1000 customers, the contract value of each customer is "weighted" or multiplied by 1/1000 and then summed up. Because each customer is weighted equally, the resulting metric tells us something about the typical customer. Thus we can rewrite ACV as:

However, if you want to understand revenue, you need to use a revenue-weighted metric.

Just like the market capitalization-weighting of the S&P 500 index signals the changing value of the overall equity market (but not of the individual companies in the index), a revenue or contract value-weighted metric tells us much about the overall revenue of the business (rather than the individual customers).

Introducing: weighted ACV (WACV).

Revenue-weighting the revenue each customer generates might sound like double counting, but we need to weight individual customer revenues by something. Nothing limits us to equal-weighting. Revenue-weighting is not only perfectly reasonable, it's the more appropriate methodology when analyzing quality of revenues.

A perfect example is net dollar retention, a common tool for understanding the quality and behavior of software revenues:

• Believe it or not, net dollar retention is also a revenue-weighted metric: it weights the individual dollar retention of each customer by the revenue that customer represents. That's not how the calculation is typically conceptualized, but that is the underlying math (grab some data and prove this to yourself)
• Thus, larger customers matter more for net dollar retention; the churning of a tiny customer has infinitesimal impact.

To calculate weighted ACV:

• Take the contract value of each customer and multiply it by the proportion of revenue that customer represents. A customer representing 1% of revenue gets a 1/100 weight, a customer representing 3% of revenue receives a weight of 3/100, and so on.
• Then sum all those numbers (i.e. SUMPRODUCT, for all you spreadsheet monkeys out there)

Weighted ACV tells you where to look if you want to best understand the revenue of the business. For example, a WACV of $25,000 tells us roughly that the typical dollar of revenue is generated by a $25,000 customer. Thus, revenue retention/churn and other revenue dynamics of the business will tend toward what we'd expect for those kinds of customers. Revenue-weighting is especially helpful if the company in question has a large number of customers spending little or zero, contaminating the equal-weighted ACV calculation.

Meanwhile, changes in WACV correspond to changing monetization among the largest chunk of the revenue base. Customers with the biggest revenue impact will have the largest influence on WACV, as with net dollar retention.

If we think about a company as a sort of physical object, its "center of mass" is concentrated at its WACV. Logically, if we want to do physics on this thing or model its movement, what matters most is its center of mass (its weighted ACV), not necessarily its literal "center" (equal-weighted ACV). That's where revenue is most "densely" concentrated.

This concept flips standard monetization logic on its head. Instead of saying "the average customer generates X revenue," as if all revenue comes from a single type of customer, WACV says "Y-size customers generate the average dollar of revenue," respecting the fact that certain customers are disproportionately impactful. ACV asks "what's the expected value of a randomly selected customer" while WACV asks "what's the expected contract size associated with a randomly selected dollar of revenue."

As I'll demonstrate in a second, this is really powerful. In a single number, we capture much more information than the equal-weighted ACV metric by accounting for the skew and concentration of the customer base. In statistics parlance, WACV gives us some sense of the variance of revenues across customers, rather than merely the mean.

Note: this essay focuses on enterprise SaaS monetization, but you can apply the same concept to B2C and other models. E.g. Average Revenue Per User (ARPU) becomes Weighted ARPU (WARPU), and so on.

An example worth its weight in revenue

Two businesses could have the same equal-weighted ACV but wildly different weighted ACVs.

Here's an example to make this vivid.

Imagine two, $50K revenue companies with five customers each, equivalent equal-weighted ACVs, but very different revenue distributions. Company 1 has five equal-sized customers, while Company 2 has a fairly typical skewed distribution, with 20% of customers representing 80% of revenue:

Notice how sensitive WACV is to the more skewed revenue distribution of the second company. It's telling us Company 2's large $40K customer is really dictating the revenue dynamics of the company. For Company 2, the typical dollar of revenue will behave like it's coming from a $32.5K customer, not a $10K customer. The standard ACV metric totally misses this.

Now that we've seen how WACV helps us at a single point in time, let's see what happens to the metric over time. Next month, both companies land a large, $100K deal with a new customer:

Both metrics increase, but weighted ACV increases by a lot more, reflecting the dominance of the new large customer within the revenue base. The standard ACV calculation unwittingly discounts the effect of large customers, muting their impact.

Again, the weighted version of ACV correctly points us towards the largest customers. The "typical" dollar of revenue for both companies will now behave like it's coming from a roughly $70-80K customer, not a $25K customer.

Lastly, the fact that WACV across the companies has converged suggests their revenue concentration is now much more similar, which is exactly right.

VICs: Very Important Customers

One last trick.

Just like dividing total revenue or contract value by ACV yields the total customer count, dividing by weighted ACV tells us something about the customer base. The calculation tells us, on a revenue basis, how many customers really matter. In other words, what minority of customers effectively determines overall revenue?

Let's call them very important customers (VICs):

It turns out, this will be a small fraction of the overall customer count for the typical enterprise software company.

Returning to the first example, total revenue was $50K, while weighted ACV was $10K for Company 1 and $32.5K for Company 2. Thus, Company 1 has 5 VICs while Company 2 has 1.5. This tells us that while Company 1 has five relevant and discernibly important customers, Company 2 only has one or two. This accords with the data – in terms of revenue risk or quality of revenue, Company 2 is effectively a one (point five) trick pony:

In the second example, total revenue is now $150K, with weighted ACV of $70K for Company 1 and $77.5K for Company 2. Some quick math yields ~2 very important customers for each company, reflecting their respective revenue concentration. Said differently, you can summarize the entire revenue profile of each company by looking at the two largest customers:

Frankly, I find this much more representative of the volatility of revenue. These companies have serious concentration risk, and from a diversification perspective really only have two customers each.

Public software companies have similar concentration, so the same logic applies: very important customers will only be 30-40% of the overall customer base.

With WACV and VICs, founders with high revenue concentration can now rigorously quantify the number and magnitude of meaningful customer relationships they have while avoiding awkward conversations that go something like, "well... we have X total customers, but only Y many are paying us more than Z per year, so if you remove those then our business is really..."

Conclusion

Are there any downsides to weighted ACV?

No, none at all.

Just kidding! The negatives of WACV mirror those of other size-weighting schemes like the S&P 500: it's extremely sensitive to idiosyncratic fluctuations in your largest customers that may not be representative of the broader customer base or long-lasting in nature.

These are fair critiques. However, all can be easily addressed by calculating both weighted and equal-weighted ACV. Coincidently, the difference between the two is a nice measure of customer concentration and the fat-tailedness of monetization: the further apart they are, the more revenue is concentrated among a few customers, and vice versa:

In fact, the main disadvantage of revenue-weighting is that it requires knowing the revenue or contract value of each and every customer. So an outsider can't calculate the metric; it must be reported by the company. You can often approximate it with just data of the top 30-40% of customers, but that won't be perfect.

If you do have enough information to calculate weighted ACV, you should always do it. WACV contains more information than the standard ACV metric, and by definition you can always calculate the latter if you can calculate the former.

If you found this helpful, need help with the calculation, or end up implementing this in your analytics, do let me know!

It turns out this rarely happens.

Most often, companies don't grow fast enough to compensate for rising valuation multiples. Instead, high valuations today imply slower value appreciation in the future, i.e. lower returns.

Thus, companies don't catch up to their valuations; their valuations catch up to them.

Expanding on multiple expansion

Investors can justify rising valuation multiples with a belief that:

• growth will be better than expected ("they will grow into their valuation"),
• returns will be lower than expected (the valuation will appreciate more slowly), or
• multiples will be even higher in the future ("someone else will pay more for this thing")

If growth accelerates, valuation multiples fall due to the denominator growing faster than the numerator (revenue or cash flow grows faster than valuation).

On the other hand, if future returns decline, the numerator grows more slowly (valuations grow slower than revenue or cash flow), also causing multiples to fall. Either outcome leads to declining valuation multiples:

Think of the "multiples are higher today because multiples will be higher tomorrow" justification as a sort of "bubble mentality" or "greater fools" explanation. Someone else will pay more for this later, so I'm OK paying more today. This of course can't go on for long: bubbles eventually pop; manias eventually subside. So, here too, multiples return to "normal."

These three explanations for valuation multiples are collectively called the Campbell-Shiller Decomposition, originally conceived in the late 1980s. It lets us cleanly account for changing valuation multiples with changing beliefs about growth, returns, and future multiples. We can even make an equation out of it ( = Change):

Therefore, higher multiples today imply (1) faster growth in the future, (2) lower future returns, or (3) even higher multiples going forward.

Importantly, this works for both forward and backward-looking analysis: changing expectations of future growth, returns, or multiples affect valuations today and changing valuation multiples today also forecast future growth, returns, and multiples. In other words, investors' subjective expectations about the future influence valuations today and those valuations also predict the objective future itself.

Regardless of the specific cause, valuation multiples eventually mean-revert; subjective expectations eventually collide with objective reality.

Subjective expectations

Let's start with subjective expectations. When researchers decompose variation in price-earnings (PE) ratios into changing expectations about cash flows, discount rates (e.g. expected returns), and future PE ratios using the Campbell-Shiller Decomposition, they find fluctuating cash flow expectations (CF in the table below) explain more than 90% of all movement in valuations ratios, while expected returns (DR) explain little:

“In the 2003 to 2015 sample, one-year subjective earnings growth expectations account for virtually all (94%) price-earnings ratio movements.”
...
“high prices can largely be accounted for by high expectations for future cash flows. Cash flow growth expectations are volatile and significantly correlated with prices. In comparison, return expectations are much less volatile and positively correlate with prices, meaning that high prices cannot come from low discount rates” — Subjective Cash Flow and Discount Rate Expectations

Another paper breaks down the expected returns vs growth dynamic between different classes of market participants, finding that sell-side analysts, institutional investors, and corporate CFOs all use growth expectations to explain fluctuating valuations ( in the table below) rather than expected returns ():

“both Wall Street and Main Street believe... the expected cash flow process is the main economic force driving asset price variations”
...
“cash flow expectations alone explain 99% of the price-dividend ratio variation, while return expectations only explain about 10%.” — Subjective Return Expectations

Investors see valuations move and ascribe these gyrations to changing expectations for future business growth. But does the growth ever materialize?

Objective reality

Not really:

“the variance decomposition results reveal that all three types of investors seem to overestimate how much cash flow variation contributes to the variation in asset prices when compared to objective measures of return expectations, which show that discount rate variation should contribute the most to asset price variation” — Subjective Return Expectations

Now we turn to objective reality. The chart below plots expected (solid blue) and realized (dotted green) 1-year earnings growth of the S&P 500 over the last 40 years. Expected earnings growth consistently exceeds actual growth, implying investors tend to overestimate earnings growth. Notice as well how expected earnings growth tend to show "upside volatility", whereas actual earnings show more downside volatility.

“Earnings growth expectations typically fail to predict the change in earnings during busts, but they do tend to predict recoveries and track future earnings growth reasonably well during normal times.” — Subjective Cash Flow and Discount Rate Expectations

Investors rarely predict poor earnings growth in advance. They often correctly predict strong earnings growth after recessions, but that's kind of cheating isn't it? Quick bounce-backs follow recessions almost universally.

When financial researchers do look-backs, they find that high valuation multiples precede low returns. In fact, they literally predict lower returns. Per finance legend John Cochrane's 2011 Presidential Address at the American Finance Association (apologies for the pixelated 360p screenshot, YouTube was less generous back then):

"These regressions are telling us that all variation in price-dividend ratios, all of that volatility, corresponds to variations in expected returns. None of it corresponds to variation in expected dividend growth and none to rational bubbles or prices that are ever higher."
...
"The valuation ratio translates one to one to expected returns and doesn't forecast the cash flows or price change that we might have expected." — John Cochrane

High valuation multiples correspond to lower returns, and vice versa. After periods of frothy valuations, returns end up lower than expected, bringing lofty valuation multiples back down to reality.

Despite this sobering reality, investors are extremely stubborn about future return expectations, even in the face of rising prices. Undeterred by nose-bleed valuations, they continue to expect roughly the same return in all periods, leading actual returns (dotted green) to be much more volatile than expected returns (solid blue), which look nearly constant in comparison (right-hand chart):

Equity investors always expect the same rate of return, regardless of how high or low valuations multiples are today, which doesn't make a ton of sense.

Waiting for Godot growth

The data from the public markets is clear: valuations rise, returns fall.

Investors don't like to think this way, and thus they continue to expect the same return in all periods, leading to inevitable disappointment. Meanwhile, they justify their stubbornness about returns by convincing themselves that growth will save the day. It typically does not.

This data only covers public equity investors. It'd be fair to point out that growth among startups is much more variable than that for public companies, so much so that variation in startup growth could explain much more valuation variation.

That said, I find it hard to believe the future for startups could brighten so dramatically, practically overnight:

If anything, the high variability of startup growth may be a cognitive trap, seducing us to think that herculean growth alone can shoulder weighty valuations.

I think it's past time we in the venture world get serious about the valuation craze. Do we really think the future is so much brighter? Are we merely accepting lower future returns, and if so, do our LPs know this? Is it all just a bubble?

Investors rely too much on growth to justify current valuations and underestimate the risk that higher multiples portend poor returns. Investors like to think companies will grow into their valuations, but more often than not, the stock simply underperforms.

The burden is on venture investors to prove there's something special about the asset class that will help it escape the fateful relationship between valuations and returns we see in public equities.

Until then, growth is just a cop out.

1/ How investors justify high valuation multiples: "they will grow into their valuation"

Pay it, and growth will come

This rarely happens

Companies don't catch up to their valuations; their valuations catch up to *them* 🤯 pic.twitter.com/PRbaj8xskc

— Nnamdi Iregbulem (@whoisnnamdi) November 11, 2021

WebAssembly, or WASM, is one of the most exciting up-and-coming technologies in software development today. WebAssembly lets developers write code in any major language, translate that code to the compact WASM format, and run it on the web with the high performance of a native application.

Redpanda is one of the first infrastructure technologies to take advantage of WASM, enabling developers to “write and edit code in their favorite programming language to perform one-shot transformations, like guaranteeing GDPR compliance by removing personal information or to provide filtering and simple aggregation functions.”

JavaScript, Python, Rust, Go — anything that compiles to WebAssembly (basically everything at this point) can be used to transform data. Again the key is accessibility — inline WASM transforms in Redpanda represent just that.

Ever since partnering with Vectorized, we at Lightspeed knew that Redpanda was much more than a streaming engine. Redpanda truly is an Intelligent Data API, and it’s Data Policy Engine unlocks one more piece of that puzzle.

Today, I want to expand on this and talk a bit more about the exciting developments coming down the pipe at Vectorized.

Here's the big idea: rather than ship data to code, which is expensive and latency-prone, why not ship code to the data?

The keystone problem in streaming

Streaming has seen a ton of innovation over the last few years, but one area that remains painful today is stream processing. While a full explanation of these difficulties probably warrants its own article, I’ll cover the most salient points here.

The great thing about streaming is the immutable message queue. The not so good thing about streaming is, also, the immutable message queue.

What do I mean by this? The immutability of the message queue is great because it guarantees that data produced to the topic can be replicated one-to-one by simply consuming the whole queue from the start. This lets you effectively “replay” data like a VHS tape and also gives you built-in auditing, since you know the ordering hasn’t changed. It also decouples producers of data from the consumers of that data, letting the two operate independently without significant coordination work.

However, there are some problems. Immutability not only means you *can* consume the data in the same order it was produced — it means you must (at least as implemented today in the Kafka-API world). Further, immutability means one must consume at least as much as was produced, often many times as much, leading to additional costs in the form of network traffic and other resources. These issues can be avoided to some extent by being very diligent in setting up your architecture a priori, careful in segregating your use cases, etc., but few do this in practice.

In summary, the things that make streaming great also make stream processing complex and difficult.

Shipping code to data

Here’s where WebAssembly comes in. WebAssembly lets us do a small amount of extra compute work to save us a ton of cost in the form of network utilization and latency. Mission critical streams can stay as large as they want without saturating the network.

We’re excited about the power and promise of WebAssembly, and we think it has the potential to do for server-side compute today what JavaScript did for the web in the late 90s. Like JavaScript enabled the shipping of code to the user’s client, WebAssembly lets engineers ship code to the storage engine.

Here’s how.

The Redpanda Data Policies Engine simultaneously extends the Kafka API and what you can do with server-side compute. It’s a server-side compute API for Redpanda that’s compatible with all existing Kafka tooling and the broader ecosystem, including Spark Streaming, Kafka Streams, Flink, and more.

With Data Policies, all your favorite tools can benefit from server-side WebAssembly filters, allowing for easy and simple data scrubbing, cleaning, filtering, normalization and more. As a runtime, WebAssembly allows code to execute inside Redpanda, allowing code to be injected into an active cluster at runtime.

Redpanda Transforms enable extremely low-latency transformations by avoiding the “data ping-pong” that happens in many systems and stream processors today, where data gets sent back and forth between storage and compute, even for very minor operations. Further, Redpanda Transform is built on the V8 engine, the high-performance JavaScript and WebAssembly engine that also powers Chrome and NodeJS.

Transforms in Redpanda are effectively a stateless predicate pushdown. In simple terms, that means performance is massively improved by “pushing down” code to the data, only touching the relevant data (and ignoring the rest), cutting down on wasteful network traffic, I/O, and latency.

Redpanda Transforms are also completely auditable. The controller can report on which specific piece of code is touching which specific piece of data, on which machines, at which topic offsets.

As easy as one, two, THREE

Few things are simple in data engineering, but, as usual, Vectorized is changing the game. For all the cool things happening under the hood, getting started with Data Policies is shockingly easy.

First, write your policy/transform in a WebAssembly compatible language (read: everything). To make this even easier, Redpanda provides native plugins for inline transforms. In the end, you end up with something that looks like fairly standard (JavaScript in this case) code. Here’s a simple proof of concept: transforming all lowercase alphabetic characters in a record to uppercase:

import { InlineTransform } from “@vectorizedio/InlineTransform”;
const transform = new InlineTransform();
transform.topics([{“input”: “lowercase”, “output”: ”uppercase”}]):
…
const uppercase = async (record) => {
    const newRecord = {
        ...record,
        value: record.value.map((char) => {
            return char.toUpperCase();
        }),
    };
    return newRecord;
}

Next, apply the policy to a Kafka topic. Again this is incredibly straightforward. All it takes is a single terminal command:

> bin/kafka-topics.sh \
— alter — topic my_topic_name \
— config x-data-policy={…}
(redpanda has a tool called `rpk` that is similar to kafka-topics.sh)

Now, Redpanda handles the rest.

• Upon a TCP connection to consume from the topic, Redpanda initiates a V8 context
• Bytes are read from disk
• Payload is transformed and re-checksumed (ensures Kafka protocol remains unchanged)
• Redpanda returns the transformed record

Pretty slick.

Simplicity is the ultimate sophistication

I’ve talked before about Vectorized’s deep understanding and appreciation for the power of simplicity:

Redpanda abstracts away the complexity that often prevents the typical developer from adopting real-time streaming.

The Data Policy Engine continues that tradition.

This was just an example use case, but they only get more interesting from here. GDPR compliance via masking rules. Encryption at-rest, on-disk. All at runtime, with near-native performance.

Curious to learn more after this quick introduction? Alex Gallego, founder of Vectorized, gave a sneak preview of the Data Policy Engine at this year’s Kafka Summit, which you can check out here:

I'm about to show a sneak preview of on the most anticipated features by the community here, as a sponsored talk.

tl;dr - take an *unmodified spark app* and use a push-down predicate in #wasm 🤯🤯🤯

Demo at the end.
#redpanda #kafka #KafkaSummit https://t.co/NiL2SNHrvq

— 🕺💃🤟 Alexander Gallego (@emaxerrno) September 15, 2021

"A high internal rate of return does not in and of itself put money in one’s pocket. Only when it’s applied to a material amount of invested capital for a significant period of time does IRR produce wealth"

I've noticed a similar problem in how we measure and evaluate startup success, particularly growth rates.

There are two kinds of growth: absolute and relative:

• Absolute growth is the absolute increase in a metric, like dollars of revenue added or customers acquired in a given period. The metric this period minus its value last period gives you the absolute growth:

• Relative growth is measured relative to what came before: "We ended the year with 150% more customers", or "We grew revenue 3X this year." It's effectively unitless, but we tend to measure it in percentages, as in "percentage points of growth." It's the absolute growth divided by the metric's value last period

In startup land, we talk way too much about relative growth. It muddles our thinking and makes compound, exponential growth seem easier than it really is. We'd do better to ground our thinking in absolute growth.

Obsessing too much over relative growth injects a number of "bugs" into our thinking:

• It screws up our language
• It makes us forget what startups actually do
• It sets up unrealistic expectations

No sense, this makes

The first problem with the concept of relative growth is that it warps our language in nonsensical ways.

A common scenario: a startup previously growing 100% Y/Y "slows" to 50% the subsequent year.

The board of directors asks "why did growth slow so much?" External investors evaluating the company think, "Wow, they really hit a wall."

But they did no such thing. Yes the growth rate measured in percentage terms has declined, but the growth measured in terms of dollars of revenue added hasn't slowed at all.

For example, let's say revenue went from $5M to $10M to $15M, which corresponds to 100% growth followed by 50%. Technically, the company added $5M in revenue each year, so growth hasn't slowed, per se. The problem isn't so much that growth "slowed" so much as it is that absolute growth didn't increase, hence the fall in the relative growth rate.

Think back to your high school physics class. Imagine we have an object, say a car, traveling in some direction. That car has a position, x, and a velocity, v. In business terms, imagine the level of revenue (or whatever metric) is the position of that car and it's absolute growth maps to the velocity.

As long as that car keeps moving at the same velocity, we would say it neither accelerates nor decelerates. In startup land however, a business growing at constant absolute growth is said to to somehow be decelerating because its relative growth is declining.

This is super strange.

In fact, the only way to make sense of this thinking is to measure the size of the business on a logarithmic scale. As I discuss in "You Don't Understand Compound Growth", a constant slope (velocity, effectively) on a log scale implies constant relative growth. In this view, adding a constant $5M in revenue annually is in fact a declining amount of log revenue added, thus matching the language of deceleration we tend to use.

OK fine, but this subtle rescaling goes completely unsaid. Further, as I'll discuss later on, constant relative growth is fairly unrealistic, yet it is effectively implied by our language on the topic.

We talk as if a reduction in relative growth implies something fundamentally changed about a business. And it would, if that's what companies actually did — that is, grow "relatively". However, more often than not, a decline in relative growth in fact suggests that not enough changed about the business.

As Yoda said, "Grow absolutely. There is no relative"

The second problem with relative growth is that it misleads us about what businesses fundamentally do.

The fundamental unit of growth is a dollar of additional revenue, not an additional percentage point of growth. In other words, absolute growth is the more fundamental concept. We can calculate a relative growth rate after the fact if we want, but the core result is absolute growth.

Relative growth is merely a derived concept, a way to normalize absolute growth into a metric that can be compared across companies. That's great, and it's a valuable tool. But we seriously pervert the concept when we go from using it as an ex post calculation to thinking that the primary activity of companies is to generate "percentage points of relative growth". That is putting the cart before the horse, the percentage points before the dollars.

Relative growth is not what literally happens. It's a way to benchmark what happened. Absolute growth is what actually happened, what was actually accomplished.

Absolute growth is the "hard thing" that startups must do each and every day. Relative growth imperfectly normalizes those efforts in order to make them comparable across scales of companies. It falls well short of its own goal. Further, the choice to normalize absolute growth by the previous level of revenue is simply a convention and a somewhat arbitrary choice. Hypothetically we could normalize growth by anything.

The job of the business is to generate absolute growth, to bring in more dollars this year than last year. The most informative information about the team's ability to do that comes from how much they were able to add in the past. Knowing that revenue tripled last year is not in fact informative about the likelihood of revenue tripling this year, because "tripling" is not what the company is in the business of doing. It's in the business of generating more absolute dollars of revenue each year. That it tripled last year doesn't really tell you whether it can triple this year. That it added $5M in revenue last year tells you A LOT about the likelihood of generating $5M in additional revenue this year.

Thinking in relative growth terms causes you to lose your sense of scale. Conversely, thinking in absolute growth makes things quite vivid. "How exactly are we going to find $5M in additional revenue next year" is a much more tangible and precise discussion than "How do we grow 80% next year" even if they mean the exact same thing.

The same relative growth rate can describe many different situations. For example, you hear that a company has grown 3X year-over-year, a relative growth metric. That single number is consistent with all of the following scenarios:

• $300K to $900K
• $1M to $3M
• $3M to $9M

Yet there is nothing similar about any of these growth stories. It is a fundamentally different task (and therefore, accomplishment) to go from $300K in revenue to $900K than to go from $3M to $9M. To describe these in the same terms ("3X growth") is a complete misnomer that obfuscates an incredible amount of context. These are nothing alike. In this way, relative growth represents a loss of information compared to absolute growth.

It's seductive to think that, "well, we grew 100% last year so we should expect to grow similarly this year" when in fact the two tasks are completely different. Constant relative growth of 100% two years in a row in fact requires doubling your growth rate in absolute terms because your existing revenue base is now twice as large. Absolute growth is the "real" growth target you need to hit, and for most companies that won't be easy. This is sobering.

Said differently, you can't eat percentage points of growth, you can only eat dollars of growth.

Rational expectations

It's cliche to point out that humans don't understand compound, exponential growth. In startups we have the opposite problem — people are so familiar with the concept that they apply it everywhere, in situations where exponential growth isn't likely or even plausible.

Exponential growth implies constant relative growth. Rather than seeing relative growth fall off, the business continues to grow at nearly the same rate for multiple periods.

Exponential growth is a great aspirational target to set for companies. Y Combinator does this and it works fantastically for focusing the minds of startup founders. When exponential growth is the goal, one realizes there are really very few ways to achieve it, and most growth hacks won't get the job done.

However, exponential growth is not a rational expectation to have or prediction to make. Few phenomena exhibit this sort of behavior outside of the natural world, like bacteria growing in a petri dish. Modern economies are often modeled as perpetual, exponential growth machines, but for most of human history, economies did not grow exponentially at all. Perhaps it's a fine approximation for a large economy, which aggregates an uncountable set of activities that sum up to something approximating constant exponential growth. But it's certainly not a good model for a single company.

Only at low growth rates is past relative growth predictive of future relative growth. At high growth rates the two are not very correlated at all. A business (or an economy for that matter) that grew 2% last year has a very good chance of growing roughly 2% this year. For a business that grew 200% last year... this year's growth is anyone's guess.

There's nothing inherent about startups that makes them grow exponentially other than the fact that it is easier to pull off for small companies than it is for larger ones. Any other correlation between startups and exponential growth is aspirational and attitudinal: startups grow exponentially because we will them to. It is not otherwise a reasonable expectation.

For forecasting purposes, constant absolute growth is a much better "baseline" or "base case" than constant relative growth. Constant relative growth may be a great target for setting the tone in the organization, raising ambitions, etc., but mark my words: it probably won't happen.

Constant growth in absolute terms is totally normal and unremarkable: startups often add similar amounts of revenue as they did the previous year. Constant relative growth is totally abnormal. The idea that a startup growing 100% should somehow manage to grow close to that the subsequent year is a mental bias we tend to have in Silicon Valley that doesn't properly reflect reality. It rarely happens.

It is much more common to see a company go from $5 to 10 to 15 to 20 (constant absolute growth of $5M per year) than go from $5 to 10 to 20 to 40 (constant relative growth of 100% per year). Doing the latter requires a fundamental re-architecture of the company and go-to-market motion every 6 months. Going from $5-10M looks very different than going from $20-40M, even though they both involve a doubling of the business.

I find that founders and operators tend to understand this all too well, while investors consistently underappreciate and underestimate the difficulty of scaling at a constant relative rate.

Operators, by dint of living in the real world, viscerally understand that what got them here won't necessarily get them there. Investors on the other hand like to think of companies "2X-ing year-over-year," as if the 2X relative growth is some fundamental property of the business, like its mass or electric charge. It is not, as they often discover 12 months later.

The investor mentality is in part driven by their relationship with the business. They invest at a certain valuation, and returns are measured relative to that entry price. Founders on the other hand start from zero — so they don't really measure themselves "relative" to anything. What matters is building as large a business as they possibly can in absolute terms.

Deal in absolutes

To close, I want to make sure I'm not misunderstood in my diatribe against relative growth.

There are actually very good theoretical reasons to focus on relative growth. For example, when relative growth is predictable and easy to pin down, we can use it to calculate the net present value of an asset, which is a direct function of the relative growth of cash flows. This is how discounted cash flow analysis works, for example.

Speaking of assets, the relative growth of asset prices (in other words, returns) is a fundamental building block of modern finance theory and is one of the primary traits that determines how attractive an asset is to potential investors (and the other is the risk or volatility of those returns).

So investors' focus on relative growth is not totally unfounded. That said, these notions all rely on either precisely knowing the relevant parameters (growth, discount rate, etc) or knowing precisely the variability of those metrics (their volatility, variance, etc). This is the world of quantifiable risk, and upon its core principles rests trillions of assets and millions of pages of academic finance research.

But startups do not live in the world of risk, they are squarely domiciled in the world of uncertainty (and the state of Delaware). Here, we barely know what we think we know and reliable invariants are hard to come by. Nothing can be taken for granted, certainly not growth.

So in our board rooms, pitch decks, and quarterly business reviews, let's ground ourselves in the harsh, brutal world of absolutes rather than the theoretically and mathematically convenient domain of relatives. We'll likely do better on both metrics, that way.

This month I wrote about the difference between absolute and relative growth and the fact that Silicon Valley (esp. VC) focuses WAY too much on the latter

Inspired by an old @HowardMarksBook memo...https://t.co/JlIT9T60Y4

— Nnamdi Iregbulem (@whoisnnamdi) September 24, 2021

Something nefarious, or something else?

The rich get richer, faster

Wealth and financial returns are positively correlated. Here's a plot of average annualized portfolio returns against (log) wealth, coming from a study using data from Addepar, a wealth management technology company:

There's a strong positive relationship between wealth and financial returns. To translate the range on the x-axis into something a bit more meaningful, you can think of the left side as representing investors with less than $3 million in assets and the right side representing investors with more than $100M in wealth.

Here are the exact returns at various levels of wealth:

Comparing the <$3M group to the >$300M group, the richest investors earn roughly 2 percentage points more on their investment portfolios annually, a sizable advantage.

High class asset classes?

Richer investors could be allocating their capital to different asset classes than other investors, and that could be driving the return differential.

Is it true that asset allocations differ meaningfully across wealth groups? Yes:

Richer investors put more of their money to work in alternative investments (venture capital, private equity, hedge funds, etc.), private companies (owned directly, not via venture capital or private equity funds), and housing.

However, it turns out that while asset allocations do tend to correlate with wealth levels, the differences in returns cannot be fully or even mostly explained by high-level asset allocation decisions.

To test for this, the authors calculated the expected returns of various wealth groupings based purely on the typical asset allocation of those groups. So stocks are assumed to get the returns of the S&P 500, bonds are expected to earn the returns of the US Aggregate Bond Index, and so on.

When they do this, they find that the expected returns are not divergent enough to explain the rich investor advantage:

Using this approach, the gap between investors with less than $3M in assets and those with $100M+ is only 0.35 percentage points, whereas the difference in raw returns was a full 2 percentage points. So asset mix does not even come close to explaining the differences.

We can also look at returns across wealth levels within each asset class. The table below shows the results of a regression of realized returns of individual investors on various wealth buckets, repeated for different asset classes. Interpret the numbers as the annualized returns of that group relative to the returns of investors with less than $3M in assets. For now, focus on panel A at the top:

Richer investors earn significantly higher returns in public equities, alternatives, and privately held companies. The only major category they don't dominate is fixed income, which perhaps makes some sense given the spread of returns at the individual bonds tends to be much more compressed that other assets.

You have to risk money to make money

So what gives? Why do the super rich earn so much more on their investments than everyone else, even after controlling for asset class?

Here's a thought: is raw, realized returns even the right metric to compare? (Hint: No)

Risk vs. reward is a classic dynamic in finance. Financial returns compensate investors for risks they take, with riskier investments generating higher returns, which encourages investors to buy them in the first place.

The Sharpe Ratio quantifies the degree to which a portfolio earns excess returns relative to the risk undertaken. The Sharpe Ratio divides the excess returns of a portfolio relative to some risk-free asset by the standard deviation of the portfolios excess returns, which represents volatility and risk, yielding "risk-adjusted" returns.

When the researchers do this, the relationship between wealth and returns completely breaks down:

The Sharpe Ratios of wealthier investor portfolios are no better than those of less wealthy investors:

“having controlled for risk, the point estimates of the Sharpe ratio for very wealthy investors is not higher, and the relationship between returns and wealth remains statistically insignificant”

So wealthy investors earn more but those earnings are explained by more risk-taking.

Remember, this risk-taking is happening within asset classes rather merely than across, so this result implies that the rich are investing in riskier individual assets (e.g. companies) within each category. For example, the wealthiest investors hold a much larger share of their public equity portfolio in individual stocks (vs. ETFs or mutual funds) relative to other investors, leading to more volatile portfolios:

High net worth investors also tend to invest in smaller, higher growth companies, naturally riskier. Interestingly, this behavior generates a lower market "beta" for rich investors in the CAPM model, as their portfolios correlate less well with the overall equity market than do those of less wealthy investors, who have betas close to 1:

“Higher-wealth households load more heavily on the SMB factor, reflecting an increased focus on small-cap companies. High net worth portfolios also load more negatively on the HML factor, meaning that they have more exposure to growth companies than lower wealth investors.”

Looking at all the alternatives

There's one more wrinkle though. Here's that last table again, showing returns across wealth groups and asset classes. This time focus on panel B, which focuses on risk-adjusted returns:

For public equities and privately-held companies, controlling for risk largely eliminates / accounts for the return premium earned by wealth investors. However, accounting for risk doesn't totally do the trick for alternative investments, where richer investors do better even taking risk into account. In fact, less wealthy investors do so badly in alternatives that the average investor with less than $3M in assets only earned an annualized return of 1.75%, which is not much higher than the risk-free rate during the same period of 1.33%, despite taking on significant risk.

We can dive a layer deeper by examining the returns across subgroups within alternatives, primarily hedge funds, venture capital and private equity, and real estate. Richer investors earn higher raw and risk-adjusted returns in hedge funds and VC/PE but do no better in real estate, where they actually earn lower risk-adjusted returns:

The least wealthy earn less than 1% annually on their hedge fund, venture capital, and private equity investments, which is stunning.

It's hard to pinpoint the exact drivers of the alternative investments advantage of the richest investors. What we do know is that the alternative investment portfolios of the richest investors differ quite substantially from those of less wealthy investors:

For example, investors in the sub-$3M group have only two alternative investments on average, while the $100M+ group has 35. On that basis alone we would expect the better diversification to generate lower volatility for the richest investors and potentially even higher returns based on my argument in "Why Don't VCs Index Invest?" (citing Abe Othman):

... investors increase their expected return by indexing as broadly as possible at the seed stage (i.e., by putting money into every credible deal), because any selective policy for seed-stage investing—absent perfect foresight—will eventually be outperformed by an indexing approach.

Another quirk: the alternatives positions of richer investors are updated much less frequently, meaning the valuations at which the companies are held are updated less often. This makes the portfolios of richer investors appear less volatile. Further, as folks in the industry know, companies that are doing well will typically see their valuations rise faster and their carrying values updated more frequently. So less updating means less volatility, and any volatility they do see is largely to the upside:

The smoothed returns of private equity understate the true economic risk and are an artifact of the lack of mark-to-market for illiquid assets — Demystifying Illiquid Assets: Expected Returns for Private Equity

Still, having stared at the data for a while, my sense is the alternative advantage represents a real returns premium and isn't merely an artifact of mark-to-market gamesmanship. Let's keep digging.

Real estate: location, location, location. Alternatives: access, access, access

We've crossed out anything nefarious in all the other asset classes. In all asset classes outside of alternatives, wealthier investors earn higher returns in the same way anyone earns them — by taking more risk. But alternatives remain a mystery. What's going on there?

Is it skill? Probably not, at least not in traditional sense of "stock picking". Most investments in alternatives are intermediated by managers — hedge, venture capital, and private equity funds — who make the actual investments. Therefore, it's unlikely that better stock picking ability on the part of the wealthy investors themselves explains their enhanced returns.

But that's also why the returns gap is so strange. If the individual investor is not actually making the investments, then why should their personal net worth correlate so highly with their portfolio returns in alternatives? The only explanation is differences in skill at picking great managers or access to those managers across wealth levels:

"Higher-wealth investors may receive preferential access to better-performing managers because they can offer larger amounts of funds at once, which reduces marketing and related overhead costs to the fund manager... In contrast, lower-wealth investors may only receive access to hedge fund and private equity solutions that are distributed through advanced marketing networks and are originated by large platform operators. However, funds that provide more accessibility may deliver worse performance"

For example, the authors find that fund-of-funds, investment funds that invest in other funds rather than in securities directly, earn "almost exactly two percentage points lower" returns on an annualized basis "evidence for fees imposed by additional layers of management." Another piece of evidence — the fewer investors who hold the same alternatives security, the higher than returns, and vice versa: "assets with limited investor participation are significantly related to higher investment returns." In other words, exclusivity generates better returns. This points to the importance of access.

One highly suggestive piece of evidence comes from the returns of investors whose funds are managed by a single family office (SFO), which is an investment firm setup solely to manage the wealth and investments of a single family or person. Portfolios managed by family offices earn significantly higher returns than portfolios of investors in the same wealth bracket who do not have a family office. Not only that, the return differential across wealth groups disappears when we look only at portfolios managed by family offices:

Depending on the asset class, among portfolios managed by family offices, investors with less than $3M in assets earn more than investors with over $100M in assets:

"While lower-wealth investors earn substantially lower returns on their hedge fund and private equity investments than ultra-high net worth investors, this does not apply to investors with smaller portfolios that are managed by an SFO. Even SFO investors with less than three million, or 3-10 million earn roughly the same return as those with more than 100 million on their investments in hedge funds and private equity, and substantially more than investors in the same wealth brackets but without SFO management."

Something about the nature of family offices and their influence on manager selection causes the wealth-returns relationship to disappear entirely. The study's authors speculate that this "likely [relates] to the difficulty of identifying and accessing high-performing alternative investment funds."

Access to assets matters. In the public markets, access is effectively democratized, so we can completely explain the advantage of richer investors in the public markets via differential risk-taking. However, the advantage in alternative assets reflects more than mere risk tolerance or preference. Ironically, family offices, which likely exacerbate inequality between the rich and the rest, level the playing field *among* the rich.

The 1% vs. the 0.1%

So do wealthy investors have an edge? Relative to non-wealthy, not really. It's risk-loving turtles all the way down.

But do the wealthiest have an edge versus their somewhat less wealthy peers? Yes, though I won't lose much sleep over it. It's an access advantage, and it's limited to the assets where access remains important: the relatively opaque and exclusive worlds of hedge funds, venture capital, and private equity.

Candidly, I'm more focused on expanding access to these alternative asset classes in general than on helping already well-off investors get access to the best funds. That said, the excess returns of alternatives are quite low for investors with only a few million in assets. On that basis, they aren't much better than public equities except for the extremely wealthy.

If these new investors only receive access to the worst-performing funds, the effort will be worthless. Worth pondering further.

The super-rich earn more on their investments than the rest of us.

In the battle between the 1% and the 0.1%, the richest get richer, faster.

Something nefarious? Or something else?https://t.co/Mc8imZTiH3

— Nnamdi Iregbulem (@whoisnnamdi) August 20, 2021

Thanks to Cynthia Balloch and Julian Richers, the authors of the study from which much of this essay is derived.

STEM PhDs aren't starting companies like they used to. Meanwhile, fresh PhD founders earn less than they did 20 years ago.

Cushy tech jobs are looking better and better to our most highly-educated as entrepreneurship loses its relative attractiveness.

Here's why.

Two decades of declining PhD entrepreneurship

STEM PhDs are creating fewer startups than they used to:

"We... show a decline over the past 20 years in both the rate of startups founded and the share of employment at startups by the highest-educated science and engineering portion of the U.S. workforce. The declines are wide-ranging and not driven by any particular founder demographic category or geographic region or scientific discipline."

Here's what the data shows: today, of STEM PhDs working in the private sector, only ~20% start companies, down from 30%+ in the late 90s:

"Figure 1 displays the large decline over the past two decades in the share of founders among PhDs in science and engineering. In 1997, 34 percent reported being a founder of a startup, but by 2017 this rate had declined to 21 percent, a decline of 38 percent."

Further, STEM PhDs are joining startups at lower rates in general, even outside of founder roles. Startups are much less attractive to PhDs than they used to be. This is true across demographic groups (gender, race, location, citizenship, etc.):

"Since 1997, the share of founders in these startups has declined by around 38 percent, not limited to any particular founder demographic or ethnic group or occupation, and this decline is widespread across regions of the United States. The share of workers at startups has followed the same path of decline."

"the employment share of science and engineering PhDs at startups is also falling over time. The downward trend is not driven by any particular category of PhDs. For example, figures similar to Figure 1 for males versus females, whites versus non-whites, California versus the rest of the U.S., and for U.S. versus non-U.S. born PhDs differ in levels but their dynamics are almost exactly the same as those in Figure 1... The dramatically falling share of founders and employment at startups raises the prospect of a drying up of high-tech, high-opportunity startups."

Why the change of heart among our PhD graduates? As the study authors speculate:

"A potential source of this decline is the exponential increase in the amount of scientific knowledge."

As the fields from which these PhDs come have become increasingly complex and extensive over time, the amount of scientific knowledge one must cram in to operate at the cutting edge of the field has exploded. Take any field and you see the same trend — becoming an "expert" is much, much harder than it used to be, and this burden of scientific knowledge weighs on potential founders.

Not only are fewer PhDs starting or joining startups, they're also waiting longer after completing their graduate coursework to found companies:

"Consistent with the burden of knowledge increasing for founders, the years of work experience among founders shows a steady increase. The regression results imply that the average founder had about 14 percent longer post-PhD work experience in 2017 than in 1997."

Here again we see the impact of the burden of scientific knowledge. With so much existing research and scientific knowledge out there to assimilate, PhD graduates are delaying entrepreneurship, taking more time to gather work experience before setting off on their own.

Learn to earn

Even worse, relative to their more experienced peers, newly minted PhDs who do found startups earn much less than they used to, making founding a company an even less appealing proposition:

"Figure 2 illustrates a pronounced decline over time in the earnings of less experienced founders relative to their more experienced peers. Here, we separate less and more experienced founders by the median number of years after PhD (13 years), although other reasonable cut-offs lead to similar results. Between 1997 and 2001, founders with post-PhD experience at or below the median earned more... but by the 2010s, the situation is reversed, with less experienced founders earning on average 30-40 percent less than other founders. Prior work experience apparently is becoming much more valuable for founders over time."

The figure compares the earnings of PhD founders with below-median and above-median work experience and demonstrates that this ratio has fallen meaningfully over the past two decades. Whereas the groups used to have earnings parity, today young PhD founders suffer a significant earnings disadvantage relative to their more experienced peers, making 30-40% less.

Absolute earnings for the highly-educated in general have risen over the same period, so this might not be terrible news. Young PhD founders could still earn more than they used to in absolute terms.

Not so. Earnings for PhD founders with below-median work experience declined in real (inflation-adjusted) terms, from $73K in 1997 to only ​$58K in 2017, a 20%+ pay cut:

"The earnings of less experienced founders declined not just in relative but also in absolute terms. The average inflation-adjusted earnings of founders with below the median post-PhD experience were $72,616 in 1997, whereas 20 years later their earnings were ​$57,517, a decline of more than 20 percent"

The earnings of young STEM PhD founders are their lowest level in more than 20 years.

What if we combine both inexperienced and experienced founders and look at the overall trend in PhD founder earnings? Surely this has trended positively?

Nope:

"Founders' earnings decline on average by about 1.6 percent per year (column 1). However, this is offset by an opposite time trend in returns to experience. The mean number of years after founders receive a PhD is 15.9 years; hence... at the mean work experience, the negative baseline time trend is completely offset."

OK, this takes a bit of unpacking. Earnings for less experienced PhD founders have fallen, while more experienced founders have seen rising earnings. The breakeven point is ~15 years, which is to say founders with fewer than 15 years of post-PhD experience have seen declining real earnings over time, while founders with more than 15 years of experience have seen earnings growth. It just so happens the average PhD founder in the survey had 15.9 years of post-PhD experience, so the two stories roughly balance out.

The good news: overall PhD founder earnings aren't declining! The bad news: PhD founder earnings aren't rising either!

The craziest thing about all this? PhD employees are doing much better:

"In stark contrast to founders, workers' real earnings grow over time, although the increase is relatively small, 0.4 percent per year... The trend toward increasing returns to experience is much weaker among workers than among founders..."

Again, let's unpack. STEM PhDs who work at established companies rather than startups have seen their earnings rise across the board, regardless of experience level:

As discussed earlier, founding a company as a young PhD is less attractive than it used to be. While young PhD founders used to make slightly more than young PhDs at established firms, this relationship has completely reversed, with inexperienced PhD founders now earning significantly less than their less entrepreneurial associates:

In summary, new PhD founders get the worst deal, earning less than both their equally and more experienced peers:

That was all a bit complicated, so before moving on, let's summarize the trends.

• Previously, experience mattered most within established companies, but over time the "gains from experience" grew much more for founders than workers
• Experienced PhD founders went from being no better off than their less experienced peers to earning significantly more, whereas the gap between experienced and inexperienced PhD employees stayed relatively stable.
• Inexperienced PhD founders used to make more than inexperienced PhD workers, but this relationship has flipped
• Inexperienced PhD founders are now the worst-paid PhDs

The division of tasks

So far we've established:

1. PhDs are founding far fewer companies, and

2. When they do found companies, young PhD graduates earn much less than they used to and significantly less than peers working at established companies

Not good. But it couldn't get any worse for our youthful, bright, wide-eyed PhDs could it?

Turns out, it can.

"established firms have an advantage over startups in creating a division of labor in R&D... by introducing more hierarchical layers, reducing knowledge workers' span of control, and allocating more experienced workers to positions with greater managerial responsibility. Further, established firms compensate workers for performing more R&D tasks and supervising more individuals. These developments are not seen among founders. The differences follow from the natural limits imposed by running a small firm with less division of labor and a high amount of multitasking by the founder. The largest firms are even more active in reorganizing job tasks, increasing the depth of hierarchy at twice the rate of all established firms."

Big Tech and other large companies have become more attractive places to innovate for our most well-educated workers. Large companies have better dealt with the accumulated burden of scientific knowledge by enforcing a "division of labor" among employees. Large companies form "knowledge hierarchies", such that no one person needs to know everything, and information can be aggregated up the chain in a rationale fashion. This lets organizations to scale to meet the needs of cutting edge scientific research and development, which increasingly incorporates impossibly vast sums of knowledge.

On the other hand, startup founders do not have armies of willing cadets to offload and delegate to. They are the captain of the ship and also it's most valuable crewmember, so as the burdens of both scientific knowledge and management complex grow, it all gets placed on their shoulders.

The job of a founder has gotten harder — running a startup today requires more management and R&D activities than it did 20 years ago. Quantified, this is about 15% more "tasks", with R&D tasks growing by about 50% and management tasks growing 5%:

"The average number of all tasks for founders increased by about 15 percent from the beginning to the end of our sample, a statistically significant difference... Furthermore, although the number of both R&D and management tasks increased, the increase is more pronounced in R&D, for which it rose by more than 50 percent from 1997 to 2017... As a result, R&D tasks that comprised about 25 percent of all tasks conducted by founders in 1997 increased to 34 percent of all tasks in 2017. As can be seen in Figure 3, this was not accompanied by any decline in management tasks, so the founders had to shoulder the burden of doing more R&D tasks while also running the same or more administrative tasks."

PhD workers at established firms also saw their total tasks grow, but to a lesser degree. R&D tasks for workers only grew 12% over the same 20 year period, and management tasks did not increase at all:

"Workers were also affected by the need to perform more R&D tasks, as the increase in the number of R&D tasks from 1997 to 2017 is about 12 percent. Notably, however, as the figure reveals, workers did not need to handle more management tasks, which remained relatively flat for them."

What gives? It goes back to the division of labor point I mentioned earlier. Let's start with R&D:

"The number of R&D tasks is rising, but established firms take advantage of a division of labor and knowledge hierarchies"

The growing burden of scientific knowledge increases the number of R&D tasks that founder must occupy themselves with. With little hierarchy or staff to offload this work, founders have no operating leverage. Conversely, large, established firms can shift much of the burden to armies of researchers, more efficiently dealing with the growing stack of research output.

"We find that the number of different R&D tasks has increased more for founders than for workers. And the returns to experience have increased over time for founders but not for workers, highlighting the increasing need for a single person—the founder—to cope with the burden of knowledge in startups. Workers at established firms have, instead, comfortably narrowed their span of control, employed more people indirectly under their control to support their work, and kept administrative duties low. They are also better rewarded for taking on more diverse work and managerial responsibilities than founders. ...established firms have coped more effectively with the increasing burden of knowledge in science by better utilizing the division of labor in innovative work through reorganizing tasks and hierarchies."

Individual workers at established firms have limited scope, while the remaining load gets shared with other team members and subordinates. Further, the firm captures the gains from the successful division of labor and then shares this with employees in the form of better pay for the managers who corral these efforts internally.

"Running a startup might constrain founders' ability to organize its hierarchy efficiently, at least until it has succeeded in growing well beyond its initial size. As science accumulates more knowledge, we would therefore expect PhD founders to have to take on more R&D tasks."

Founders have no ability to pull off similar organizational tricks until their startups reach meaningful scale. Thus, as scientific knowledge has accumulated, they've taken on the additional R&D work themselves.

This has implications for startup management too:

"Also, founders had to deal with significantly more management tasks than workers in terms of levels: about 30-40 percent more at the beginning of the study period, increasing to 50 percent more at the end. The explanation for this difference likely lies in how the two types of firms differ in their organization of work..."

Founders and startups cannot afford the overhead that comes with increasing layers of professional management. This is why startup's delay building executive teams until at least a few years into a startup's lifecycle. Large organizations can and do invest in these management layers, gladly taking on the extra expense:

"... the firm responds by increasing the number of layers... and... [allowing] greater job specialization. Increasing the number of layers of management adds a fixed cost of operations... [Larger] firms are more likely to become hierarchically taller by adding more layers of management... as they can more easily absorb the added fixed cost. Founders at startups appear not to have recourse to this mitigation strategy."

The study's authors quantify these additional layers by tracking two metrics:

• the number of individuals directly supervised by PhD founders and workers, and
• the number of individuals indirectly supervised (i.e. supervised by one's own direct reports)

The directly supervised corresponds to the managerial burden that each PhD-holder has (their span of control), while the indirectly supervised tells us how much managerial leverage each person has (the depth of hierarchy). The theory would suggest that workers within organizations should have fewer direct reports over time and more indirect reports, as organizations build out these knowledge hierarchies.

And that's exactly what we see:

• Workers with PhDs saw the number of employees directly managed decline over time while the number of employees indirectly managed increased over time. In others words, the proliferation of middle management reduces the management burden for any individual worker but increases overall managerial leverage.

• Founders on the other hand saw no statistically significant trend in their span of control or depth of hierarchy. The organizational structure of startups has not evolved to keep up with the increased burden of scientific knowledge.

"workers [perform] fewer R&D tasks as they age... the span of control decreases for workers at established firms... However... hierarchies deepen over time for workers at established firms... Together, the results suggest that established firms cope with the increasing burden of knowledge on their workers by introducing additional layers of hierarchy, while simultaneously reducing the number of employees who report directly to managers"

Doing *too many* things that don't scale

We need to think harder about making knowledge work... work.

In summary:

"Our findings suggest that if the goal is to restore business dynamism in the high-tech sector, alleviating the burden of knowledge should be front and center in the strategy to attain it."

The declining in PhD entrepreneurship mimics a broader drop in startup formation chronicled elsewhere. But PhD founders are special... and so these trends are especially worrying.

STEM PhDs have founded some of the world's most successful and impactful enterprises. Among them: Google, Intel, VMware, and others.

Ironically, however, today these same companies gobble up the brightest minds we have, discouraging or delaying their own entrepreneurial pursuits.

In Silicon Valley we talk a lot about management practices to help founders and executives "scale." But deeply technical founders face a separate but equally important problem: scaling knowledge.

PhDs aren't starting companies like they used to

Cushy tech jobs are more and more attractive to STEM PhDs

The burden of scientific knowledge and managerial complexity is crushing our best and brightest

I explore the precipitous decline in PhD founders:https://t.co/NxpFn3NrZv

— Nnamdi Iregbulem (@whoisnnamdi) July 8, 2021

Thanks to Thomas Astebro, Serguey Braguinsky, and Yuheng Ding, the authors of the study from which much of this essay is derived.

In Four Challenges Facing Developer Productivity Startups I identified scaling developer relations as one of the key challenges facing developer productivity startups today:

... even if you're fully onboard with the idea of developer relations, doing it well is tough. To start, developer advocates are hard to come by. There just aren't many DevRel folks out there. The function is still relatively new, and career pipelines into the role have yet to fully materialize — Four Challenges Facing Developer Productivity Startups

To some extent this is a supply and demand issue — many more startups want to bring on developer advocates than there are available developer advocates.

But there are structural issues behind the small quantity of available developer advocates — it leverages a combination of skill sets not often found in a single package, making it a tough role to hire for:

DevRel is a hard spec to hire for. Technical folks are usually introverts. Extroverts are typically not technical — Developer Advocate, Analytics Infrastructure Startup

I want them to be technical, and I'm always looking for empathy. It's very hard to find these things in a single person, this combination — Head of Developer Relations, Web Development Startup

Even among the available pool of developer relations talent, companies have a hard time knowing who will be better at the role and what the key success factors are. How much does prior experience matter? How critical is it to have a large social media following?

I've spoken to numerous founders and developer relations professionals to tease out best practices for DevRel hiring. This piece focuses on how to find high quality developer advocates despite the scarce supply.

Here's some concrete advice:

Most developer advocates are former engineers. So, talk to some current engineers

Current engineers are future developer advocates hiding in plain sight:

Getting into this career can be hard, but there are many developers that I'd bet on to be great first time developer advocates — Tessa Kriesel @ Devocate

At some point, everyone gets their first gig in Developer Relations, and most of those people were developers themselves just before. Thus, current engineers are a great potential source of DevRel talent, if you're willing to do the work to pitch them on the role.

In fact, many people who could be great developer advocates don't know how to break into the field. For example, many engineers simply haven't spent the the time to build up a large social media following, but this doesn't mean they couldn't be successful in the role:

... the ones who could be good often have a hard time getting a foot in the door without a Twitter presence — Developer Advocate, Application Infrastructure Startup

Some companies have realized this and shifted most of their DevRel hiring internally, focusing mainly on convincing existing engineers to try it out.

The pitch? Developer advocates can often have impact much sooner and more frequently than engineers:

We hire most of the DevRel team internally. Folks realized they could make an impact much sooner as DevRel than as engineers. Software releases take so long, but content takes much less time — Head of Developer Relations, Web Development Startup

Developer advocates "ship" faster. A piece of content — a blog post, a Twitch stream, etc — is often much quicker to assemble and publish than shipping some piece of software, like a new feature, to production:

Sometimes it's enough just to write an article just to start a conversation. It doesn't have to be a full fledged feature release like in software — Head of Developer Relations, Web Development Startup

I haven't run the numbers, but it wouldn't surprise me if there were more potential developer advocates among current software engineers than actual developer advocates out there.

So if you're having difficulty finding candidates for your open DevRel role, talk to some current engineers, even one's working at your company.

And when you do, leave your biases at the door with respect to experience or social media following...

Don't filter for DevRel experience or social media following

As with any role, it's super tempting to assume "only someone who's done this before could be good at it." Not so with DevRel:

You can't look for experience, DevRel has barely existed for very long. If anything, non-traditional folks are often the best — Developer Success Engineer, Application Infrastructure Startup

DevRel as a formal practice has not existed for long. You can't expect most people to have meaningful experience doing it. Again, this reinforces the idea that many potentially great developer advocates aren't currently in the role.

Another important point to keep in mind is the job description for DevRel tends to be loose and ill-defined in many ways due to the nature of the role:

The job description is extremely loose. Best people need to grow into it to some extent — Developer Success Engineer, Application Infrastructure Startup

Regardless of what you think the person needs to do, the reality on the ground will look somewhat different. Accept this, and know that the best developer advocates will navigate this ambiguity regardless of whether they are an "obvious" fit based on their professional experience to date.

One of the "obvious" characteristics people often filter on is social media following. But not so fast:

Social media is just one part of the toolbox. Further, social following in one area does not necessarily translate to another — Developer Advocate, Analytics Infrastructure Startup

It's debatable how much a social media following is in fact required for success in DevRel. It's typically helpful for sure, but it's not everything. Being able to authentically and credibly connect with other developers is much more important:

You don't really need to have a following to be good at DevRel. It's much more important to have street cred [with developers] — Founder and Former Developer Advocate, Application Infrastructure Startup

In fact, social media can sometimes backfire (as we all know). In DevRel in particular, it can sometimes create the impression the person was merely "acqui-hired" for their social media sway rather than technical skills developers tend to appreciate and respect. I wouldn't say this is enough of a risk to be too worried about, but it's worth keeping in mind:

Social media helps, but it can also be problematic in certain ways. It can sometimes create the perception that they were just hired for the followers they have, rather than their technical capacity. That can problematic for the person and the company — Product Manager and Former Developer Advocate, Big Tech

Look for the core characteristics, regardless of current title

Rather than hire based on surface-level characteristics like current title or social media following, focus on identifying raw talent and passion for a key aspect of the role — working in public:

There's a lot of raw talent out there. They are often already blogging. They are trying to get permission to speak at conferences — Director of Developer Relations, Web Development Startup

Look for people who are already doing DevRel without necessarily having a title for their efforts (yet). These are engineers who are writing blog posts (either for their employer or their personal blog), speaking at conferences, and publicly interacting with other developers. This is where social media can be value — it's not so much about having a following as much as it's about having a participatory presence:

Look for people who are already stepping up and doing the work. They are speaking at conferences, authoring blog posts, interacting on Twitter. Doesn't matter what their title is — Product Manager and Former Developer Advocate, Application Infrastructure Startup

In addition to behaviors, attitudes matter too. We've already discussed the importance of empathy, but a core desire to teach and help others is important too. Self-taught programmers are often great about this (since they know how hard it is to learn something on your own):

Self-taught programmers are great. They often have a chip on their shoulder and a willingness to teach others. They understand how valuable help is — Developer Success Engineer, Application Infrastructure Startup

The founders and DevRel folks I spoke to agreed: these core characteristics are more critical to success in the role than prior experience or social media following. You likely have engineers in your team today with these traits. Maybe it's worth having a conversation about DevRel?

Acres of DevRel diamonds

Constraints drive creativity and innovation. The market for developer relations talent is certainly constrained, and we need to get creative in response. Expanding your perspective for what DevRel "looks like" beyond social media following and title will help you find acres of DevRel diamonds where you least expect.

Again, remember this simple logic:

• IF most current DevRel professionals were past engineers,
• THEN most future DevRel professionals are current engineers.

The sum is striking but also predictable in light of data showing we don’t even come close to maximizing developer productivity.

Mountains of technical debt and poor development practices burden and bog down developers. We lose billions as a result.

The Developer Productivity Manifesto has three parts, this is part 3:

• Part 1: The Developer Productivity Flywheel
• Part 2: More (Developers) Isn’t Always More
• Part 3: Leaving Software on the Table (you are here)

In part 1, I talked about falling developer productivity and how spinning the developer productivity flywheel can counteract this trend.

In part 2, I argued that more developers won’t solve all our software engineering problems.

I’ve thrown around a lot of equations, literary references, and even some memes. In this third and final piece, I'll talk dollars and cents, quantifying the impact of lost developer productivity and how much software we're "leaving on the table" as a result.

(Technical) debt to (developer) GDP

In part 2, I made the case that software maintenance is not sufficiently value generating to cover engineering salary expenses. In other words, busy work doesn’t work:

According to one analysis, an engineer engaged in purely non-innovative activity destroys nearly $600K in employer market value. On the other hand, the average engineer, working on a combination of maintenance and innovation activities, adds $855K in market value to their employer.

In a global, cross-industry survey, Stripe asked developers why productivity was lower than it otherwise could be. Maintenance of legacy systems / technical debt took the top spot:

Source: The Developer Coefficient, Stripe

Even developers themselves see maintenance work as unproductive. Asking them to quantify this reveals that the typical developer spends 13.5 hours per week addressing technical debt:

Source: The Developer Coefficient, Stripe

Add to that another 3.8 hours per week fixing “bad code” (debugging, refactoring, etc.). That totals 17.3 hours per week spent fixing the past rather than building the future. The typical work week among the surveyed was 41.1 hours, implying that a full 42% of developer time is lost to drudgery:

Source: The Developer Coefficient, Stripe

I like to call this technical debt to developer GDP. While some level of technical debt is unavoidable, it eventually becomes an unbearable drag on software output and developer productivity. It eats up engineering time leaves little for generative development work.

Developers could be 46% more productive

In another question, Stripe asked developers to rate the productivity of their engineering teams on a scale of 0–100%. The average response? 68.4%:

Source: The Developer Coefficient, Stripe

Said differently, the average developer could potentially be (100% — 68.4%) / 68.4% = 46% more productive that they are today, nearly 50%.

With a 41.1 hour work week, such a productivity boost would be the equivalent of an additional ~19 hours of productive development work, enough to completely compensate for all that time spent on technical debt and bad, buggy code.

A $425B dollar bill on the ground

Based on the survey, Stripe conducted a back-of-the-envelope calculation multiplying estimates of the value generated by software developers around the world with the estimated productivity losses to arrive at an estimate for the global GDP lost due to software developer inefficiency.

I take issue with some of the assumptions, but the calculation is illustrative regardless. Assuming $900B of aggregate developer GDP and 31.6 percentage points of productivity lost suggests a $300B annual GDP shortfall:

Source: The Developer Coefficient, Stripe

This is an underestimate, in fact. Stripe’s math assumes 100% — 68.4% = 31.6% lost relative to existing productivity, but as I showed above, it’s in fact 31.6% / 68.4% = 46%. With this efficiency loss relative to maximum productivity, GDP lost to developer inefficiency grows to ~$425B.

So our failure to maximize productivity is losing us hundreds of billions in software production. Not exactly chump change.

SUM()-ing it all up

To end, I want to return to where I started qualitatively, my goal to increase total software output, and quantitatively, the decomposition of software output into developers and developer productivity, and finally connect the two perspectives.

How much more software output could we get with more developers and higher productivity?:

Let’s start with developers. I tend to be skeptical of perennial “shortages” unless there’s some specific, identifiable reason for it. Regardless, estimates suggest there’s a ~3M shortage of software developers globally, with 1.4M unfilled computer science jobs in the US alone.

To make the numbers easy, let’s round up Stripe’s estimate for the global software engineering labor force from 18M to 20M (I’ve seen other estimates in the 20–25M range, so this feels reasonable). That would imply a 3M / 20M = 15% developer shortage at current levels of demand.

So we have a 15% developer shortage and a 50% productivity “shortage”. Multiplied, that yields a massive 73% potential gain in software output:

Said another way, current software output is only about 100% / 173% = 58% of what it could be.

That’s $670B of software we’re leaving on the table.

This calculation is admittedly simplistic. One could argue that we’d need fewer developers if they were more productive. I push back on that — there’s so much we’ve yet to build, and more developers working more productively would unlock entirely new opportunities that we haven’t had the capacity to explore or can’t even yet imagine. This would further spur developer hiring and employment, another flywheel.

We can do better

In this series I’ve made the case that by investing in technical tools for technical people, we can reverse the trend of declining developer productivity, spin the productivity flywheel in the right direction, and see massive gains in software output as a result.

As I’ve hopefully made clear, we can do A LOT better of a job maximizing developer productivity. I hope you’ll join me in my mission to do exactly that.

Thanks so much for reading this — I hope it resonated with you. If it did, please share that feedback!

Follow me on Twitter, subscribe to my monthly essays here, and reach out to me directly via nnamdi@lsvp.com

In part 1 of “The Developer Productivity Manifesto,” I argued that developer productivity, measured in terms of new software, falls over time as low-hanging fruit is picked and new ideas become harder to find.

An understandable response would be — why not just throw more engineers at the problem?

If you’ve ever worked in software development, you know adding more cooks to the kitchen rarely helps. For those who haven’t worked in software development, this essay tells you why.

The Developer Productivity Manifesto has three parts, this is part 2:

• Part 1: The Developer Productivity Flywheel
• Part 2: More (Developers) Isn’t Always More (you are here)
• Part 3: Leaving Software on the Table

The Mythical Man-Month

“The Mythical Man-Month” is a classic and hugely influential book about software engineering and project management. Across a series of essays, Fred Brooks (himself a former IBM project manager) lays out the cognitive errors software development teams tend to make in estimating the time to completion for software projects. His core thesis is best encapsulated by Brooks’ law.

Brooks’ law of software project management states that:

Adding manpower to a late software project makes it later

If anything, adding more cooks to the kitchen lengthens cooking time. We shouldn’t assume a simple, linear, and directly causal relationship between developers as inputs and software as output. He lays this out in dire terms:

… when schedule slippage is recognized, the natural (and traditional) response is to add manpower. Like dousing a fire with gasoline, this makes matters worse, much worse. More fire requires more gasoline, and thus begins a regenerative cycle which ends in disaster.

This counterintuitive phenomenon has multiple causes:

• Ramp up time: Even seasoned developers take time to get up to speed on new projects, and newbie engineers must learn core skills on top of company-specific ones
• Communication and coordination complexity: The larger the team, the harder it is to coordinate productive work and communicate progress across teammates
• Indivisibility of work: The basic unit of work in software development can’t always be divided among multiple contributors

That last point forms the basis of the title and main thrust of the book, the fallacy of “man-months” — distinct units of work achievable by a single developer in a set period of time. As Brooks argues, there is no such thing, and thus we should be wary of simplistic solutions to complex endeavors like software development.

Software is not labor-intensive. Not many people are necessary in order to produce good software… What makes or breaks a project, it’s the amount of FOCUS developers can pour into it — RedBeardLab

Diseconomies of scale

Source

While it is common to assume economies of scale, diseconomies of scale are arguably just as relevant in software development.

Diseconomies of scale are where unit costs (the costs of producing an additional unit of output) increase rather than decrease with scale. Here, it’s better to cut back or lower output, rather than maximize it. Despite many advances, these occur more often than we’d like to admit in modern software development.

Diseconomies of scale can take many forms, and many overlap with the underlying causes behind the mythical “man-month”:

• Complexity: Things become disproportionately complex as they scale, as complexity increases non-linearly with size. Complexity creates overhead, making large organizations less efficient than medium-sized ones. Bureaucracy is one manifestation, but there are others.
• Black Swans: Large systems fail in spectacular fashion. It’s why big companies create systemic risk while small businesses and startups fail in the thousands without cause for alarm. Within software, this might be a large monolithic application, prone to serious, single point failures.

Humans, being the independent and unpredictable automatons we are, are especially prone to diseconomies of scale. Coordination costs eventually overwhelm even the most thoughtful engineering leaders. Application deployments are themselves quite brittle, necessitating vastly more manpower and attention as they grow.

We will never have enough software developers

Senior executives report that the lack of developer talent is one of the biggest potential threats to their businesses — The Developer Coefficient 2018, Stripe

Executives believe that insufficient developer talent is one of the biggest threats to their business, and yet there’s good reason to think this might never be solved.

As I discuss in a previous essay, Why We Will Never Have Enough Software Developers, changes in the underlying technologies of modern software development whittle away the accumulated human capital of software developers:

Specific skills in software development quickly become dated. Programming languages and development frameworks go out of style. Hadoop is hot one year, and it’s old news the next. Like a fast, expensive car that quickly loses value as it’s driven around town, the skills and human capital of software engineers fall apart without constant, expensive maintenance

Though young engineers can keep up with the latest programming languages, frameworks, and tooling, eventually the torrential wave of new tools becomes too much to bear, and developers either tune out or drop out:

At age 26, 59% of engineering and computer science grads work in occupations related to their field of study. By age 50, only 41% work in the same domain, meaning a full ~30% drop out of the field by mid-career

The never-ending drumbeat of new technologies drives developers out of the field. New tooling is important and valuable, but their overall effect on developer productivity depends on how much they upend existing workflows and place additional burden on already taxed developers.

As I conclude:

Growing the supply of software developers is not trivial because the field already sees high levels of developer dropout and turnover, and this would only increase if the field were to grow larger.

I favor efforts to grow the software engineering talent pool, but if we’re not careful, like quicksand, our efforts will be counterproductive.

More developers, lower productivity

As we saw in part 1, more researchers don’t necessarily lead to faster progress. In the case of Moore’s Law, it merely maintains the rate of progress, at significant expense.

But is this true of the economy overall? It turns out, the answer is yes. Cross-industry data proves that simply throwing more people at a problem doesn’t work.

The chart below plots the growth in the number of employed researchers and their productivity levels (measured in terms of revenue growth generated per researcher) for ~1,700 publicly-traded U.S.-based companies over a 20-year period. Most firms “threw more bodies at the problem” (orange histogram > 1), but ~80% of firms saw declining research productivity (blue histogram < 1):

Source

Hiring more researchers doesn’t necessarily cause productivity to decline, but regardless, growing research teams and declining productivity go hand-in-hand more often than not.

Software development could fall prey to the same trap. Most companies are growing the number of software engineers they employ, mirroring the researcher employment trends, but let’s not forget productivity:

Research productivity, and analogously developer productivity, falls over time unless acted upon by an outside force. Like gravity, this phenomenon is pervasive and ubiquitous, a force field dragging down all idea-generative industries.

Busy work doesn’t work

I love this quote from Stripe:

Despite the number of developers increasing year-over-year at most companies, developers working on the right things can accelerate a company’s move into new markets or product areas and help companies differentiate themselves at disproportionate rates. This underscores the most important point about developers as force-multipliers: It’s not just how many devs companies have; it’s also how they’re being leveraged — The Developer Coefficient 2018, Stripe

In my last essay I talked about the importance of measuring productivity in terms of new software output. Another reason why more engineers is not necessarily the solution is that much of the value of the marginal engineer is in their innovation activities rather than their maintenance work.

We can quantify the impact of engineering time spent maintaining old code rather than writing new code. According to one analysis, an engineer engaged in purely non-innovative activity destroys nearly $600K in employer market value. On the other hand, the average engineer, working on a combination of maintenance and innovation activities, adds $855K in market value to their employer.

As the study’s author speculates:

… the value of the engineer is a bundled combination of maintenance activities which have negative value and innovation activities with positive value

Further, he echoes Brook’s Law:

It may also be the case that more engineers does not always make for easier problem solving and on the margin, removing innovative activity, [they are] a net drain on the firm’s value

I want to be clear: maintenance matters too. When things break, as they inevitably do, development teams must stand at the ready to fix problems and bring systems back online. This is critical work that should not be minimized in a narrow pursuit of newer, shinier objects.

That said, mere maintenance is table stakes. It doesn’t pay the bills — an engineer’s salary, first and foremost.

Hidden figures

Again, Stripe gets it right:

While many people posit that lack of developers is the primary problem, this study… found that businesses need to better leverage their existing software engineering talent if they want to move faster, build new products, and tap into new and emerging trends — The Developer Coefficient 2018, Stripe

Notice the emphasis on newness and speed. We can and should grow the talent pool for software engineering, but we can also do a much better job with the engineering talent we already have.

Maintenance matters but so does fundamental innovation. As software projects grow, and their teams with them, thoughtful engineering managers must strike the right balance or see their most precious resource go to waste.

Industry-wide we are, unfortunately, out of balance. In my next and final piece, I’ll explore exactly how much software we’re “leaving on the table” as a result.

Ready for more? Here's Part 3.

Follow me on Twitter, subscribe to my monthly essays here, and reach out to me directly via nnamdi@lsvp.com

I’ve been obsessed with developer productivity for a long time.

As developments within software tooling, application infrastructure, and data science over the last decade have vindicated my quixotic enthusiasm, I thought I’d finally articulate my rationale in writing.

Manifestos should have mission statements. I’ve always loved Stripe’s:

Our mission is to increase the GDP (gross domestic product) of the internet

They say great artists steal. My artistic design skills are, frankly, terrible (especially relative to Stripe’s), but I’ll steal some inspiration anyway and lay out my mission as follows:

My mission is to increase total software output

Developer productivity is the best way to achieve this. This essay explains why.

The Developer Productivity Manifesto has three parts, this is part 1:

• Part 1: The Developer Productivity Flywheel (you are here)
• Part 2: More (Developers) Isn’t Always More
• Part 3: Leaving Software on the Table

Every company is becoming a software factory

It’s been said before, and it’s worth repeating: every company is becoming a software company.

Like Henry Ford’s assembly line 100 years ago, which revolutionized mass production of physical goods, innovation in the production of complex, intricate software products and services promise to transform modern software development.

The Software Revolution is the new Industrial Revolution.

Agile, scrum, waterfall etc. — the various paradigms analogize the factory assembly line and modify it for the modern context. They pose the provocative question, “What can the world of bits learn from the world of atoms?”

In this light, I modify the above: every company is becoming a software factory.

But analogies to old-school industry only take us so far.

In the factories of old, labor was poorly-skilled, poorly educated, poorly treated, and poorly compensated. Today, as before, labor remains the key input to production. However, in the modern software factory, labor (i.e. software developers) is highly-skilled, well-educated, well-treated (at least in more progressive companies), and mostly importantly, well-paid.

My goal as an investor and citizen of Silicon Valley is to increase total software output. Maximizing the software output requires understanding what I call, the software production function — how we map from inputs to outputs in software development. The exact function is likely unknowable, but we know for sure developers are the key input. Therefore, let’s start with this simple mapping:

Basic enough, but we can do better. Economists often decompose variables into the extensive (“how much input” / “how many software developers”) and intensive (“how much per input” / “how productive”) margins:

So, we can grow software output in two ways: more developers or higher developer productivity.

Why we should care about developer productivity

“While many people posit that lack of developers is the primary problem, this study… found that businesses need to better leverage their existing software engineering talent if they want to move faster, build new products, and tap into new and emerging trends” — The Developer Coefficient 2018, Stripe

Software developers are the scarce, precious resource of software development — extremely expensive to obtain, train, and retain. Anything that makes software developers more productive will itself be highly valuable.

However, productivity in software development tends to decline over time rather than increase.

This is a counterintuitive claim, so let’s unpack it.

First, it helps to delineate two different types of production — the production of tangible and intangible goods:

Tangible goods include traditional, physical products and services like cars, televisions, clothing, etc. Importantly, due to their tangibility, replicas have value. Two cars are better than one, and so on. It therefore makes sense to talk about gross “units” of production — cars assembled, televisions manufactured, etc.

On the other hand, intangible goods constitute non-physical products like ideas, patents, and… software. Intangibles are infinitely reproducible at nearly zero marginal cost. As such, it doesn’t make sense to measure intangible output in terms of gross “units’’ of output. Rather, only net new output matters.

Software is one such intangible. Copying existing code is as easy as running git clone on a repository. This alone does not generate incremental value — the value was in writing the original code.

With intangibles, novelty is what matters: new ideas, new designs, and new software. There’s a reason you can’t patent something already patented — there’d be no value in doing so. Similarly, new code moves the ball forward.

Let’s edit our previous equation to emphasize novelty:

It turns out, this trivial change makes a huge difference.

Why? Economic evidence suggests that, unlike physical goods, idea productivity tends to decline over time. The intensive margin of the “idea production function”, i.e. the number of new ideas generated by a given number of researchers, falls dramatically as a field progresses:

​Don’t believe me? Here’s one example that should resonate with fellow technologists: Moore’s Law.

Are ideas getting harder to find?

Moore’s Law describes how manufacturers cram twice as many transistors onto computer chips every two years.

Sustaining this self-fulling prophecy of “2X every 2 years” requires massive research teams to churn out new ideas and insights around chip design and manufacturing. It’s clear from the data that Moore’s Law is not some a priori law of the universe, but rather a goal set by chip manufacturers and researchers:

“Many commentators note that Moore’s Law is not a law of nature but instead results from intense research effort: doubling the transistor density is often viewed as a goal or target for research programs.” — Are Ideas Getting Harder to Find?

Transistors are tangible, but the ability or know-how to condense their size over time is an idea and, therefore, intangible. Intellectually, we should separate the physical production of literal computer chips from the ideas that enable this feat.

As it turns out, the old ideas just won’t do — what got you here won’t get you there.

In a fascinating and landmark paper, Stanford and MIT economists Nicholas Bloom, Charles Jones (hey professor!), John Van Reenan, and Michael Webb frame the constant growth in the density of transistors as evidence of a constant flow of new ideas and innovations:

If idea output is constant, growth in the number of researchers implies shrinking research productivity. If the flow of new ideas in semiconductor manufacturing is constant (as evidenced by Moore’s Law roughly holding steady) and the number of chip researchers increases over time, then we know (via simple arithmetic) research or idea productivity must decline:

​And that’s exactly what we see. Generating constant growth in the number of transistors on a chip has required many more semiconductor researchers and scientists over time, about 18 times as many in 2014 relative to 1971:

Measured in terms of transistor density gains, research productivity has declined 18X over a 45-year period, which is to say it’s about 5% of what it used to be, a dramatic, precipitous decline in idea productivity. And as presented in the paper, the same phenomenon holds true in many sectors, such as agricultural and medical research.

The evidence is clear: new ideas are hard to find and only getting harder. Analogously, if we assume new software represents new ideas, it only follows that new software is increasingly difficult to create too.

The work of software developers is analogous to semiconductor R&D. Engineering teams are tiny idea factories, and new ideas get harder to produce over time.

This is why developer productivity falls over time. The ability of software developers to write novel applications tends to decline over time, as happens in almost any idea-centric production process. When so much code has already been written, it’s difficult to improve upon the status quo.

The developer productivity flywheel

Breaking away from this gravitational, productivity-sucking force field won’t be easy. However, we can reach escape velocity by aggressively enhancing developer productivity rather than watching it deteriorate.

Flywheels help technology startups achieve velocity, and software development is no exception. Since no mention of flywheels in a technology essay is complete without a visualized loop, here it is:

To explain:

1. New developer productivity tools make software developers more productive.

Simple enough, though we shouldn’t be so naive as to think it always works out so cleanly. New tools initially impair developer productivity, as individual contributors and teams adjust to new software, interfaces, and workflows. Complex tooling can drive productivity through the ground if recklessly implemented.

This dynamic generates the familiar “J-Curve” of initially declining productivity before tangible benefits are eventually realized:

Original version

So I should be precise — new productivity tools thoughtfully applied make developers more productive. That productivity may not immediately materialize, either because it’s difficult to measure (Commits? Lines of code? Shipped releases?) or because adjustment takes time. This only reinforces the importance of investing sooner, as to frontload the trough.

2. Higher developer productivity drives companies to hire more software engineers.

Remember your supply and demand curves from economics 101? Higher productivity pushes out the developer demand curve. Companies take advantage of enhanced productivity by hiring more developers and paying them more too:

In some cases, the enhanced productivity not only makes existing developers more productive but lowers the bar enough that others can now become software developers too. This pushes out the the developer supply curve, increasing employment further:

3. More developers working at higher productivity levels ship more software, a subset of which is itself developer productivity tooling.

As previously discussed, if we simply increase the number of developers without maintaining productivity, we merely tread water. What’s crucial here is that developer productivity and employment rise together. If they do, we get more software output, and that new software will itself drive higher productivity.

4. Loop

For the technically inclined, here’s how the logic goes in code form (much more compact 😅):

In other words:

This is what I like to call, the developer productivity flywheel.

The future of productive, economical software development rests on running this flywheel as quickly as possible, iteratively looping through virtuous cycles of productivity, employment, and software output growth.

However, there is “no silver bullet.” There isn’t “one weird trick” to massively supercharge developer productivity.

Instead, we must attack the problem from every available angle if we are to succeed.

Ready for more? Here's Part 2.

Follow me on Twitter, subscribe to my monthly essays here, and reach out to me directly via nnamdi@lsvp.com

To better understand the strategic landscape, my good friend and colleague, Clio Smurro, and I interviewed founders and executives at next-generation software and infrastructure startups pushing the developer productivity frontier to get their thoughts and insights. They shared their views on:

• major industry trends,
• top strategic priorities, and
• biggest challenges and pain points (you are here)

In this third and final chapter, we share our findings on the top challenges and pain points facing developer productivity startups, including:

• Defining and planning for success
• Integrating with other technologies
• Building mindshare
• Scaling developer relations

Subscribe below to receive a nicely formatted PDF of our research!

Defining and planning for success

It's common for developer productivity startups to take an engineering-first approach to development, where engineers take the lead on deciding what to build and when to build it, often leveraging personal intuitions as consumers of the product.

We don't have PMs. We decide what to build democratically. For each sprint, everyone comes up with an idea and we discuss — Manager, Developer Tools Startup

While this works initially, without some sort of formalized product management function, teams soon hit a wall of confusion:

We emulated the Stripe model initially, focusing on engineering and growth. We don't have any PMs and so we don't have a clear framework for what to build — Manager, Developer Tools Startup

It's difficult to chart out long-term priorities and product roadmaps that support the business and revenue generation without product-minded individuals that can liaison between the technical and commercial needs of the company.

If not tackled early, the more haphazard approach to product development can lead to a nonsensical multi-year roadmap of features no one can rigorously justify. With no North Star, new, potentially interesting ideas get shut out and ignored, for lack of space:

We know all the things we would like to build but our roadmap is full — Director, Data Science Startup

To where should developer productivity startups look for signals on what to build next? Customers, ideally. But it's not always that simple. Depending on the customer, signals can be more or less easy to detect. Self-serve users, for example, often silently churn, never indicating what drove the decision:

Self-serve users don’t really tell you what they want, they just churn if they are unsatisfied — Manager, Developer Tools Startup

Telemetry can drive insight into these churn patterns. The problem, however, is that many developer productivity tools are open source, so users typically have the ability to disable usage tracking, cutting off a vital information source. Many tools launch without any tracking in the first place, later causing a riot among their community when implementing telemetry later.

When we can collect user feedback, the next question arises — what to do with it? Should we first solve the problems of our loudest, most-vocal users? Should we focus on features that will help us close revenue in the near-term, perhaps even features that were specifically asked for by prospects? These are all valid questions with no easy answer.

This is a general problem that all dev tools companies run into... building features that specific customers need so that you can close the deal, without any holistic product management framework — Manager, Developer Tools Startup

Though it's somewhat unknowable, developer productivity companies must grapple with and come to a view on the state of the product and whether it's ready for prime-time. One could always collect more data, but eventually you reach critical mass, and decisions can be made. Knowing when you've crossed this key milestone is critical. In other words, what does success look like?

Before reaching this point, it can be hard to tell whether you are on your way or marching in an entirely wrong direction product-wise:

How do you know when you know enough? How do you know when you've seen enough? Are you consensus or non-consensus? If you talk to the market and get 100 "NO"s, but 5 "YES"es, are you right about your idea or are you wrong? — Founder, Data Science Startup

How do you know if you need to change the product geometry? Also, how do you set the right KPIs? — Founder, Data Science Startup

Once product-market fit has been confidently achieved, it's time to pour fuel on the flame and scale up the team. As the team grows, engineering processes inevitably need to change to keep up. This is a key advantages of big tech organizations like Google or Amazon — they've built development processes that work at scale and have been doing so for some time. Developer productivity startups must also set themselves up for success:

Having an effective engineering organization will be a key advantage for us. Big companies like Google and Amazon don’t just have innovative products. They also have innovative engineering processes — Founder, Application Infrastructure Startup

Integrating with other technologies

Few developer productivity startups begin by building an end-to-end solution. Invariably, developer productivity products integrate with other parts of the software development toolchain.

The ML landscape is very messy; very noisy. For the different steps of the ML workflow, people will use the tools which integrate best with each other — Founder, Data Science Startup

Tools should follow UNIX philosophy of working well individually AND integrating with other tools well — Founder, Data Science Startup

It turns out, this is a different skill set from core product building, as many quickly find out:

Seamless integration to our partners technically is challenging. We’re trying to mitigate that by hiring very well-seasoned experts in each of the areas that we’re growing — Founder, Data Science Startup

Integrating with other technologies often involves hiring that particular skill set into the organization. Though it's possible to build integration into tooling that the team doesn't have personal experience with, teams with first-hand knowledge of the complementary technology build the best integrations.

Integration is as much a technological problem as a people problem:

And the human aspect...if you make this huge effort, you want to make sure it’s not just you entering their community, but them really integrating with yours — Founder, Data Science Startup

We need to go one by one in building integrations with each of these open-source communities — Founder, Data Science Startup

Communities already exist around the technologies with which developer productivity startups want to integrate. Ingratiating oneself with these communities is as important as integrating with their tooling.

We’re open source...we have to go into their databases and make sure we get buy-in from their existing developers — Founder, Data Science Startup

Greasing the wheels in this way helps build goodwill among users of the target technologies, which can open doors. The core contributors of the target technology may make an architectural change on their end that makes your integration work much easier. Mutual trust and respect enable this kind of deep collaboration.

Further, people who've spent time in these other ecosystems bring a wealth of knowledge and insights, and can help fledgling developer productivity startups avoid the mistakes of their elders:

Our biggest challenge...we want to learn from each of these communities, and make sure that we’re not repeating any mistakes — Founder, Data Science Startup

Developer productivity startups can't afford to integrate with everything. Resources are limited, and integration work is often not a core competency. Startups must prioritize different integration options. Like sizing up the market opportunity for an entire company, startups should evaluate the market potential of complementary technologies:

Postgres and MySQL….you’ll cover ~70% of the market if you serve those two — Manager, Developer Tools Startup

The proliferation of developer tooling makes it harder make such decisions:

Addressable market for applications is large, but that makes it harder to focus. We need to make sure we’re strategic in choosing which applications to prioritize — Founder, Application Infrastructure Startup

Even among the tools that teams make an affirmative decision to integrate with, quality will vary meaningfully:

We serve like 45 different tools with integration. We do a really good job at serving a few key integrations — Manager, Developer Tools Startup

Building mindshare

Mindshare always precedes dollar share. Potential users make important architectural decisions early, requiring that they know about your technology well ahead of project intiation. Further, up-and-coming vendors must navigate any risk aversion or reticence towards new tooling, which is easier said than done:

Unless developers have the foresight to future proof, they’ll still be building with Postgres or some other older database. We often hear "we'll start with MySQL and the move to you guys if we run into issues" — Executive, Application Infrastructure Startup

Methods for building mindshare are many and varied. Some are subtle. One answer that surprised us but makes complete sense in retrospect? Online classes. Online courses are a common learning vehicle for developers, and the content of most courses naturally tends to skew toward more mature technologies with larger userbases.

This is a sort of network effect whereby MongoDB developers, for example, benefit from the presence of other MongoDB developers because those same developers create an ecosystem of learning content around the database, flattening the learning curve:

An example of mindshare: online courses are teaching other databases, not ours — Executive, Application Infrastructure Startup

If the community isn't large enough, vendors themselves can help cover the gap. Tutorials and documentation help developers both learn how to use a tool but also learn of a tool in the first place. This is classic content marketing, with a developer-focused spin:

One challenge we face is improving our content. Writing tutorials, making videos, helping people use the product better — Manager, Developer Tools Startup

Mindshare has been especially difficult to build during the era of COVID-19, which has put a damper on in-person meetups and other ways in which upstart developer productivity companies build community and evangelism. Startups have rethought more formal events like conferences:

Evangelism has been stymied. The developer ecosystem is very meetup-driven, so that’s been cut — Executive, Application Security Startup

Conferences used to be really important, so we’ll see how being virtual affects that — Developer Advocate, Application Infrastructure Startup

These events often serve as meaningful lead generation channels for developer productivity companies, leading to potential pipeline impacts:

We have a strong pipeline, but we need to make sure we keep that up, especially with everything being virtual now — Developer Advocate, Application Infrastructure Startup

Mindshare, once achieved, can act as powerful social proof. Though many developers claim to be driven by first principles in their design decisions and choice of tools, like all other homo sapiens, they care about what other people think. C-suite buyers care even more about the choices of their peers, so high quality customer logos definitely matter and should be displayed prominently:

We need more logos on our website. Getting named case studies is key for us. CIOs want to know that someone else is using this thing — Executive, Application Infrastructure Startup

Scaling developer relations

Speaking of building mindshare, developer relations (DevRel), also known as developer evangelism, developer advocacy, developer experience, etc. has emerged as an incredible way for developer productivity startups to ramp up mindshare and market awareness. Developers may not like being sold to, but they don't mind nerding out on cool tech with someone who speaks the same language:

Developers don't want to hear from sales folks. They don't want to be told what to do by their CIO. They want to be part of broader community, contribute, participate — Developer Advocate, Analytics Infrastructure Startup

Developers don't like being sold to, but they trust other developers — Developer Advocate, Application Infrastructure Startup

Simple enough. But even if you're fully onboard with the idea of developer relations, doing it well is tough. To start, developer advocates are hard to come by. There just aren't many DevRel folks out there. The function is still relatively new, and career pipelines into the role have yet to fully materialize:

There is a serious lack of talent for DevRel. The ones who are good have already been hired. And the ones who could be good often have a hard time getting a foot in the door without a Twitter presence — Developer Advocate, Application Infrastructure Startup

DevRel is a hard spec to hire for. Technical folks are usually introverts. Extroverts are typically not technical — Developer Advocate, Analytics Infrastructure Startup

Knowing who to hire among available candidates is also tricky. Companies often use social media following as an indicator of DevRel potential. But nearly of the DevRel professionals we spoke to cautioned against using social media following as an rubric for hiring developer advocates, with many saying it's entirely unnecessary for the job. Further, a large social media following is only helpful if it's backed up by technical ability and credibility with the developer community:

You don't really need to have a following to be good at DevRel. It's much more important to have street cred [with developers] — Developer Advocate, Analytics Infrastructure Startup

Social media is just one part of the toolbox. Further, social following in one area does not necessarily translate to another — Developer Advocate, Analytics Infrastructure Startup

There are lots of people with big social media following who no one actually likes — Developer Advocate, Application Infrastructure Startup (author note: 🤣)

Once the DevRel team begins to scale, questions emerge around where it fits within the broader org chart and hierarchy. This quickly gets contentious, and rarely do DevRel teams get to decide their own fate. This leads to bad situations where DevRel ends up in a part of the organization that isn't truly aligned with the practice, or worse, DevRel ends up straddling multiple functions:

Developer relations often gets stuck in the middle between different functions — Developer Advocate, Developer Tools Startup

In the past people didn't really understand DevRel. This lack of understanding drove poor outcomes. Future will be DevRel as its own organization — Developer Advocate, Application Infrastructure Startup

Poor understanding of what DevRel is and its potential drives significant consternation. Again, because there are so few of these individuals and the function is so nascent, identifying enlightened executives and manager to lead these efforts is not trivial. With the right leadership, however, DevRel can thrive:

It's much better to have someone run the team who is involved in developer relations themselves. Having adamant buy in really helps — Developer Advocate, Application Infrastructure Startup

But inevitably the question emerges — how should we measure DevRel's successes (or failures for that matter)?

A lot of people don't know how to be effective in DevRel. You need analytics of some sort. You need to understand your funnel — Developer Advocate, Analytics Infrastructure Startup

This is another point of serious contention within the developer relations community. There are no standard metrics or KPIs in DevRel, leading to difficult conversations within the DevRel team and the rest of the company about whether and how DevRel is pulling its weight:

Metrics are tough. The metrics you end up measuring are often defined by the function you fall under — Developer Advocate, Developer Tools Startup

You will just be looked at as a cost center if you can't quantify this stuff. Really you are revenue generating, and you need to think of yourself that way — Developer Advocate, Analytics Infrastructure Startup

Though there may not be a perfect KPI for DevRel that works for all companies and communities, in picking a set of metrics, it helps to keep in mind a "North Star" tied to the fundamental values and purpose of the business:

Define success by how you serve other developers. Are you giving them what they need so they can build? — Developer Advocate, Application Infrastructure Startup

Conclusion

It was amazing to speak to so many developer productivity founders and operators about the major trends shaping their businesses, their strategic priorities, and their biggest challenges and pain points as they've scaled up. The discussions around challenges were especially humbling. With all the glitz and glamour around startups and Silicon Valley these days, it's easy to forget the fundamentals — building a company in any sector is really, really hard, and developer productivity is no exception.

We hope you've enjoyed this series on developer productivity. We'd love to continue the conversation. If you are a developer productivity founder or operator who's resonated with any of these findings, let's chat!

Clio Smurro & Nnamdi Iregbulem

One mechanism, payment for order flow, plays a large role in how Robinhood provides commission-free trading to its users.

Payment for order flow is often characterized as a shady practice that enables high-frequency traders (HFTs) to front-run Robinhood traders, siphoning off a sliver of profit as they do.

I tend to think such concerns are overblown. From the perspective of an individual Robinhood trader, HFTs are of almost no importance, and their impact on trading profits is imperceptible, especially if one only transacts periodically.

The truth is much more ironic.

Robinhood traders get fleeced not by HFTs front-running milliseconds before their order hits but by other retail investors, days earlier.

The party ended *five* days ago

It turns out, trading activity among Robinhood users lags the rest of the retail community by multiple days, more than enough time to squeeze out any potential profits.

A few charts demonstrate this dynamic. The graph below plots "abnormal retail trading volume" against the days before and after a stock peaks on WallStreetBets, the popular Reddit community, measured in terms of mentions. You can think of day zero as representing the day when mentions of a particular stock, say GME, peaked, with "-5" representing five days before, "5" representing five days after, and so on. "Abnormal" simply means retail trading activity relative to the prior 20-day moving average of retail trading volume for a particular stock.

Notice how retail trading volume peaks about two days before WSB mentions peak:

In other words, popularity on WallStreetBets lags the broader retail market. Stocks first get popular among non-Robinhood retail traders, and the subreddit subsequently picks this up, increasing how often the name is mentioned.

Now let's look at Robinhood volume relative to WallStreetBets mentions. Robinhood activity peaks 2-3 days after WSB activity:

All in all, that's a five day delay between when a stock begins to cool off among most retail investors and when Robinhood traders finally get the memo.

By the time most Robinhood users actually trade, the party is over. And when the festivities end, so do the profits.

Noise traders

I started the piece with my skepticism about the impact HFTs have on the profits of Robinhood traders. But if retail investors are taking advantage of market opportunities a full five days ahead of Robinhood traders, I'm much more inclined to think that could have an effect.

The data proves this out. While recent research has shown that retail traders are in fact informed (measure as a positive correlation between retail trading activity and future stock returns), Robinhood activity has no positive relationship with future returns.

The table below shows the results of a regression of future returns at 3, 5, and 20-day intervals on Robinhood user ownership and aggregate retail trading volume, along with a number of control variables. While the coefficient on retail volume is positive and statistically significant in all cases, suggesting that non-Robinhood retail trading volume predicts future returns, the coefficient on Robinhood ownership is negative and statistically insignificant, meaning that Robinhood activity has little relationship with future returns:

This finding led the authors to conclude the following:

Contrasting with recent evidence that retail traders are informed, we find that Robinhood ownership changes are unrelated with future returns, suggesting that zero-commission investors behave as noise traders.

If case you don't read a ton of academic finance research, that's an extremely polite and understated way of saying: Robinhood trading is random (with respect to financial returns at least). In other words, there's no there there.

And I should say: nothing about the app necessitates that. It's the behavior of the folks using it.

There's no such thing as free alpha

I think it's great that Robinhood exists. I love that individuals have more ways of accessing the financial markets without coughing up exorbitant fees.

But access isn't everything. We all have "access" to Las Vegas, but that doesn't make it a path to wealth creation. Likewise, if Robinhood traders use the app to merely gamble on 50/50 bets that a particular stock will go up or down, little wealth or value creation will result.

Despite the app's name, Robinhood users for the most part redistribute among themselves the breadcrumbs left by the rest of the market.

No longer. As with whales (Docker) and elephants (Hadoop, Postgres) before them, red pandas finally have the backing of a hardcore application infrastructure technology.

Meet Redpanda, a real-time event streaming platform backed by Vectorized. We at Lightspeed led the company’s recently announced Seed and Series A rounds, and we couldn’t be more excited about the potential for Vectorized to revolutionize real-time streaming.

Vectorized is one of the most technical companies I’ve ever worked with. But unlike many products, where deep technical innovation often comes at the cost of incredible complexity, Redpanda bucks this trade-off. Like its animal namesake, Redpanda is approachable, combining a simple and accessible developer experience with an underlying engine that pushes streaming to never-before-seen performance levels.

Vectorized is maniacally focused on the developer experience. As a result, it “just works”:

“RedPanda’s performance, simplicity and ease of operation dramatically improves next generation data applications. Switching our development environment from Kafka to Redpanda dramatically lowered development overhead, while pipelines feeding Clickhouse “just worked” and moved our I/O bottleneck back to the disks where it belongs. Recommended.” — Eric LaBianca, CTO, The Seventh Sense

In this post, I want to highlight three key aspects of the Redpanda developer experience — simplicity, accessibility, and performance — and discuss why we think Vectorized’s emphasis on usability will unlock real-time streaming for the great majority of developers, who are underserved by existing solutions. You’ll also hear it straight from members of the community, who I’ve quoted throughout.

Simple is beautiful

Redpanda abstracts away the complexity that often prevents the typical developer from adopting real-time streaming. There’s a long list of optimizations that I won’t entirely do justice to here, but I wanted to highlight two of the most impactful: No Zookeeper and No JVM (Java Virtual Machine).

No Zookeeper

Apache Zookeeper is a critical piece of infrastructure in Kafka and many other big data technologies. Its core purpose is to manage coordination of nodes and metadata in distributed systems, and it runs as a separate set of machines that must themselves be managed by the operator.

Zookeeper is hardened tech at this point and does its job reasonably well. However, it’s a pain to manage and requires a separate set of Zookeeper-specific expertise to deal with problems as they inevitably occur. No one asked to manage an entirely separate distributed system, yet it has historically been a hard requirement in streaming technologies like Kafka, creating additional operational overhead and burden for many Kafka users:

“Running five node zookeeper clusters was pure overhead.” — Hacker News

The issue is near and dear to me — while a product manager at Confluent, I made the case for the removal of the Zookeeper dependency, eventually resulting in the landmark KIP (Kafka Improvement Proposal)-500 which proposed replacing Zookeeper with a self-managed quorum. KIP-500 was met with applause from much of the Kafka community, who were sick and tired of dealing with Zookeeper. It was time for the Zookeeper to retire.

Removing a core dependency from a decade-old technology is not trivial. A year and a half after the publication of KIP-500, Zookeeper removal proceeds with incremental steps, necessary to mitigate potential migration issues. This can’t be done in a single update.

Redpanda takes a different approach. Thoughtfully architected from the start to leverage the open source Raft consensus algorithm, Redpanda obviates the need for a third-party consensus system like Zookeeper. This meaningfully reduces operational complexity and has a direct, positive impact on developer productivity:

“We care about reliability and performance at Zenly, so no Zookeeper and 10x faster was a no brainer.” — Jean-Baptiste Dalido, Head of Infrastructure Engineering, Zenly

No JVM

“I was avoiding Kafka for some time because of admin costs and lack of JVM expertise” — Vectorized Community Slack

Another driver of complexity in the event streaming ecosystem historically has been the Java Virtual Machine, or JVM, a hard requirement in Java-based systems. The JVM is the virtual machine that enables applications compiled to Java bytecode to run, acting as an intermediary between the source code and the system.

Unfortunately, JVM expertise is in low supply outside of the Java/Scala developers. Ironically, the JVM is often the source of issues when working with Kafka.

Developers want streaming, but they don’t necessarily want to become Java (assuming they weren’t already familiar) or distributed systems experts. There is meaningful pent-up demand among developers for a non-JVM based streaming architecture to power modern real-time applications. Sizing this up, the Python and JavaScript communities alone could be an order of magnitude greater than the existing Kafka/Java population:

Written in C++, Redpanda needs no JVM and thus users need no JVM knowledge or expertise. This finally moves streaming technology away from Java, a big deal given how many data infrastructure technologies have been built around Java or the JVM over the years:

“This looks awesome! I’ve been waiting for a long time for someone to think outside the JVM, and I really hope this is a growing trend. The “big data” industry has seemingly been joined at the hip with Java ever since Hadoop came onto the scene, and the Apache community in particular has a lot of apps that are deeply unfriendly to non-Java apps” — Hacker News

Power to the developers

Real-time event streaming technologies are not always accessible to the typical software developer. In fact, most individual developers have been left behind in the streaming revolution. Seeing this untapped opportunity, the team at Vectorized architected Redpanda in ways that expand the pool of developers that can productively operate the system.

Kafka compatibility

Too often, when new technology comes around it bifurcates the existing community in an attempt to grow adoption. This leaves individual developers and teams in the ensuing crossfire.

In building the future of real-time streaming, Redpanda respects what came before it. Despite all its improvement under the hood, Redpanda maintains full API compatibility with Kafka. This means that existing Kafka-based systems can be swapped over to Redpanda with no changes to existing applications, making for an easy and straightforward migration path.

More than a smart business move, maintaining Kafka compatibility makes Redpanda much more accessible to existing Kafka users. This is important because developers in fact love the Kafka API, though they don’t always love managing the associated infrastructure.

Further, Kafka API compatibility means that Redpanda users can continue to leverage the amazing Kafka ecosystem that has built up over the years. This can lead to interesting combinations of Kaka-related tools and Redpanda as the core streaming engine:

“Very cool — I was able to use a kafka connector to get websocket fanout out of a redpanda installation no problem. I’ll be writing a blogpost about this.” — Vectorized Community Slack

WebAssembly

WebAssembly, or WASM, is one of the most exciting up-and-coming technologies in software development today. WebAssembly lets developers write code in any major language, translate that code to the compact WASM format, and run it on the web with the high performance of a native application.

Redpanda is one of the first infrastructure technologies to take advantage of WASM, enabling developers to “write and edit code in their favorite programming language to perform one-shot transformations, like guaranteeing GDPR compliance by removing personal information or to provide filtering and simple aggregation functions.” Here’s how one community member described Redpanda’s WASM engine:

“Very clever and useful way to take advantage of WASM… It reminds me a little bit of JS-derived views in CouchDB, just way more powerful and performant thanks to WASM rather than plain JS interpreter” — Vectorized Community Slack

JavaScript, Python, Rust, Go — anything that compiles to WebAssembly (basically everything at this point) can be used to transform data. Again the key is accessibility — inline WASM transforms in Redpanda represent just that. WASM also unlocks interesting use cases beginning to emerge among the community:

“What excites me the most is the WebAssembly feature, as it enables us to create a “Data Firewall’’, the last mile of access, transforms and policy.” — Jean-Baptiste Dalido, Head of Infrastructure Engineering, Zenly

Gotta go fast

Performance isn’t often pitched as a productivity boost. But it is, especially at scale.

Better performance at the infrastructure level leaves more room for the application itself to function as intended without running into resource constraints. This means less optimization work by the developer and also opens up streaming to other languages like JavaScript and Python, whose worse performance as high-level languages is balanced out by speed at the infrastructure level.

To better understand this, let’s talk cores and tails.

Hold my Coors, I mean, cores

Hardware is moving target, continuously evolving and improving. The last 15 years have been no exception. The underlying hardware targeted by streaming and message queue systems has changed meaningfully since the advent of real-time systems, opening up new opportunities for performance enhancements that take advantage of new physical resources.

Written in lower-level C++, Redpanda’s thread-per-core architecture is optimized for modern hardware and squeezes out every last bit of performance, fully exploiting the resources it runs on. Redpanda also comes with intelligent auto-tuning out-of-the-box, which automatically generates optimal settings for your specific hardware/kernel/Redpanda setup. Organizations can do more with less, and the benefits extends down to the level of the individual developer too, who now has more “breathing room” when it comes to performance due to Redpanda’s more efficient streaming engine. Developers can worry less about optimization and just write the applications they want, enhancing developer productivity.

Don’t fail in the tail

Averages rarely tell the whole story when it comes to performance. Latency, for example, can be, on average, quite similar between two systems and yet diverge meaningfully at the 99th+ percentile. Reliable performance “in the tails” is critical for certain use cases like fraud detection, where financial institutions must return decisions ASAP at the point of sale.

Unfortunately, performance issues often “hide” in the tails of the latency distribution, only rearing their head periodically. But when they do pop up, the delays can be monstrous.

Because Vectorized built a new storage engine from scratch that removes much of the overhead in Kafka and can fully saturate the underlying device, Redpanda operates with stable tail latencies. This means architects get predictable performance in their applications and fewer unexpected spikes in latency.

This sort of reliability is difficult to achieve in JVM-based systems like Kafka. With a C++-based architecture and CPU-level optimizations, Redpanda achieves 19x better tail latency.

Developer love always wins

We love Vectorized’s focus on improving the developer experience for real-time infrastructure. Redpanda represents an improvement to the status quo along multiple dimensions. Through simplicity, accessibility, and performance enhancements, Redpanda opens up streaming to a wider audience of developers who, at the end of the day, simply want to write applications — without having to struggle with operational complexity.

Here’s how one engineer put it:

Congratulations to @emaxerrno and the @VectorizedIO team on their funding!

Their mission to make a faster and more reliable Kafka alternative is a worthy one. Our industry deserves diverse data infrastructure options, and it's exciting to see so many new ones getting traction.

— Large Data Bank (@JordanALewis) January 26, 2021

We at Lightspeed couldn’t agree more.

Want to learn more, connect with the Vectorized team, and meet other Redpanda users? Join the Slack community, check out the repo, and sign up for early access to Vectorized Cloud.

It always struck me as somewhat contrived — a simple metric with intuitive appeal, but one that is so overused and confused at this point that it harms analysis more than it helps.

My core beef with the metric is that it takes what is fundamentally a correlational relationship and confuses it with causality, to the chagrin of many strategic finance teams on the inside and investors attempting to model and forecast businesses from the outside.

While it'd be an overstep to say that sales and marketing doesn't cause revenue growth at all, I've realized most companies and investors are totally confused about the extent to which it does.

I want to set the story straight.

First, let's get definitions out of the way. "Magic number" is defined as follows:

Magic number is typically calculated on a quarterly basis, often with S&M shifted back one period, better matching and reflecting that software sales typically take anywhere from 3-6 months.

Magic number is just a ratio — it's New ARR divided by S&M expense. It tells us nothing about the degree to which that spending caused the New ARR. In fact, for reasons I will illuminate, the degree to which S&M spending actually causes new revenue is almost certainly less than is implied by this not-so-magical ratio. In other words, magic number overestimates the causal impact of S&M spending, rendering it useless for decision making.

Here's why:

• Omitted / confounding variables
• Reverse causality and simultaneity
• Observation vs. intervention

I'll walk through each in turn.

Omitted / confounding variables

Magic number is a faulty metric because it ignores important revenue growth factors and incorrectly attributes those growth contributions to "sales and marketing efficiency."

To illustrate this, I'll start with a very simple point. Refer back to the definition of magic number — New ARR divided by sales and marketing spend:

The first thing to note is that New ARR has two sources — S&M and "other stuff":

New ARR can either come from our investments in S&M or it can come from other sources. This could include organic inbound (which may or may not be "marketing" driven depending on the situation), R&D (we release a new feature or product that the market already wants, and it effectively sells itself), etc:

Open source and product lead growth rely heavily on R&D for GTM. Yet, sales metrics don't account for that.

Lately I pay more attention to gross burn vs. growth rather than sales metrics. The latter simply don't provide an accurate picture of the variable costs in the business.

— martin_casado (@martin_casado) January 31, 2021

Rather than enumerate all the possible alternative sources of revenue, let's just crudely summarize them under "Other Stuff" and modify our equation to reflect this attribution:

If we break this fraction into two parts, the first would represent "true" magic number — the true causal impact of S&M spend — while the second would represent the degree to which the typical measure of magic number is inflated by misattribution of New ARR:

In other words, magic number omits important non-S&M contributors to revenue and in doing so falsely attributes the New ARR from these alternate sources to S&M. Thus, in a perverse way, we reward ourselves for S&M "efficiency" when in fact it is other factors that meaningfully drive revenue growth. For all we know, our sales and marketing expenditures could be woefully inefficient, but we fool ourselves into thinking otherwise.

Because these other sources of revenue nearly always contribute positively to ARR growth, the magic number inflation factor is almost always positive, meaning that the standard measure for magic number is typically an inflated estimate of the true causal influence of S&M spend. This has serious implications for our ability to model and forecast revenue growth based on assumed S&M inputs.

This also explains why magic number tends to decline when S&M grows rapidly and tends to increase when S&M spend is suddenly cut. It's partly because S&M has diminishing returns, i.e. "true magic number" tends to fluctuate with S&M spend itself. But the other reason is the inflation factor, which includes S&M in the denominator. This means that when S&M grows quickly (specifically, faster than "New ARR from Other Stuff"), the inflation factor decreases, so our measured magic number decreases. The reverse happens when S&M grows more slowly — magic number inflation increases.

In statistics parlance, magic number is biased estimate of the "true" magic number due to omitted or confounding variables:

In the diagram above, Y is New ARR, X is S&M, Z is a set of observed variables that influence both S&M and New ARR, and U is a set of unobserved variables that also influence our P&L. Unobservables are, by definition, not observed and therefore difficult to account for, but at a minimum we should attempt to control for those observed variables that influence S&M and/or New ARR.

Magic number does neither, guaranteeing that it won't properly measure the impact of S&M on revenue. Magic number is therefore less a measure of sales efficiency and more a measure of its correlation with new revenue. Another name for this is spurious correlation. You've probably seen one of these funny charts before:

Two variables can be totally unrelated, yet be nearly perfectly correlated. In this case, the "magic number" between margarine consumption and Maine divorces is totally meaningless. Again, the point here isn't that S&M and New ARR aren't at all related; the point is that because they tend to move together and are plausibly related, we tend to think they are much more related than they in fact are.

Reverse causality and simultaneity

There's a tendency to think, "Well, S&M is in the denominator and revenue is in the numerator, so S&M must be driving revenue." That way of thinking also accords with the way sales and marketing is framed most of the time — they are the "revenue generating" part of the business, so investments in S&M must, to some extent, drive revenue.

But rarely asked is the reverse — to what degree does revenue cause S&M expense?

Now, that might at first seem absurd, but don't dismiss it so easily. Revenue almost certain does cause S&M. How? Well, in the most basic sense, aside from capital raised from the outside, the main source of funding for all expenses on the income statement is revenue. That's where the business earns the cash that it can then spend on various initiatives.

If you still don't believe me, think about this proof of concept. Does COGS (Cost of Goods Sold) cause revenue, or does revenue cause COGS? Clearly revenue causes COGS, by definition, since COGS is the cost of goods sold, so something has to be sold (and revenue has to be generated), in order for COGS expense to show up. So clearly, revenue can cause expenses to some degree.

What we choose to spend on S&M is dependent on the revenue we expect to earn. For example, if you know your S&M is highly effective at driving revenue growth, that is likely to encourage you to invest more in S&M. Conversely, if S&M is not very effective at driving revenue, you will likely spend less than you might otherwise. In that subtle way, revenue (or more accurately in this case, expected revenue) influences S&M spend.

You may not have ever considered this, but it shouldn't be too surprising. There's a reason you rarely see companies spending, say, 500% of revenue on S&M. Companies typically limit their S&M spend to some reasonable level relative to current revenue. Therefore, revenue in some way influences S&M spend. Economists would say that S&M is endogenously determined — it's not some number we pull out of thin air. Our expectations for how valuable it will be and how much revenue we have in the first place determine how much we invest in S&M. Further, our business plans and objectives ("hitting XM in ARR by year end") influences our S&M spending decisions.

This phenomenon — where A causes B and B also causes A — is called simultaneity in statistics and econometrics. More colloquially, you might know this as reverse causation.

Reverse causation poses a serious problem to our attempts to forecast new revenue from sales and marketing expenditure. Because magic number estimates the relationship between S&M and revenue from observational data rather than some kind of controlled experiment, we don't know the degree to which one causes the other or the reverse. If a business has a magic number of 1, that could mean that 1 dollar of S&M causes 1 dollar of New ARR, or it could mean that businesses that expect to add 1 dollar in New ARR choose to limit their S&M to roughly 1 dollar.

The answer is somewhere in the middle, and yet magic number assumes the former. Not good.

Observation vs. intervention

The third and final problem with magic number is that is confuses observation and intervention. This is very related to the notion of correlation vs. causation but goes further by personifying causation a bit.

Observation means watching something happen, making no attempts to influence the outcome or the relevant variables. As an analyst measuring the sales efficiency of a software business from the outside, magic number is an observational metric. You can't do anything to influence any of the inputs, you are merely reporting their values. Magic number is not a good measure of sales efficiency, but as long as you acknowledge this caveat, there's nothing inherently wrong with calculating its value, useless though it may be.

However, if you are an actor inside the business intending to intervene in some way, choosing how much to spend on S&M, the entire game changes. In this case, magic number is not only useless as a measure of sales efficiency, but it is in fact a logical fallacy to use it to forecast the revenue impact from assumed S&M spend.

The reasons behind this are somewhat subtle, but this analogy should make it quite clear.

Imagine you have a barometer measuring the air pressure around you. If the barometer is functioning properly, we should observe a tight relationship between the pressure reading on the barometer and the actual air pressure. In other words, air pressure and our barometer reading are highly correlated.

However, if we were to hack the barometer and manually change the measurement it reports, would we expect the pressure in the room to change as a result? Obviously not. Intervening by changing the pressure reading of the barometer does not impact actual air pressure, despite the 1:1 observational correlation between the two. The interventional or causal relationship between the barometer and air pressure is exactly zero in that direction (though there is a very strong causal influence in the other direction, air pressure to barometer reading). The observational relationship breaks down when we interfere with the system.

"In summary, y and x are correlated or statistically dependent and therefore seeing x allows me to predict the value of y, but y is not caused by x so setting the value of x won't affect the distribution of y. Hence, p(y|x) and p(y|do(x)) behave very differently." — ML beyond Curve Fitting

Economists know this as the Lucas Critique, which tells us that attempting to conduct economic policy based on some macroeconomic model trained on observational data will inevitably fail, as the identified relationships disappear into the ether.

The same logic applies to sales and marketing spending. We should not expect magic number, which is effectively a measure of the correlation between New ARR and S&M, to hold constant if we were to purposefully increase S&M. New ARR and S&M can be tightly correlated, and yet the observational correlation between the two breaks down when we intervene. Choosing to, say, double S&M next year does not in any way guarantee that we will book twice as much New ARR. Worse, the breakdown in the relationship tends to go against us — after a significant ramp in expense, sales efficiency almost always ends up being worse than expected, rather than better.

If we are interested in forecasting new revenue based on an intentional choice of S&M spending, magic number won't do. We need to find a better magic number, one that approximates the true causal impact:

"In applications where you ultimately want to control or choose x based on the conditional you estimated, you should seek to estimate p(y|do(x)) instead. For example, if x is a medical treatment and y is the outcome, you are not merely interested in observing a naturally occurring treatment x and predicting the outcome, we want to proactively choose the treatment x given our understanding of how it affects the outcome y." — ML beyond Curve Fitting

Magic multipliers

If you're a regular reader of mine, you'll know that this is the part where we transition from theory to data. Can we identify the true causal impact of S&M on revenue growth?

We can try.

Funnily enough, economists regularly tackle a very similar issue — calculating how much the economy grows when the government spends more money, otherwise known as the "fiscal multiplier." How exactly to calculate these multipliers is a constant debate among economists, but a few methods have emerged with some consensus. Here I will leverage a statistical method called "local projections" which I won't explain in detail here, but I will summarize by saying it enables us to calculate the causal impact of an increase in S&M spend on New ARR over time via a special linear regression. The ratio of the increase in New ARR to the increase in S&M spend yields the "multiplier", or true magic number in our case.

I've assembled a dataset of anonymized quarterly New ARR and sales and marketing spend for a sample of early to mid-stage software companies. This is far from a representative sample of all software vendors, but with 230 data points across 26 companies we can generate insights nonetheless. As evidence of its non-representativeness, the average magic number in my dataset is 2.1. Most companies in fact hover closer to 1, but the logic is the same regardless.

First we'll calculate the impact of an increase in S&M (relative to what it otherwise would have been) on New ARR (again, relative to the counterfactual). We'll recalculate this at various horizons — current quarter (labeled "Q1" below), one quarter out, two quarters out etc (chart on the left). To get the total effect we'll sum up the impacts across quarters (chart on the right):

A 1M increase in S&M this quarter generates 0.38 more New ARR that same quarter, 0.39 the next quarter, and so on. Cumulatively, over four quarters this adds up to about 2 of additional New ARR vs. if S&M spend hadn't increased.

Now if this was the entire story we'd say that the true magic number reaches two after four quarters and perhaps continues to increase after that. OK, so perhaps magic number as typically calculated doesn't measure the immediate impact of S&M spending, but it gets there eventually, right?

But it's not the whole story. We've cheated a bit here by only counting the initial 1 dollar bump in S&M spend. But if we increase spending today, spending tomorrow will also be higher. Part of the effect on New ARR we measured above is due to higher spend in the current quarter, but part of it is also driven by higher spend next quarter (and the quarter after that, and so on).

To account for this, we must calculate the effect of S&M this quarter on S&M next quarter AND the quarters thereafter and include the cumulative effect in the denominator of our magic number calculation:

The chart above shows the impact of S&M on itself into the future. A 1 dollar increase this quarter leads to the next quarter being 0.70 higher than it would have otherwise been, 0.86 the quarter after that, and so on. Cumulatively, we spend about 3.5M more after four quarters.

Now we can compare the cumulative effect on New ARR to the cumulative increase in S&M to get a proper measure of the "true" magic number:

The true magic number is estimated to be only 0.6 after four quarters, this is less than a third of the average magic number in this dataset of 2.1. So not only is the true magic number much lower than the typical calculation, it takes many quarters to reach this much lower cumulative impact. Importantly — we don't get nearly the bang-for-buck that the traditional magic number implies.

To make this real — a forecast that New ARR will be 2.1x higher next quarter based on a 1 dollar increase in S&M this quarter will be off by 80% (per the first chart, New ARR one quarter out is only 0.39 higher) and cumulatively (including the first and second quarter impact) will be off by 78% (based on 0.46 cumulative magic number at Q2).

Magic isn't real

It's worth pointing out some caveats here.

I wasn't able to fully control for the "other stuff" that also drives New ARR. In my regressions I do control for the long-term trends of each variable (via a cubic polynomial time trend interacted with company fixed effects for those who care) which helps account for these unobserved factors, but this is not a perfect solution. I also don't identify a completely exogenous increase in S&M. Inherently, I assume that S&M deviations from trend imply unexpected increases in S&M, but that's also not totally right.

This is amateur econometrics, so it'll do.

My conclusions:

• Magic number is a bad metric. It misleads us into thinking that by turning some knobs we can precisely control revenue growth. It commits the scandalous statistical sin of mixing correlation and causation

• S&M alone has little effect on New ARR. Certainly not in the near term and to a much lesser degree than you might think in the longer run

• We don't really know how to calculate software sales efficiency. This analysis leveraged 230 data points to get some level of certainty, but for any particular company it'd be difficult to replicate with only a few quarters or years of data

So what can we use as an alternative?

I've always liked the easy heuristic of New ARR vs. cash burned. This ignores specific expense line items and gives companies credit for "all-in" capital efficiency, which is arguably what matters most anyway. It avoids the perils of attributing revenue growth to particular initiatives, like sales and marketing or research and development. Of course, we still have an omitted variable problem (burning cash is not the only way to drive revenue growth), but it's much less severe.

3,000 words later, I hope I've convinced you that magic number doesn't tell you much about sales efficiency or productivity. If you've found a better metric, I'd love to hear about it. Feel free to shoot me a note at first.last@gmail.com

This phenomenon is called the Lindy effect.

To be "Lindy" means the longer something survives, the more time it has left. Remaining life extends, rather than contracts, with age.

Perishable objects like flesh-and-blood humans don't work this way. As we age, our remaining time on this Earth decreases — a 90 year-old has less expected time left on the clock than an 80 year-old.

But certain non-perishables follow a different rule. Per Wikipedia:

The Lindy effect is a theory that the future life expectancy of some non-perishable things like a technology or an idea is proportional to their current age, so that every additional period of survival implies a longer remaining life expectancy. Where the Lindy effect applies, mortality rate decreases with time.

Product-market fit follows the Lindy effect.

More precisely, lack of product-market fit is "Lindy". The longer you don't have it, the longer you won't have it.

An additional year of "no product-market fit" implies a longer remaining period of "no product-market fit." The odds of achieving product-market fit with any particular idea decline with time. Thus, if you don't achieve product-market fit quickly, you may never achieve it at all.

Product-market fit, like the elusive "cure" for cancer, is not a fixed destination, guaranteed to be reached with enough time spent running toward it. In a weird way, moving "toward" it doesn't actually get you any closer to it — it only moves further away.

Product-market fit escapes from you.

Product-market fit isn't normal

I was inspired to write this essay by a recent tweet by Matt MacInnis, COO of Rippling, where he claimed product-market fit is obvious once you have it, suggesting business performance is lognormally distributed vis-a-vis product-market fit:

(1) It's been said, but now I get it in my gut: if you have to ask whether you’ve found product-market fit, you haven’t. Like most things, it’s lognormal: better PMF yields way, way better biz performance. At a certain point, most assumptions about how to build a co. break. 2/7

— Matt MacInnis (@stanine) August 4, 2020

No, he didn't mention the Lindy effect directly, but that's what instantly came to mind as I read the tweet.

Here's why.

A lognormal distribution is one which, if you take the logarithm of all its possible values, yields a normal distribution. Hence "log" "normal". Due to the exponential nature of the logarithm, lognormal distributions are skewed:

Lognormal distributions are neither normally distributed nor thin-tailed. Instead, they are fat-tailed.

This is an important fact.

When MacInnis says performance is lognormal, or skewed, with respect to product-market fit, he's making the point that, once you have product-market fit, you go straight to the tails. Business performance accelerates so meaningfully that any ambiguity goes out the window. You get it "in your gut." It becomes obvious, as nearly everything about the business gets better.

A/B testing with fat tails

In Why Don't VCs Index Invest, I make the point that, when facing sufficiently fat-tailed distributions, it makes sense to spread your bets widely, avoiding concentration or dependence on any one investment or opportunity. Instead "index invest," placing small bets in everything reasonable without overthinking it too much.

This extends to all domains where fat-tails reign. A fascinating paper by researchers at Microsoft, HomeAway, Wharton, and Columbia University applies this logic to the domain of internet search engines and reaches similar conclusions.

Search platforms like Microsoft Bing and Google face a trade-off when A/B testing various changes:

• run large, data-hungry experiments to get precise effect estimates for potential improvements to the platform
• run small, "lean" experiments that do not have much statistical power but nevertheless detect effects if they are large enough

The conventional wisdom is that larger A/B tests are better because they increase statistical significance of the experimental results, building confidence that seemingly positive results are, in fact, positive.

It turns out, the optimal choice for how many users to assign to an experiment depends on the fat-tailedness of the distribution of potential returns, or what the researchers call, the "innovation value distribution":

... with sufficiently fat tails, this conventional wisdom reverses and the go lean approach of trying many small experiments is preferred. Intuitively, with sufficiently fat tails, even small experiments are sufficient to detect the largest effects; which in this case account for most total value. Larger experiments detect subtler effects, but these constitute less of the total value; making the value of information concave. This case also has different implications for the marginal value of information.

To translate — when most of the potential gains come from a small number of experimental changes (i.e., innovation is fat-tailed), you're better off running as many experiments as possible so as not to miss the potential "big one." Yes, this might come at the cost of statistical precision, but large effects are so large they can be detected even in small samples.

Another way to think about this is in terms of the "value" of information:

while the production function is always concave for large numbers of assigned users, its shape with few users depends critically on the thickness of the tails of the prior. If the prior is not too fat-tailed... then the production function is convex. However, we show that if the prior is very fat-tailed... the production function is concave.

Again, I translate — when the value of experimentation is fat-tailed, the marginal value of additional information declines as you gather more data because any large, positive innovation is easily detectable with only a small amount of data. Thus, collecting more information doesn't help much — the value of information is concave.

On the other hand, if the value of experimentation is normally distributed or thin-tailed, then there are almost no mammoths out there to find. Instead, most innovations are of middling value. In that range, precision matters, as it can be the difference between accidentally implementing a slightly worse change over a somewhat better one. More information helps decipher between the two. Thus, the value of information is convex.

We can see this in the chart below from the Bing paper, which shows the shape of the experiment "production function" (which effectively represents the value of experimentally gained information) when graphed against experiment sample size. The different lines represent various levels of , or the assumed fat-tailedness of the innovation distribution. Lower means fatter tails, and vice versa.

Fat-tailedness (small ) leads to a concave information value curve, whereas thin-tailedness (high ) leads to a convex information value curve:

You'll know it when you see it

So which is it?

We present evidence—using a sample of approximately 1,505 experiments—suggesting that innovations at Microsoft Bing have very fat tails.

The innovation value distribution is fat-tailed. So small experiments can detect the largest effects, which represent most of the value of A/B testing:

If the distribution of innovation quality is sufficiently thick tailed, a few ideas are large outliers, with very large negative or positive impacts. These are commonly referred to as black swans, or as big wins when they are positive. The production function is concave and has an infinite derivative at n = 0. The optimal innovation strategy in this case is to run many small experiments, and to test all ideas.

OK, but what does A/B testing with fat tails have to do with product-market fit?

Think about the search for product-market fit as a sort of multifaceted A/B test. You try various permutations of the product, business, and go-to-market model, looking for something that especially resonates with users or customers. The vast majority of these combinations won't work out, but a handful will. Those few will work so well in fact that they will dwarf the performance of all the other iterations.

If it is true that product-market fit yields much better business performance, then product-market fit should be relatively obvious, even with little data. This suggests that, if you do have some data for an idea and it still isn't obvious to you whether or not it achieves product-market fit, you may have a problem, especially if you've been at it for a while.

But isn't that exactly what MacInnis said?

"If you have to ask whether you've found product-market fit, you haven't."

The Bing experiments provide the mathematical and empirical justification for MacInnis' bold statement. While I won't quibble over whether business performance is lognormal or some other distribution, it's clearly quite skewed, just as in the Big experiments. In that kind of world, product-market fit loses much of its mystique — it doesn't take a genius to know whether you have it:

Consider a startup firm that uses a lean experimentation strategy. The firm tries out many ideas in small A/B tests, in hopes of finding one idea that is a big positive outlier. Even though the A/B tests are imprecise, the firm knows that, if a signal is several standard errors above the mean, it is likely to be an outlier. So the firm decides to only implement ideas that are, say, 5 standard errors above the mean. This means that the firm will almost certainly detect all outliers that are more than, say, 7 standard errors above the mean.

"You'll know it when you see it" rings quite true.

Due to fat-tailed business performance, it doesn't take much data to know you've achieved product-market fit, nor does it require some complicated, made-up framework from a supposed startup guru. In the context of the Bing analysis, any A/B test that yielded a result more than several standard deviations above the baseline essentially achieved "fit," with a high degree of certainty.

Starting over

Let's end by returning to where we began — the Lindy effect.

MacInnis' statement is effectively a rephrasing of the Lindy effect —  if it's taken long to find product-market fit, it's likely going to take a lot longer.

To rephrase once more: since it doesn't take much data to know you have product-market fit (assuming you do, in fact, have it), then if you haven't achieved it yet, you probably have a long way to go.

This is brutal stuff. However, via lean experimentation, planting many flags in various places and quickly shifting gears when no treasure is found, you can, in expectation, start "closer" to the goal. Per the Microsoft paper:

We call this the "lean experimentation" strategy, as it involves running many cheap experiments in the hopes of finding big wins (or avoiding a negative outlier). This strategy is in line with the lean startup approach, which encourages companies to quickly iterate through many ideas, experiment, and pivot from ideas that are not resounding successes.

I want to emphasize that these are are all probabilistic statements and say nothing about any particular circumstance.

You and your team could have been chasing product-market fit for a given idea for the past year, and, for all anyone knows, it could be just around the corner. Many ideas seemed hopeless and for the longest time didn't work... until they finally did.

But that's not the norm.

Uber, Netflix, Facebook, DoorDash. In each case, soon after launch, it became quite clear they were doing something right. Even in companies that went through many iterations, things started to work very soon after landing on the right idea.

To be clear — the chase does not cause product-market fit to never be achieved. It represents growing evidence it won't be achieved. The chase represents information, the value of which is concave, meaning it has diminishing returns. That one needs to chase at all is, strangely enough, a sign you're on the wrong path.

More viscerally, if the trail through the forest seems much longer than it should be, you're probably lost. Breath, regroup, and try something different.

Developers are dropping out of the profession in large numbers despite efforts to grow the number of computer science graduates and software engineers.

Here's why.

Developer dropout is real

Software development has a serious retention problem:

• At age 26, 59% of engineering and computer science grads work in occupations related to their field of study. By age 50, only 41% work in the same domain, meaning a full ~30% drop out of the field by mid-career
• In contrast, engineering and computer science majors who join unrelated fields upon graduation retain at much higher rates, with only 10-15% switching out after the age of 26:

Engineers often leave engineering for non-STEM management roles. Graduation into management is not surprising. What's surprising is that these are non-STEM positions. Engineers swap technical roles for non-technical roles over time.

This phenomenon, which I'll call "developer dropout," is a real problem. What's behind it?

Out with the old skills, in with the new skills

Programming-related jobs have high rates of skill turnover. Over time, the types of skills required by companies hiring software developers change more rapidly than any other profession.

To demonstrate this, researchers analyzed job postings on more than 40,000 online job boards and company websites between 2007 and 2019, controlling for employer, location, and occupation. They defined "new" skills as those that were rare or non-existent in 2007 but prevalent in 2019 and "old" skills as those that were prevalent in 2007 but rare or extinct in 2019.

• While only 30% of all job vacancies required at least one new skill by 2019, 47% of computer and mathematical jobs required at least one new skill (i.e. a skill that was not common back in 2007)
• This compares to less than 20% of jobs in fields like education, law, and community and social services
• In addition, 16% of jobs in computer and mathematical fields in 2007 required a skill that was obsolete by 2019 (i.e. a skill that was common in 2007 but relatively rare in 2019), more than double any other job category:

About a third of the change in required skills in computer-related occupations is due to specific new software:

• The fastest-growing software skills between 2007 and 2019 include Python, R, and Apache Hadoop
• Software that was popular in 2007 but effectively obsolete by 2019 includes QuarkXpress, ActionScript, Solaris, IBM Websphere, and Adobe Flash (ah, finally a name I recognize)

Data science, machine learning, and AI saw big increases among technology-intensive jobs as well. For example, the number of STEM-related jobs requiring skills in machine learning and AI grew more than 4x from 2007-2017, touching more than 15% of STEM jobs:

To better compare rates of skill change across occupations, the researchers came up with a measure of skill change that tracks the absolute growth or decline of various skills within each profession from 2007 to 2019. Occupations whose required skills change rapidly in prevalence among job postings receive a high score, while jobs whose skills do not change much receive a lower score:

• Computer-related occupations receive the highest score by far, 4.8. Note that the mean and standard deviation of this measure are ~3 and ~1 respectively, so computer-related jobs are nearly two standard deviations away from the typical job in America
• Meanwhile, jobs in education and and those involving manual labor have very low skill change scores, typically less than 2.

We can get even more granular and look at specific job roles. This level of detail makes the difference even more stark (only showing the fastest changing roles):

Web development has the highest rate of skill change among all jobs in the country. Next up are sales engineers, another often technical role. Database administrators, computer network architects, sysadmins, and application developers all make the top 10, and we see many other technical roles among the top 30. The mean and standard deviation are similar here, placing web development more than 3 standard deviations away from the typical job in America in terms of skill change over time.

Suffice to say, software development is a rapidly changing profession.

You might think, however, that skill change would eventually settle down as one becomes more experienced.

You'd be wrong. The skills for software engineering jobs change rapidly throughout the entire career lifecycle:

• In entry-level roles in computer and engineering occupations all the way through those requiring 12+ years of experience, the proportion of job postings requiring at least one new skill in 2019 was effectively the same, 40-45%
• In contrast, 29% of entry-level non-computing and engineering roles in 2019 required at least one new skill, but this proportion declines to 24% for jobs requiring more than four years of experience

This means that experienced STEM workers seeking employment in 2019 are often required to possess skills that were not required when they entered the labor market in 2007 or earlier.

Software engineers never escape the skill-change vortex, even many years into their careers. Experienced engineers must learn and adopt technologies that didn't even exist when they started out. Developers must constantly retool themselves, even well after their formal education ends.

Nothing's changed but my change

College majors associated with faster changing jobs pay more early on.

• In professions with one standard deviation increased skill change, pay is ~30% higher in the first few years of one's career
• If we exclude both the fastest and slowest-changing fields (Engineering/Computer Science at the high end, Health/Education at the low end), the early earnings premium for faster-changing roles increases to ~60%:

Fast-changing fields pay better.

Notice however that the pay advantage declines over time. By the time one approaches the age of 50, the pay premium for working in rapidly changing fields falls dramatically to only 20-30% vs slower changing professions.

Here's another way to see the eroding pay advantage. The below chart simulates the earnings of the average worker by category of college degree from ages 23 to 50 in 2016 dollars.

• Computer science and engineering grads start off with sizable advantage vs any other major
• However, this premium falls over time as the earnings of CS and engineering graduates plateau over time while the earnings of their peers grow faster for longer
• In fact, life and physical science graduates' earnings surpass their computer and engineering classmates by the age of 40:

Excluding business majors, the earnings premium of software engineering declines over time in both percentage and absolute dollar terms, to the point where engineers barely out-earn social science majors:

But the focus on college major is somewhat misleading. This phenomenon has less to do with one's field of study and more to do with choice of occupation.

To show this, researchers plotted the earnings premium of various categories workers relative to those with a non-Engineering/Computer Science major working in a non-Engineering/Computer Science job.

• Workers who major in Engineering or Computer Science but work in unrelated fields actually see their earnings advantage compound over time, rather than decline
• On the other hand, regardless of major, individuals who work in Engineering or Computer Science jobs see their earnings advantage erode over the years:

Declining relative returns is a feature of STEM jobs, not majors. The earnings premium for non-STEM majors in STEM occupations starts off near 40%, but declines to 20% within a decade. In contrast, the relative earnings advantage grows over time for computer science and engineering majors working in non-STEM occupations.

The profession of software development drives the declining earnings premium, not the college major.

In fact, computer science majors who work in non-CS fields experience the opposite dynamic of their non-developer peers — their relative earnings premium rises as they advance. A CS major who eschews the profession doesn't earn much more than otherwise similar non-CS majors early on, but eventually out-earns their peers by nearly 20%.

OK, that's enough about what is happening. Now let's see why it's happening.

Human capital depreciates too

Imagine a simple model where workers choose their profession in order to maximize income, which is a derivative of their own skill or human capital. Over time, workers gain new skills, while the value of their existing skills depreciates somewhat due to changing times.

Some workers, endowed with superior ability, learn faster than others, picking up skills at a quicker pace. Those workers will tend to sort into high-skilled, fast-changing professions initially, maximizing their early career earnings. Less impressive workers will sort into low-skilled, slower-changing professions.

In a world where human capital never depreciated, we could imagine that high-skilled individuals like software developers would maintain a relative human capital (and earnings) advantage over other professionals, leading to consistently increasing pay and a stable relative premium:

But, if human capital depreciates over time and that rate of depreciation is higher in rapidly-changing fields like software development, then developers' initial advantage would erode over time, narrowing the gap vs. non-developers:

This simple model helps explain what we see in the data — the software engineering earnings advantage disappears as the effective human capital gap narrows.

Applied majors such as computer science, engineering, and business teach vintage-specific skills that become less valuable as new skills are introduced to the workplace over time.

Specific skills in software development quickly become dated. Programming languages and development frameworks go out of style. Hadoop is hot one year, and it's old news the next. Like a fast, expensive car that quickly loses value as it's driven around town, the skills and human capital of software engineers fall apart without constant, expensive maintenance:

Intuitively, careers with high rates of obsolescence require workers to learn many new tasks each year, which diminishes learning gains and lowers the returns to experience.

Quick learners are fast dropouts

The hits don't stop there. Ironically, the quickest learners are also the quickest dropouts.

To understand why, think back to the model we just outlined. Quick learners accumulate human capital faster than their slower peers, which means they have the most to lose when certain skills or abilities fall out of favor. In fact, the return to being a fast learner is higher in jobs with low rates of skill change because the learnings can compound over time instead of dwindle in relevance.

High-ability workers are faster learners, in all jobs. However, the relative return to ability is higher in careers that change less, because learning gains accumulate.

Said another way, the opportunity cost of working in a rapidly-changing field is highest for the best learners. This creates immense pressure to drop out of software engineering and other fast-changing careers into more stable roles and industries.

The researchers show this by regressing STEM job status on a number of other variables, including an interaction between age and score on the Armed Forces Qualifying Test (AFQT), a common measure of cognitive ability. The coefficient comes out negative and statistically significant, implying that the relative probability of working in STEM at any given age declines with cognitive ability (column 1 and 2 below):

The results imply that a worker with cognitive ability one standard deviation above average is 4.9 percentage points more likely to work in STEM at age 23, but only 1.6 percentage points more likely to be working in a STEM job by age 34.

In fact, by age 40, the regression predicts workers with higher cognitive ability are less likely to work in STEM than those with lower cognitive ability.

Simply put, professionals with higher cognitive ability drop out of STEM careers earlier and faster:

... STEM majors with higher scores on the Armed Forces Qualifying Test (AFQT)—a widely used proxy for academic aptitude—leave STEM careers more often and at younger ages.

Implication: Growing the software engineering talent pool is harder than you think

Two points I want to drive home:

First — software development has a turnover problem.

Growing the supply of software developers is not trivial because the field already sees high levels of developer dropout and turnover, and this would only increase if the field were to grow larger. A larger software development labor pool would presumably drive down wages, encouraging even more developers to shift out of the profession, especially those past their career midpoints. On that flat part of the earnings curve, the incentive to remain a developer is weak at best.

Second — software development also has a selection problem.

The highest ability, fastest learners disproportionately leave the field over time. They have a multitude of other ways to profitably leverage their intellect and skills. Software development carries serious opportunity cost. Again, this is ironic because one would normally expect the best to stay and worst to leave, but that's not what we see in the data. The software development talent pool mix shifts toward lower cognitive ability as any given cohort matures.

Combined, these points suggest software development may be destined for perennial labor shortages unless the pace of change slows sufficiently.

... rapid technological change can lead to a short shelf life for technical skills. This tradeoff between technology-specific and general skills is an important consideration for policymakers and colleges seeking to educate the workers of today, while also building the skills of the next generation.

To conclude, I emphasize: highly skilled people prefer highly stable careers in the long-run. This lets their relative ability and human capital advantage compound over time. Rapid deterioration of skills continuously levels the playing field, preventing the best from separating themselves from the pack. In such a situation, it makes more sense to quit the race early than get caught in the pile up.

Thanks to David Deming, Kadeem Noray, the authors of the study from which much of this essay is derived.

Paying too much attention to the "average" customer leads many founders and investors astray.

Instead, focus on the tails.

Here's why.

The basics

Define "monetization" as the average revenue a software vendor extracts from its customers:

Wild variation in monetization across customers means most enterprise software customers contribute little to overall revenue.

For example, a late-stage software company with 50M+ in ARR and ~2,000 customers might have a small number of customers with enormous contracts, possibly greater than 500K or even 1M in ARR each, and a large number with tiny contracts in the ~10K range.

The extreme, non-negative variation around the average produces a right or positively skewed monetization distribution:

Sufficiently skewed distributions are "fat-tailed." Large customer contracts will determine the properties of the distribution, like its mean or variance. Similarly, large customers will account for an enormous proportion of overall revenue. In other words, software monetization is a power law.

In a previous essay, I introduced the notion of , or alpha, the shape or tail parameter, which characterizes the "fat-tailedness" of a power law distribution. The smaller , the more skewed the distribution, the fatter the tails, with indicating extreme skew and fat-tailedness. For software monetization, the fatter the tail, the more common and impactful are those "whale" customers.

Evidence for skewed monetization is tough to come by without access to a company's commercial contracts. However, we can infer the fat-tailedness from the SEC filings of public software companies with a simple trick.

The math

The trick? A formula exists that calculates the concentration in the top percentiles of a power law distribution based on the of the distribution:

where is the share of the total and is the percentile.

Plug in the and the percentile you are interested in to get the share of the total that the top-p% of customers represent.

Invert the formula to yield the of a power law distribution given a certain percentile and share:

This means we can infer ​ and therefore how fat-tailed the revenue distribution is if we know the share of revenue represented by the largest customers of a given software vendor.

We can estimate the shape of the customer distribution by plugging back into the first equation along with some other percentile X in order to estimate the share of revenue earned from the top-X% of customers, which we can repeat for other percentiles. For example, the relationship between and the revenue concentrated in the top-20% of customer looks like this:

Source: David Salazar

An yields the classic Pareto 80/20 distribution, where 20% of customers account for 80% of revenue.

Before we proceed, know that this method only works if we assume upfront that the distribution is in fact power law distributed, at least in the tails. We never proved this, so don't interpolate/extrapolate too far with this method.

With that caveat acknowledged, let's throw caution to the wind!

The evidence

Public companies do their best to hide customer concentration. But at a certain point fiduciary duty requires that they disclose revenue concentration, especially if a few customers account for a large enough chunk of revenue.

Commonly, companies will state that "no customer represents more than X% of revenue." Less frequently, companies will go further, disclosing the number of customers that exceed some revenue threshold, typically 100,000, and the proportion of revenue they represent. This gets framed as a point of strength — "look how many large customers we have" — but it also indicates customer concentration since the X customers with greater than 100K contracts are by definition the X largest customers.

Here's an example from Slack's S-1:

Source: Slack's S-1

From this sort of disclosure we can calculate the and fat-tailedness of customer monetization for public software vendors via the procedure outlined above, plugging in the share of total revenue and total customers represented by these large customers. I've done the hard work for you for a subset of public software companies:

Source

The alphas are universally below 2, implying a high level of skew.

A more visual representation of the alphas (with the average in black):

The implied top-20% and top-1% revenue concentration are quite large for most companies (blue = top 20%, red = top 1%):

So, the top 20% typically represent ~70% of revenue, while the top 1% represent ~40%. Not quite Pareto 80/20, but pretty close! Interestingly, many companies tied to infrastructure in some way like Datadog, Fastly, and Twilio do have 80/20 monetization distributions, at least in some years.

As I caveated, these figures will be off to some degree. I'd guess they overestimate revenue concentration somewhat.

However, I'm comforted by corroboration from a report by Theta Equity Partners that uses a completely different methodology to estimate the monetization distribution of Slack's customer base. Based on the S-1 filing, they found that Slack's top 1% of customers account for 40% of total revenue in 2019:

As we mentioned previously, the data and the model imply a high level of variability in the “goodness” of Slack’s customers – a small (<1%) segment of “heavy” customers accounts for 40% of company’s revenues and generates revenue per customer which is more than 100 times larger than everyone else.

Using my methodology, I find that the top 1% of Slack's customers in 2019 represented 43% of revenue, which is quite similar.

The implications

It's easy to see these results and think "Yes, sure, customer concentration is a thing. So what?" But the implications of a fat-tailed monetization distribution are profound.

Why don't software companies index invest?

First, as I discussed previously in Why Don't VCs Index Invest, when facing a sufficiently fat-tailed distribution of returns, it doesn't make sense to be picky or overly concentrated in one's investments. Index investing is the optimal allocation strategy.

Here, returns are synonymous with revenue and investment is synonymous with customer acquisition costs, or CAC. If the distribution of revenue is fat-tailed, vendors should be trying to insert their software into as many customers as possible, as cheaply as possible. Don't try to land large upfront, as this requires investing in a heavy and expensive enterprise sales motion that may not yield results. Instead, spend small and land small, with each customer contract acting as a potential "lottery ticket" that may unlock a much larger contract later on, similar to an early-stage startup investment.

There's some merit to this analogy between venture capital and software go-to-market strategies. One only has to look at estimates of for venture capital investments to see that we are dealing with similar phenomena here (ignore the orange footnotes):

Source: Reaction Wheel

Whale hunting

Second, as I allude to in the aforementioned essay, finite samples of a positively skewed, fat-tailed distribution tend to underestimate the average, or mean, of the distribution. Large values are rare, so small samples will tend to miss them. Unless you have an extremely large dataset, the "true" mean is typically larger than the mean you measure from the data. So the calculated sample mean tends to increase as the sample size grows, reflecting those large, infrequent outcomes.

Said more precisely:

An additional difficulty in the numerical estimation of moments—and, therefore, of risk—is due to the very slow convergence of estimated values to the exact values of the process, even if the associated moments are finite. This “slow Law of Large Numbers” is caused by the large weight of rare events (black swans), which take a lot of data to show up, and prevent a proper estimation of the moments of such processes through the moments of a sample. — Fat tails and black swans: Exact results for multiplicative processes with resets

Further, the largest value you are likely to see in a sample of power law distribution (the expected value of the maximum value) is proportional to the sample size and inversely proportional to :

In plain English — your personal best can only get better with more attempts. In the realm of venture, that looks like this:

Source: Reaction Wheel

This is why returns in venture capital tend to increase with portfolio size:

Source: AngelList

In the context of software monetization, the "true distribution" is the set of all potential customers while the "sample" is the set of current customers. Each customer is like a draw from a random variable representing all potential customers, just like a venture capital investment is like a random draw from a fat-tailed distribution of potential returns.

My claim is that the average monetization across your customer base — ARR/customer, revenue/customer, etc. — is an underestimate of the "true" or "potential" monetization. As you land more customers, so the logic goes, the revenue you extract will tend to rise due to this fat-tailed phenomenon, with no change in pricing model or customer targeting.

More is more, or specifically, more customers is more monetization, for the same reason that larger venture portfolios yield higher returns. Your wins get bigger the more broadly you penetrate the market.

It's natural to ask — "how much does current monetization underestimate potential monetization?"

Nassim Taleb has already done the math for us. Imagine that true average monetization is the sum of the monetization of customers smaller than the largest customer we've acquired thus far, which he calls , and the contribution of potential customers larger than our largest:

In the chart below, the shaded region represents larger customers yet to be acquired:

Source: Statistical Consequences of Fat Tails

What proportion of the total does this extra bit represent? That depends both on , or how fat-tailed the distribution is, and on the sample size. The smaller and the smaller our sample, the more we underestimate the true mean:

Source: Statistical Consequences of Fat Tails

So for an enterprise software company with , which is typical in my data set, and 1000 customers, these not yet acquired customers will account for 20% of the true mean. This means the true monetization is about 1 / 0.8 = 25% higher than current monetization.

That number is conservative — it assumes that we've already acquired every customer up to a certain size, thereby maximizing monetization of customers equal to or below that size. We haven't really done this though, so even current monetization underestimates how well we could monetize, even without acquiring larger customers. Factor that in, and the degree to which we underestimate true monetization only increases.

The math behind land and expand

Combined, the above insights form a mathematical justification for "land and expand"-style go-to-market strategies.

Here, land and expand is effectively an indexing strategy — land at as many organizations with as little investment as possible. Every once in a while you'll land a Google, a Facebook, or an Amazon (both figuratively and literally) which will drive a disproportionate share of revenue.

Even if those customers start off small, any given customer could potentially become quite large.

Further, it can make sense to overspend somewhat on establishing those small beachheads, as they likely underestimate the true average contract value. For this reason, common metrics for evaluating the efficiency of software sales like the "magic number" may underestimate the efficiency of land and expand models, especially during the land phase.

If we suspect right-skewness, the true mean is more likely to be underestimated by measurement of past realizations, and the total potential is likewise poorly gauged. — The Fourth Quadrant: A Map of the Limits of Statistics

In fact, one of the best software investments I ever made (that shall remain nameless) was in a company that on its face seemed quite inefficient, with a magic number well below 1. The leadership team preached the virtues of its land and expand model, but our static analysis of its sales metrics was doomed to underestimate its true efficiency, even after many hours spent (by yours truly) wrangling and analyzing the data.

Luckily, we got over our concerns and made what turned out to be a great investment.

With more mathematical context, the story is a visceral personal reminder to properly grapple with the implications and dynamics of fat-tailed software monetization.

The end

This is just a small taste of fat tails, and I plan to write more in the coming months on their broader implications for high-growth startups. Much ink has been spilled on this topic within the context of venture investing, but not so much for operating the underlying businesses themselves.

Here's a preview of the topics:

• Why software markets are always larger than we think
• Why "Weighted ACV" beats the traditional ACV metric
• Why investors consistently undervalue enterprise software and overvalue consumer startups
• Why open source is built by individuals rather than communities
• Why product-market fit gets harder to achieve the longer you search for it

I've been thinking about some of these essays for the better part of a year, so I'm excited to finally share these ideas.

Summary

"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" introduces the Visual Transformer, an architecture which leverages mostly standard Transformer components from the original NLP-focused "Attention is All You Need" paper but instead applies them to computer vision, specifically image recognition. Images are transformed into sequences of image patches representing "tokens," similar to word tokens in NLP. The model is trained in supervised fashion on image classification using these patch sequences as input.

The image-based Transformer does not outperform CNNs when trained on mid-sized datasets such as ImageNet, underperforming similar-sized ResNet models. This is likely due to an inability overcome the inherent advantage of CNNs (inductive biases like translational equivariance and locality). However, when the Transformer model is pre-trained on large image datasets (specifically, JFT) and transferred to other tasks, the model achieves SOTA results.

The strong results suggest the long hoped-for convergence of architectures across NLP and computer vision may finally be here in the form of Transformers. Per Andrej Karpathy, Director of AI at Tesla:

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale https://t.co/r5a0RuWyZE v cool. Further steps towards deprecating ConvNets with Transformers. Loving the increasing convergence of Vision/NLP and the much more efficient/flexible class of architectures. pic.twitter.com/muj3cR6uGA

— Andrej Karpathy (@karpathy) October 3, 2020

Key results and takeaways

• Large Vision Transformer model mapping patches of image to classification labels outperforms CNN-based architectures and achieves state of the art results when trained on large (100M+ images) datasets
• Transformer model underperforms CNNs when only trained on mid-sized datasets
• Early layers of the Vision Transformer are able to attend to large chunks of the image, unlike traditional convolutional layers with are constrained to a local window
• Self-supervised pre-training with "masked patch prediction" achieves decent results but underperforms supervised pre-training

Methodology

Architecture

The model effectively analogizes between words as tokens of larger sentences and groups of pixels as "tokens" of larger images. Like a sequence of word tokens makes a sentence, a sequence of pixel patches makes an image. Thus, the input image is broken up into multiple patches of dimensions representing square subsections of the original image (including all color channels ), forming a sequence of image patches of length .

Image patches , typically 16x16 pixels, are embedded into dimension vectors using an embedding matrix . The sequence of "patch embeddings" is prepended with a learnable token, similar to BERT, telling the model to classify the image, leaving us  with a dimension vector, .