We will never have enough software developers.
Developers are dropping out of the profession in large numbers despite efforts to grow the number of computer science graduates and software engineers.
Software development has a serious retention problem:
Engineers often leave engineering for non-STEM management roles. Graduation into management is not surprising. What's surprising is that these are non-STEM positions. Engineers swap technical roles for non-technical roles over time.
This phenomenon, which I'll call "developer dropout," is a real problem. What's behind it?
Thoughtful analysis of the business and economics of tech
Programming-related jobs have high rates of skill turnover. Over time, the types of skills required by companies hiring software developers change more rapidly than any other profession.
To demonstrate this, researchers analyzed job postings on more than 40,000 online job boards and company websites between 2007 and 2019, controlling for employer, location, and occupation. They defined "new" skills as those that were rare or non-existent in 2007 but prevalent in 2019 and "old" skills as those that were prevalent in 2007 but rare or extinct in 2019.
About a third of the change in required skills in computer-related occupations is due to specific new software:
Data science, machine learning, and AI saw big increases among technology-intensive jobs as well. For example, the number of STEM-related jobs requiring skills in machine learning and AI grew more than 4x from 2007-2017, touching more than 15% of STEM jobs:
To better compare rates of skill change across occupations, the researchers came up with a measure of skill change that tracks the absolute growth or decline of various skills within each profession from 2007 to 2019. Occupations whose required skills change rapidly in prevalence among job postings receive a high score, while jobs whose skills do not change much receive a lower score:
We can get even more granular and look at specific job roles. This level of detail makes the difference even more stark (only showing the fastest changing roles):
Web development has the highest rate of skill change among all jobs in the country. Next up are sales engineers, another often technical role. Database administrators, computer network architects, sysadmins, and application developers all make the top 10, and we see many other technical roles among the top 30. The mean and standard deviation are similar here, placing web development more than 3 standard deviations away from the typical job in America in terms of skill change over time.
Suffice to say, software development is a rapidly changing profession.
You might think, however, that skill change would eventually settle down as one becomes more experienced.
You'd be wrong. The skills for software engineering jobs change rapidly throughout the entire career lifecycle:
This means that experienced STEM workers seeking employment in 2019 are often required to possess skills that were not required when they entered the labor market in 2007 or earlier.
Software engineers never escape the skill-change vortex, even many years into their careers. Experienced engineers must learn and adopt technologies that didn't even exist when they started out. Developers must constantly retool themselves, even well after their formal education ends.
College majors associated with faster changing jobs pay more early on.
Fast-changing fields pay better.
Notice however that the pay advantage declines over time. By the time one approaches the age of 50, the pay premium for working in rapidly changing fields falls dramatically to only 20-30% vs slower changing professions.
Here's another way to see the eroding pay advantage. The below chart simulates the earnings of the average worker by category of college degree from ages 23 to 50 in 2016 dollars.
Excluding business majors, the earnings premium of software engineering declines over time in both percentage and absolute dollar terms, to the point where engineers barely out-earn social science majors:
But the focus on college major is somewhat misleading. This phenomenon has less to do with one's field of study and more to do with choice of occupation.
To show this, researchers plotted the earnings premium of various categories workers relative to those with a non-Engineering/Computer Science major working in a non-Engineering/Computer Science job.
Declining relative returns is a feature of STEM jobs, not majors. The earnings premium for non-STEM majors in STEM occupations starts off near 40%, but declines to 20% within a decade. In contrast, the relative earnings advantage grows over time for computer science and engineering majors working in non-STEM occupations.
The profession of software development drives the declining earnings premium, not the college major.
In fact, computer science majors who work in non-CS fields experience the opposite dynamic of their non-developer peers — their relative earnings premium rises as they advance. A CS major who eschews the profession doesn't earn much more than otherwise similar non-CS majors early on, but eventually out-earns their peers by nearly 20%.
OK, that's enough about what is happening. Now let's see why it's happening.
Imagine a simple model where workers choose their profession in order to maximize income, which is a derivative of their own skill or human capital. Over time, workers gain new skills, while the value of their existing skills depreciates somewhat due to changing times.
Some workers, endowed with superior ability, learn faster than others, picking up skills at a quicker pace. Those workers will tend to sort into high-skilled, fast-changing professions initially, maximizing their early career earnings. Less impressive workers will sort into low-skilled, slower-changing professions.
In a world where human capital never depreciated, we could imagine that high-skilled individuals like software developers would maintain a relative human capital (and earnings) advantage over other professionals, leading to consistently increasing pay and a stable relative premium:
But, if human capital depreciates over time and that rate of depreciation is higher in rapidly-changing fields like software development, then developers' initial advantage would erode over time, narrowing the gap vs. non-developers:
This simple model helps explain what we see in the data — the software engineering earnings advantage disappears as the effective human capital gap narrows.
Applied majors such as computer science, engineering, and business teach vintage-specific skills that become less valuable as new skills are introduced to the workplace over time.
Specific skills in software development quickly become dated. Programming languages and development frameworks go out of style. Hadoop is hot one year, and it's old news the next. Like a fast, expensive car that quickly loses value as it's driven around town, the skills and human capital of software engineers fall apart without constant, expensive maintenance:
Intuitively, careers with high rates of obsolescence require workers to learn many new tasks each year, which diminishes learning gains and lowers the returns to experience.
The hits don't stop there. Ironically, the quickest learners are also the quickest dropouts.
To understand why, think back to the model we just outlined. Quick learners accumulate human capital faster than their slower peers, which means they have the most to lose when certain skills or abilities fall out of favor. In fact, the return to being a fast learner is higher in jobs with low rates of skill change because the learnings can compound over time instead of dwindle in relevance.
High-ability workers are faster learners, in all jobs. However, the relative return to ability is higher in careers that change less, because learning gains accumulate.
Said another way, the opportunity cost of working in a rapidly-changing field is highest for the best learners. This creates immense pressure to drop out of software engineering and other fast-changing careers into more stable roles and industries.
The researchers show this by regressing STEM job status on a number of other variables, including an interaction between age and score on the Armed Forces Qualifying Test (AFQT), a common measure of cognitive ability. The coefficient comes out negative and statistically significant, implying that the relative probability of working in STEM at any given age declines with cognitive ability (column 1 and 2 below):
The results imply that a worker with cognitive ability one standard deviation above average is 4.9 percentage points more likely to work in STEM at age 23, but only 1.6 percentage points more likely to be working in a STEM job by age 34.
In fact, by age 40, the regression predicts workers with higher cognitive ability are less likely to work in STEM than those with lower cognitive ability.
Simply put, professionals with higher cognitive ability drop out of STEM careers earlier and faster:
... STEM majors with higher scores on the Armed Forces Qualifying Test (AFQT)—a widely used proxy for academic aptitude—leave STEM careers more often and at younger ages.
Two points I want to drive home:
First — software development has a turnover problem.
Growing the supply of software developers is not trivial because the field already sees high levels of developer dropout and turnover, and this would only increase if the field were to grow larger. A larger software development labor pool would presumably drive down wages, encouraging even more developers to shift out of the profession, especially those past their career midpoints. On that flat part of the earnings curve, the incentive to remain a developer is weak at best.
Second — software development also has a selection problem.
The highest ability, fastest learners disproportionately leave the field over time. They have a multitude of other ways to profitably leverage their intellect and skills. Software development carries serious opportunity cost. Again, this is ironic because one would normally expect the best to stay and worst to leave, but that's not what we see in the data. The software development talent pool mix shifts toward lower cognitive ability as any given cohort matures.
Combined, these points suggest software development may be destined for perennial labor shortages unless the pace of change slows sufficiently.
... rapid technological change can lead to a short shelf life for technical skills. This tradeoff between technology-specific and general skills is an important consideration for policymakers and colleges seeking to educate the workers of today, while also building the skills of the next generation.
To conclude, I emphasize: highly skilled people prefer highly stable careers in the long-run. This lets their relative ability and human capital advantage compound over time. Rapid deterioration of skills continuously levels the playing field, preventing the best from separating themselves from the pack. In such a situation, it makes more sense to quit the race early than get caught in the pile up.
Thanks to David Deming, Kadeem Noray, the authors of the study from which much of this essay is derived.
Thoughtful analysis of the business and economics of tech