Talk Is Cheap: The Operational Impact of LLM Use
What the data says about the operational impacts of LLM use in the software industry
This is a continuation of the discussion in How I’m thinking about the value of LLMs . I’m arguing elsewhere that LLMs will never be geniuses. This is not part 2 of The Ontology Argument .
In How I’m thinking I said I wasn’t ready to take a stance on LLM value creation. That changes in this post. Here is the stance I’m taking:
On average, how we’re using LLMs is likely destroying value.
My stance originates from stumbling on Faros.ai - a software development telemetry firm. They have products that pipe into common development tools like Jira, Github, and CI/CD pipelines to directly measure major operational metrics for software development teams.
Faros published a report in March that directly compares transaction level data between teams using AI in their software development process vs those that are not across their customer base. 22,000 developers, 4000 teams in the sample. This is, by far, the best data I’ve been able to locate that directly measures the operational impact of use of LLMs in the software development process.