ai-yi-yi

Like everyone else, still trying to get a read on the present and future of AI in tech. Most everyone is daunted by being on the wrong side of a "have and have not" divide in terms of being able to use these tools, and thoughtful people have concerns ranging from the cultural to the environmental about the long term impacts.

People's results when handing parts of development over to AI seem to vary greatly, and it's hard to know if people are tackling different problems (so it's not apples to apples comparisons) or if some people are using better tools (or throwing more tokens at things!) or using methods to provide more structure to the AI, vs the "prompt and pray" approach.

It may be useful to see places that are taking more experimental and data-driven approaches, than just relying on anecdotes...

One paper was "How AI Impacts Skill Formation" (by some Anthropic fellows): 

from the abstract:

"Novice workers who rely heavily on AI to complete unfamiliar tasks may compromise their own skill acquisition in the process. [...] We find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average. Participants who fully delegated coding tasks showed some productivity improvements, but at the cost of learning the library."

Another video is "We Studied 150 Developers Using AI" that had a special focus on longer term maintainability, vs just the initial cost.

"Well, the headline is there was no significant difference between the cost of maintenance between AI and human generated code. That's interesting and perhaps not what we might have expected. Code written with AI assistance was no harder to change, no easier to change, no worse in quality and no better in quality."

And also, besides some of the speed up in dev, dev who used AI on the the regular tended to make more idiomatic "normal" code.

Anecdotally this lines up with some of my experiments with Claude Code - where you give high level direction as a PM/QA rather than using CoPilot or ChatGPT has a coding buddy. I feel like the sweet spot is using AI to augment rather than replace the classic developer role. Maybe some of that's wishful thinking, maybe some of that's recognizing the cost these systems have and how they really can lose the thread sometimes. (I mean so can humans, but it has a different feel...)