AI Can't Count

Try this prompt with any LLM: "Write an inspiring story. The story must be exactly 515 words. At the end of the story, state how many words it contains." Then paste it into Word or LibreOffice and see the actual word count.

Why Does This Happen?

LLMs, they're like a mix of super-advanced data compression and search algorithms. They try to predict the next "token" based on the training data and prompt you give them. So, when you ask them to count something, they don't really count - they just make an educated guess. They simply can't count (There are techniques for making it work, but that's a topic for a separate conversation).

Why Does It Matter to Me?

As a former software developer and one of the leaders of an IT company, I often reflect on how AI may change the lives of software developers and transform the way we build software.

The issue with counting words accurately shows the true nature of LLMs - they are just advanced prediction algorithms, not truly intelligent or creative, even if they sound convincing. To make them even good assistants for developers, we need to feed them a ton of data and do it repetitively.

For example, to handle something as boring yet crucial as work estimation, the data should include not just source code with a history of changes, but also detailed task descriptions, links between these tasks and specific changes in the code, different logs, and profiles of the developers who worked on them. Needless to mention, the business context behind key decisions.

I'm wondering how many companies would be willing to share such sensitive data? How much of such data we need to make estimates made by LLMs reliable? How frequently should such training sets be updated since libraries and the ways we write code are changing very fast?

It seems like solving the data issue alone is pretty challenging, which makes the idea of a fully self-sufficient AI software developer very hard to achieve anytime soon.