On AI agents in software engineering

In certain corners of the Net, one hears that software engineering is undergoing a great change. Code is no longer carefully hand-crafted by people who just want to deliver a reliable product. Now we place agents in custom-built harnesses and drive them in loops on graphs.

Myths are only as true as the worlds their bearers create, and in the world I see, people are quite uncertain about the role of agents in the production of software. As a PhD student, I use tools like ChatGPT, Claude, Gemini, and co. daily in my work to bypass the tedium of reviving old academic machine learning code. My productivity has gone up, and I can read papers or think about experimental design while AI monitors my experiments and analyzes my data. But some things about using these tools make me feel icky and uneasy.

The tools are useful, but I can’t inspect or control the systems I depend on today. What if my AI agent injects someone else’s intellectual property into my codebases? Can I trust my AI agent to not hack me? If I’m using a corp-provided model, is this AI agent even mine?

I concede that I’ve come up in a world that benefits massively from the effects of the free software movement, but are we really going to let closed AI models with bad security run our (computing) lives? Perhaps the answer, insofar as one must use LLMs, lies in using more transparently developed models like Olmo or Apertus?