AI Engineer's Challenges — Part 1: Intro

I’m leading the development of an AI (agentic) platform at Productive. While my VPE role doesn’t leave me much room for focused dev time, I’m still coding a lot and trying to stay as low-level as possible in day-to-day operations with my team. I believe this is essential for engineering leaders in the AI engineering space because we’re operating in an uncharted territory we don’t yet fully understand.

When you develop AI-enhanced software, you introduce nondeterminism into code, and that changes the game of software development completely. Instructions (prompts) are (what used to be) code, and the model provider is a runtime for that code. All “traditional” software design patterns (and problems) still apply, but it’s kinda weird when you recognize and try to use them.

Basically, you code by talking to your AI runtime about your data structures and logic. Interestingly, this makes some parts of the development process easier and some much harder. 🙂

The core issue is that there’s no way to know if your code is working. Not because of the AI, but because your code is pure English, and not a programming language. Your “code” and your LLM runtime now contain a lot of ambiguity, miscommunication, biases, wrong assumptions, etc. The only way to know whether it works (whatever you think “works” means) is to test all possible scenarios and validate their outputs. This process is called “evals”, and as you can imagine, you can’t achieve 100% test (eval) coverage.

On top of that, you still have “regular” code and servers/browsers/ux to deal with, of course.

When explaining this to other people, I usually say something like: “It’s easy to get the first 50% working, really hard to progress through the next 30%, and impossible to finish.”

This idea can be represented as a logarithmic curve of effort vs. progress. It’s both scary and an exciting realization.

Originally posted on LinkedIn, where it ended with a call for AI engineers to join the team at Productive.