Note: AI agents are sometimes not the right tools for the job.
Recently, a popular prediction is that software engineering will primarily be done by AI agents working autonomously in the background, while human developers mainly manage and guide these agents (supported by figures like Mark Zuckerberg and others). This differs from the “co-pilot” model of AI interaction (e.g., using ChatGPT, Cursor, etc.), which requires humans to actively guide AI outputs through small chat interactions.
Given the continuous advancements in reasoning technologies (like O3) and the growing number of users of AI assistants, the enticing vision of AI agents working 24/7 in the background feels inevitable. Clearly, people enjoy working with AI assistants. But would they be comfortable handing over significant tasks to AI, especially in a corporate environment?
In this article, I argue that the industry severely underestimates the difficulty of transitioning from a “co-pilot” system to an “autonomous AI agent” system. The core issue is that greater autonomy does not necessarily mean better outcomes—especially when it undermines reliability and explainability. In many cases, humans prefer to remain involved in critical intermediate steps to maintain control and understanding.
Essentially, the challenge is this: for many complex tasks, using AI agents to separate humans from the creative process can be dangerous and costly. What do I mean by this?
If we create a significant gap between human workers and work, filling it with AI agents, we risk creating a “knowledge vacuum” that deprives humans of rich contextual information about the final output. This increases the chance that employees will have a superficial understanding of the company’s output. When workers need to explain or troubleshoot work, or improve and refine outputs, the costs become apparent.
Of course, AI agents can provide metrics, thought trails, telemetry, etc., to impart knowledge about the work to humans, but I foresee that this will not be as simple as creating some dashboards or notifications.
It’s about balancing reliability and organizational learning with the preemptive “optical” productivity gains. Ironically, if the agent does not create enough gaps, then it is not as useful—this creates a dilemma.
This also means that the adoption of most “AI agents” will likely occur within predefined AI workflows or low-risk use cases, and they will be marketed as “AI agents.” If teams want to boost productivity through automation, they will leverage agent behavior within predefined workflows while attempting to actively “kill” autonomy to enhance reliability. That’s human nature.
To expand on this idea, suppose humans are ultimately responsible for sign-off; handing control over to autonomous AI agents is fundamentally awkward because:
-
If employees are removed from the creative process, they will learn much less: autonomous AI agents promise to free developers from granular interactions (like with ChatGPT) and shift them into auditing roles. But without actively participating in the creative process, workers cannot refine their understanding of problems and needs. They become less capable of grasping the nuances of their work.
-
Learning, guiding AI, and evaluating AI work are all interconnected: supporters of the autonomous AI agent paradigm believe that workers can reliably review AI outputs even if they are disconnected from the creative process. This might apply to simple or familiar problems but not to tasks that require many judgment calls and implicit/explicit trade-offs, which only emerge during the creative process.
-
The explainability and interpretability of autonomous software engineering are compromised: through “autonomous software development,” people trust AI to “do the right thing,” which obscures the complex thought processes and trade-offs before the final solution. This is not an issue on a happy path, but during troubleshooting or new feature development, when a superficial understanding of AI outputs is no longer sufficient, it can backfire.
-
Creating a “safe and effective” environment for AI agents is much harder than it sounds: to start using autonomous AI agents, companies must first complete the following heavy lifting: 1) organize all company data for AI agents to use, 2) provide agents with programming access to take action (e.g., Anthropic’s computer usage or Cohere’s North attempts to facilitate this “agent onboarding” process). The difficulty of this work is severely underestimated. For example, what exactly does “relevant company data” mean? Furthermore, the vast majority of Fortune 500 companies have not executed this legwork correctly.
The point here is not to publish a post destined to fail on the wholesale adoption of AI agents but to make potential AI agent buyers aware of the fallacies in the “hiring AI colleagues” narrative, which is how many startups are building their products (perhaps to justify absurd pricing based on anchoring).
In the remainder of this article, I will elaborate on these challenges of adopting autonomous AI agents. I will also make some predictions about the evolution of the AI agent market.
“AI agents” refer to systems that make autonomous decisions using large model reasoning to achieve user-specified goals and can interact with external systems (tools). The cynical ELI5 is: a script that calls the LLM API within a while loop, with some data integration and API calls. See Anthropic’s definitions of AI agents and AI workflows, which I agree with.
Why Autonomous AI Agents Are Not a Panacea for Knowledge Work?
To understand why autonomous AI agents are fundamentally at a disadvantage compared to the “co-pilot model,” let’s observe what typically happens during a ChatGPT session (or Cursor, Claude, etc.), as shown in the image above:
-
Small interaction loops (HIL) generate learning (rewards): Typically, workers start with a description of the desired output, but the original prompt rarely contains everything needed to reproduce the final output. As users “learn” by observing and evaluating intermediate AI outputs, the prompts themselves are dynamically changing. The final prompt itself ultimately becomes a “reflection” for human workers to understand the task at hand. If the AI agent makes all intermediate execution decisions, then this fine-grained learning cannot occur.
-
HIL allows for course corrections: When using AI, there are many dead ends. But dead ends can also yield insights into the limitations of the problem. By having an AI agent host the show, you miss out on counterfactuals (e.g., what didn’t work, what alternatives we considered).
-
HIL allows for injecting tacit tribal knowledge: Much valuable operational knowledge is often trapped in workers’ minds and cannot be used for RAG (retrieval-augmented generation). We call this “tribal knowledge,” “pet theories,” “experience,” etc. It is not digitized, so by definition, AI agents cannot use them, but they are often essential for task completion, especially in large, old organizations running legacy systems. Autonomous AI agents have no opportunity to absorb this into their context.
-
HIL allows for natural explainability and easier troubleshooting: Since humans are evaluating intermediate steps, explainability and traceability (the thread itself) are, by definition, out of the box. When issues arise, people can return to the thread to audit any erroneous assumptions, etc.
A more realistic AI collaboration sequence is as follows.
Typically, problem-solving requires multiple threads to occur over time, each with varying numbers of intermediate steps (chats). This is why both ChatGPT and Claude have “projects” to group related threads. Each thread helps users understand what works and what doesn’t (deep green indicates higher quality AI output).
When the first thread (thread 1) is no longer useful or the context window limitations hinder progress, we transfer what we’ve learned into a new thread (thread 2) and bake it into a new prompt, allowing the AI to start from a healthier initial state. Then we spend some time reflecting until we have enough comfort to move forward.
In practice, the co-pilot process allows AI to guide towards the final goal in a complex manner, along with multiple “escape opportunities” when we get stuck in local minima. But autonomous AI agents completely obscure this level of granularity—this could be important or not, depending on the task’s significance. In theory, performance would be affected because humans cannot provide “timely” feedback.
Shifting Responsibility to Evaluation Mechanisms
So what happens if we shift from the co-pilot model to an autonomous model? Unless the task itself is simple or easy to evaluate, I believe any optical productivity gains from AI agents come at the cost of “shifting responsibility” to evaluators (humans). Unfortunately, when humans are removed from the creative process, evaluation becomes more challenging, especially for long-running AI agents executing complex tasks.
The longer AI agents operate independently, the more micro-decisions they make based on more assumptions. This reasoning path is largely obscured, mainly because it requires too much bandwidth to evaluate both the final output and the thinking process simultaneously. This can lead to missing some critical cognitive errors made by the agent:
-
Incorrect assumptions (orange): Things assumed by the AI agent are not true (e.g., the AI decides it needs to over-design something that is incompatible with the existing codebase).
-
Unknowable judgment calls (blue): The AI agent thinks it can do something that is not explicitly prohibited but undesirable (e.g., the AI agent chooses an outdated SDK version that is not strictly prohibited).
-
Unobservable hidden decisions (black): The AI agent makes some hidden decisions that cannot be directly observed but may lower costs (e.g., the AI agent decides it is okay not to check the locale when displaying text, but this can lead to issues when deploying in right-to-left text countries).
-
The examples I provided may seem harmless, but you should easily think of which assumptions could harm your codebase or work. If human workers are merely bystanders in the creative process, it becomes difficult to parse these decisions from the outside in.
Note that all these observations apply to other knowledge work. If you are writing a long report with the help of ChatGPT, don’t just copy and paste—it might assume things that are not true. This has nothing to do with the reasoning ability of a JD, as “garbage in, garbage out” is correct. If the evaluation criteria are low, then it benefits you.
Where Do Autonomous AI Agents Belong?
Thus, the real challenge in building autonomous AI agents is finding the right “rhythm” to “loop” humans in, maximizing human understanding of how the sausage is made while saving time. I believe this will remain challenging for some time, but it’s something I am actively working on/thinking about (with those I work with).
The challenge is that this “rhythm” is highly personalized. Some people can grasp information faster and learn more from less information, while others need hand-holding. So this is not a small issue.
Meanwhile, I believe AI agents should not strive to be fully autonomous AI agents. Fortunately, there are some validated, useful patterns. For instance, agents are well-suited for prototyping or exploratory use cases (like prototyping, resource gathering, etc.) and serving as “decision gate” components in automated workflows (like tier 1 and tier 2 customer support). These use cases are significant opportunities in their own right, best suited for AI agent startups rather than distorting their value by trying to compare themselves with human workers (and failing) based on job descriptions.
The following diagram presents a simple framework for making decisions between co-pilot mode, autonomous AI agents, and AI workflows. Generally, leave complex and highly ambiguous tasks to humans. This seems almost common sense, but clearly, we need a full Substack post to offset the recent hype around AI agents.
Note that if your work is entirely following standard operating procedures (SOPs) that are 100% articulated in documentation (admittedly, many white-collar workers find themselves in this situation), then the case for “autonomous AI agents” replacing you may be stronger.
I will save my detailed thoughts on excellent AI agent use cases for future posts.
Predictions for the AI Agent Market (and Employment)
There is a “tug-of-war” between the desire for human involvement in the creative process and the desire to save time. For enterprise use cases, I believe in most cases the former will win because reliability and trust are more important than speed. That said, I am optimistic about general “agent behavior” tied to programmatic (“AI workflows”) use cases, or using exploratory AI agents as plugins or coding IDEs to perform easily understood tasks.
Moreover, I do believe AI agents—while they won’t directly replace humans—will still take on many tedious and heavy tasks, becoming a persistent force in the tech job market overall.
I will conclude this article with some predictions for the AI agent market in 2025:
-
Due to poor information transmission, AI agents will continue to confuse buyers (please share this article with anyone who feels confused): the tech industry seems intent on selling any software using a JD as “AI agents,” even if it fundamentally has no autonomy and is indistinguishable from scripts or workflows. I expect this to persist, as positioning as AI agents allows you to charge more by anchoring against human costs. However, I believe the truth will ultimately prevail, and customers will gradually become desensitized to AI agent messaging.
-
Companies will always prefer AI workflows over autonomous AI agents: Autonomous behavior is not a “feature”—it is more of a “substitute.” If autonomy can be eliminated, companies will lean towards eliminating it to increase the reliability of the program. Therefore, incorporating “autonomy” into the value proposition makes no sense.
-
AI agent startups will struggle to compete with point solutions, at least in enterprises: Several AI BDR, AI SDR startups, etc., position themselves in the “AI labor force” category, but I do not believe buyers will pay enterprise prices for glorified Slack bots. Theoretically, existing CRM or marketing automation companies are better positioned to offer these AI agents as supplements.
-
The B2C AI agent market will be very challenging; if anyone can win this market, it will be Apple + OpenAI (iOS) or Google: I do not believe segmentation will work here, as agents are only useful in universally applicable scenarios.
I hope this article provides some help. If you have any questions or suggestions, please leave a comment, and we will do our best to respond!
Let’s explore and push the frontiers of technology together!🚀💻
Good luck!😊✍️
References
[1] https://nextword.substack.com/p/why-ai-agents-feel-useless-despite?publication_id=1745191&post_id=154902773&isFreemail=true&r=1g4rup&triedRedirect=true
-END-
