DevShorts Conversations - The Goal Was Never CI - With Rob Zuber
A conversation with CircleCI CTO, Rob Zuber on Agents, CI, Sandboxes, and Software Process
AI has made code generation easier. But software engineering processes like standups, sprints, PRs, story points, and planning were designed when writing code was expensive.
Now code can be generated faster, but CI, PR review, testing and approvals are still catching up. This became the central theme of my conversation with Rob Zuber.
Rob has led engineering at CircleCI for more than a decade. In that time, he has helped rebuild the platform through major shifts in software development, including Rails monoliths on Heroku, mobile-first development, Docker, Kubernetes, and now AI agents.
CircleCI is seeing this shift from close range. Teams are generating more code and running more builds, but that does not always mean more software is getting merged into the main branch. Engineering teams are also burning CI cycles discovering broken builds from agent-generated code.
But I did not want this conversation to be only about a report or a product update. I wanted to understand how software engineering itself changes. What happens to CI as a category? What happens to the pull request as an artifact? And what happens to the role of the engineer?
Key points from Rob
Rob joined from San Francisco, and the conversation went in many directions.
A few things he said kept coming back to me after the call.
Engineering leaders are seeing their hard-earned credibility get “rapidly devalued.” For Rob, staying close to the code is how that credibility stays real. That is why he still contributes to core codebases and was on call the week we spoke.
CI should move from catching failures first to confirming that the code is ready. By the time code reaches the pipeline, it should be expected to pass.
Many of our processes exist because we are still afraid to write code. When the cost of writing code changes, the process built around that old cost should change too.
Sandbox companies are “a dime a dozen,” but just having a new compute layer does not replace the full CI business.
Directed feedback to agents can reduce token spend by 60 to 80 percent. Sometimes, even stripping junk out of test output makes a big difference.
Two-person review is just one way we chose to manage risk. It is not the only possible way. We can build better controls for the new workflow.
This is the first in a new Dev Shorts series of conversations with people building developer tools. The transcript below is edited for length and clarity.
1. “I’m on call this week”
Aravind: You run engineering at a company whose customers are engineers. AI is also changing how engineering teams work. How has your own job changed?
Rob:
There are two parts to it.
One part is product. Our customers are very technical, so our product strategy is deeply informed by how our own engineers work.
The other part is leadership. I am leading engineers who are working very differently from how I worked as an IC.
I think a lot of engineering leaders are reaching a point where the expertise and credibility they built over many years is being devalued very quickly. The way I have done this work over the years is changing.
For me, I cannot think about engineering work without doing some engineering work myself. I cannot just say, “I bet it works like this” and be credible. So I have had to sit down and do proper work. Not the kind of work where I build a side project over the weekend and then ask, “Why can’t everyone get this done?”
I mean actually contributing to core codebases within the platform, so I can understand the challenges our customers and our engineers face. I am not putting myself in the blocking path, but I am trying to stay close to the experience.
I’m on call this week, actually.
Aravind: And what has that hands-on time changed about how you see the tools?
Rob:
We went through a transition. Early on, people said these tools were great for prototyping, but we would still do the serious work ourselves. That view matched what the models could do at the time. It also made engineers feel better. But that changed really quickly.
Now the question is different. Production work has more risk than prototype work. It needs more stability, better design, and stronger checks. So yes, I am still going to use generative AI. For production work, I will use it inside a harness that helps keep the quality high.
For low-risk prototypes, I can use it in the most “YOLO, who cares” way. I just want a functioning prototype as fast as possible, and I do not care if it scales. Risk management is the big thing right now. You need to know which activity is low risk and which activity is high risk, and choose how to work accordingly.
Honestly, the models are overqualified for some of this at this point. The bigger conversation everyone is having is how to drive down token cost, because all my engineers are using Fable to write hello-world apps.
We are industrializing. That is the process we always go through. Things start extremely expensive, hard, out of reach, but motivating. Then, if they are valuable enough, we learn how to optimize them. We are in that optimization phase right now.
2. The goal was never CI
Aravind: CircleCI has been the leader in this category for a long time. I do not want to ask what is new in agentic CI, because everyone is already discussing that. I want the strategic view. How is your thinking about the company itself changing, even if it changes every three months?
Rob:
Three months is a lifetime in this space. But strategically, I am a fan of Simon Sinek’s “start with why.” Continuous integration is the category name. We can debate whether using “CI” in the name in 2011 was a great idea or a terrible idea. But the important point is that the goal was never just to build CI. The goal is to help our customers get their software into the hands of their customers as quickly as possible, with high confidence.
We used to think speed and quality were a tradeoff. Then we learned that moving faster, when done right, makes you better. If you tune your process for speed with proper checks, you get quality too. If you tune it for quality with fast feedback, you get speed too.
If that is the goal, CircleCI has already been through many versions of it. We started in 2011 with Rails monoliths deployed on Heroku. The typical customers then were startup teams building from garages around Palo Alto. Then came mobile-first around 2013. Then Docker shipped. Everyone started containerizing. Microservices became popular. Everyone started building on Kubernetes. Then came Rust, Go, TypeScript everywhere, and Next.js. The way people build and ship software has changed many times. We have kept rebuilding the platform to serve the same goal.
Now take that pattern into the AI SDLC, or whatever we end up calling it. The question becomes this. If people are primarily building with agents, how are they going to build with high confidence and ship quickly? A core part of our thesis is that we need to get feedback directly to the agent. The agent needs to know what it is building and whether it is on the right track, because everything else is too late.
What we see today is a lot of branch builds. Teams are using CI to validate their agent-generated code. But more branch builds do not always mean more code reaches the main branch. There is a lot of activity. Maybe not the same amount of productivity.
Aravind: So where does the pipeline itself end up?
Rob:
I think there is still a final gate before things go to production. Otherwise, we will all be in a very dark place. But by the time something gets to CI, you should assume it is going to pass. All of that feedback can be given much earlier. We already know how to validate software. We should just do that in a very different place.
Is that still CI, capital C, capital I, Martin Fowler-approved, the one true way to build software? I doubt it. Everything we have built over the last 25-plus years has evolved around how humans think and work. If that is not who is driving the process anymore, we have to fundamentally rethink it.
A note from me: This is the most interesting admission an incumbent CTO can make. CI still has a role, but it moves from being the main workbench to being the final validation step. The interesting work, in Rob’s framing, happens before the commit exists.
3. Software Process Assumes Code Is Expensive
Rob:
Rethinking the software process is hard because we are not good at big leaps. As an industry, and honestly as a society, we usually move in small steps. I believe in process engineering. I believe in the theory of constraints. But if you look at the software engineering process, we have built many assumptions over the years around the cost of creating software.
The time we spend on user research, planning, breaking epics into stories, and breaking stories into tickets all comes from one assumption. Writing software is expensive. We invented Agile around 25 years ago, and the first thing we did after that was create rigid processes and call them Agile. That meant sprint planning on Monday morning, daily stand-ups, and story-point poker. If you do not do those things, people say you are not Agile. That created a way for people to write books, run workshops, and build consulting businesses around “how to do Agile.” But they were not the real spirit of Agile.
Part of the reason we created those processes is that we were still afraid to write code, because writing code was expensive. If that cost changes, and generative AI has changed it, then the process built around the old cost is probably not the right process anymore.
Yes, we are still concerned about tokens, but open-weight models are already good enough for a lot of this work. So the cost of creating software has still shifted. This is not just about PRs becoming a little more expensive, or merging becoming a little more expensive. The real question is different. What process should we design when software is much easier to build?
The Agile Manifesto is four lines. One of them is working software over comprehensive documentation.
We may finally be at a point where I can say, “Build me a thing that does this,” and get something working. It may not be production-ready. It may just be a high-fidelity prototype. But that is enough to start a real conversation about what we are actually building.
That changes how a lot of engineers think about their role. Many of us have been oriented around being the implementer. Someone else figures out the business requirements, and the engineer implements them. But when implementation becomes as easy as typing a couple of sentences, the value shifts. Engineers are moving more and more toward product and strategy. That is what we promised ourselves out of Agile and then fought against by burying it in process.
4. Sandboxes are a primitive, not a business
Aravind: Validation is clearly moving into the inner loop, and that is where the sandbox companies live. I never imagined a Docker container becoming a category, but here we are. MicroVM sandboxes are becoming the new substrate for agent execution. Do sandboxes eat CI?
Rob:
CI has always been a layer of value built on top of commodity infrastructure. You could buy some EC2 VMs and run your own CI on top of them. I can assure you, I have a business. The numbers say we have a business, because people do not want to run CI infrastructure themselves.
Sandboxes are a slightly different primitive. Many of them are Firecracker microVMs running on someone’s infrastructure. But my customers do not care whether I run their workloads in a microVM, an EC2 VM, or a Docker container. What they care about is how much that environment looks like their production environment, and how fast we execute.
New technologies like microVMs can help us run things faster and cheaper, with better margins, at the same or better quality of service. But does a different compute primitive destroy the CI business? I do not think it does.
Where sandboxes are getting most of their value right now is in constraining agents. I have an untrusted entity, and I do not want to approve every action manually. I want an environment where the default is yes, and I know that environment is safe. That is real.
On the other hand, sandbox companies are a dime a dozen. I could name 25 of them right away. Running Firecracker on someone else’s box is not a defensible moat. It is a piece of technology anyone could spin up in an afternoon.
So what we will probably see is a split, with some companies moving beyond basic infrastructure and focusing on specific problems. Fly.io, for example, is offering infrastructure to run production-like environments. I think of them as competing more with Vercel than with CircleCI. Daytona is doing more in agent execution.
Will people try to build CI solutions with sandboxes? Maybe. But they will not become much better or meaningfully different just because they use sandboxes underneath. The value will come from what problem the product solves on top of it.
We use similar technology under Chunk Sidecars. Sidecars are an attempt to solve a specific problem for a specific user, not to provide an infrastructure primitive.
5. The Token Bill Comes Due
Aravind: Here is the pressure I keep hearing about. CI used to stop. There were nightly builds. Now teams ship 500 PRs a week, the pipeline never sleeps, and nobody is sure the checks are even meaningful. What happens to the cost model?
Rob:
CI performance and CI cost are absolutely becoming more important as people build more. There are two parts to it. One option is to run less CI, but that usually means batching changes and slowing down. So the better answer is to run faster CI, and do less compute for each result.
Running faster CI means continuing to invest in wall-clock time and parallelization. Using less compute is the more interesting part. Chunk Sidecars run the smallest possible piece of work next to the agent, so by the time the full CI job runs, it is probably going to pass. Fewer failed builds means teams spend less for each shipped change.
Inside the CI job, Smarter Testing looks at the change and picks the most useful tests to run first. That way, teams can build confidence with smaller pre-checks instead of running the full test suite every time. Then, at the final gate before production, run all the tests and security scans. We believe in the value of spending money on CI. I just want people spending it on the high-value parts.
On tokens specifically, if your costs are climbing but your productivity and delivered value are climbing more, you are fine. You are making more money as a business. If that is not true, that is the thing to fix.
There are also practical ways to reduce token spend. When we give the agent, directed and specific feedback early in the cycle, about whether it is on the right path, we sometimes see a 60 to 80 percent reduction in token spend. The agent is not waiting for a human to say, “No, try again, maybe try this.” It gets to the right outcome faster.
There is also a very tactical piece. When you run tests, you get back a giant pile of output. Whitespace, control characters, and junk all get consumed as tokens. We know what an error message looks like. We know what the agent is actually looking for. We have been reading this output for fifteen years. So we compress the output and send back only the useful part. The agent does not have to burn tokens scanning through noise to find what matters.
6. Convince Me With Your Tests
Aravind: Let us talk about the human role. The contract used to be like, Everything is green, then a peer checks the business logic. In a world where agents write first and merge fast, what is the artifact a human actually verifies? Model companies, agent companies, everyone is arguing for something different.
Rob:
We are very motivated by this question right now, internally and for customers. Start with how I read a PR today. The PR interface gives me a list of changed files in alphabetical order. I go through them in order, and halfway through I realize the change is too big to hold in my head.
So I look for the key changes. I try to understand their context. And then, because I am a human and not a computer, I ask, where is the test that proves this does what the author intended?
That is my mental model of the artifact, even without AI. What gives me confidence, today and in the future, that this change does what the author intended?
So often, there are no tests. Or there is one test proving the most obvious case, while complex logic is not really tested. My review feedback becomes, convince me with your tests that this works the way you expect.
The problem is that we have had a basic understanding of test quality. Code coverage is better than nothing, but it can also be a distraction. It can tell you the wrong things. I have spent days writing bad code just to hit a coverage number.
Test quality is the more interesting pursuit. We have done some work around mutation testing, which is a very old and very simple idea. Take the code and make a small change that should break it. Flip a sign. Then see whether the tests catch it. If I modify the code in a way that should break it and everything still stays green, then I do not have real coverage.
We built something to try this because we happen to have arbitrary fast compute. Take a small codebase, run a mutation analyzer across it, and you get 2,500 mutations. Now you need to run the test suite 2,500 times to find the gaps. That is suddenly feasible.
It is algorithmic and it sits below intent, but it is a start. It gives me a real measure of how good my validation is, so I can actually trust the green checkmark.
Aravind: And beyond the checkmark? PRs are getting bigger, not smaller.
Rob:
Right. We can now do so much work very quickly. But if we try to keep every PR to ten lines, we may end up shipping thousands of PRs a day. That breaks the rest of the system.
So within a larger change, there are logical components. Code is often written chronologically, through many iterations. But the change can be repartitioned into sensible slices. These components feed this function. That function provides this capability. I do not need your false starts in the Git history.
Once you have slices, you can identify which ones are low risk and which ones are high risk. A structural component that follows all our standard patterns, has no security surface, and has no embedded tokens can flow through. Another slice probably needs human review.
If you reject every fifth PR for human review, and that PR is 10,000 lines, the human is back to reading files in alphabetical order. They are hunting for a needle in a haystack that algorithms already found. Just show me the risk.
One part I do not have a great answer to yet is semantic understanding of the change. When we talk about systems, we draw diagrams. But when we talk about changes to systems, the only tool we usually have is reading lines of code. For a five-line change, that is fine. Show me the code. But for a large change, can you show me the difference semantically? Can you help me understand what is evolving, and where the risk is concentrated?
Then you still need the audit trail, because someone in compliance will ask who the second person to sign off was. But here is the thing. The requirement to have two humans read every piece of code does not exist anywhere. It is just the control we chose to reduce risk. Then auditors copied it across organizations. We can build new controls. We just have to build them.
A note from me: Most of what we defend as “engineering rigor” is actually one control that fit one cost structure. The need for rigor is real. But the ritual can change.
7. Finally doing the original Agile
Aravind: Parting thought. People will keep planning and keep validating, but the day-to-day of writing and reading code is dissolving. If we had to write the manifesto for this era, what does it say?
Rob:
The really interesting thing is that I think we might finally do the original Agile. I have had this conversation a few times recently.
Working software over comprehensive documentation. Functioning code is now cheap enough that it makes sense to build the system and evolve it. We do not have to itemize every requirement upfront before we start.
Some of the original Agile Manifesto authors recently did a 25th-anniversary revisit of the manifesto in Utah. And going back to “start with why,” the intent was there. The intent was good. The intent was right.
I do not know that I would rewrite the intent. But I would certainly question many of the actions we have taken as a result of it.
8. What AI companies get wrong
Aravind: Last one. What is one thing AI companies are doing terribly wrong, on both sides: the foundation labs and the AI-forward builders?
Rob:
I will take the builders first. When we have a new thing, we often think everything has to be new. Software development is fundamentally changing, but we have already solved a lot of problems in software development. Mutation testing is from the 1970s. A lot of useful ideas already exist. But because things look different on the surface, we reinvent them and later realize the problem was already solved. That gets expensive. Absolutely innovate. Absolutely bring those ideas into the current context. But do not assume everything has to be net new.
Interestingly, it goes both ways. In the early 2010s, there was a big wave of reinventing organizational structure in software companies. Most of it eventually got thrown out. Everyone eventually said, Peter Drucker solved this 50 years ago. Let us have normal management. But now the leverage from agents really is forcing us to rethink team shapes. Smaller teams can be more productive, and the communication cost of adding nodes is high. Back then, people were doing it because someone wrote a blog post and it seemed fun. Now the underlying constraint has actually moved.
On the foundation labs, it is easy to make lofty statements, so I will not. But here is what I am genuinely curious about. The major model companies ended up targeting software engineering as the vertical. That is the classic Crossing the Chasm move. Own one vertical completely, then replicate. Developers, or at least their bosses, were willing to pay to optimize an expensive manual process. I understand why Claude Code wins and why Codex wins. They control the fundamental component at the heart of the system. The most important thing a coding agent can do is have a great model interaction.
But now they are pursuing the rest of the SDLC. Cursor has pre-announced that they are building their own version control and moving off GitHub. Obviously, I have a dog in this fight, and so do you. I do not know that they have unique value to bring to those pieces of the puzzle once the model is not the key ingredient. I also question whether it takes attention away from where their value is highest.
If I remember Crossing the Chasm correctly, and I read it 23 years ago, the play is to own the vertical and then replicate into other verticals, like design, legal or whatever. Going deeper into software development instead risks exposing you to someone succeeding closer to your core. Then again, they are making hundreds of billions, so they are doing something right that I am not. And I am totally fine with that.
9. Do human things
Near the end, we talked about what all of this means for people who are just starting out.
Rob has two sons, one in college and one about to be.
Rob: They ask me, what is the future? And I say, I have no idea, man. But do human things. Be graceful. Think deeply about problems. Learn to problem-solve. Be a good human being. Build relationships. That stuff is never going away. I think there is a future where our craving for human interaction gets stronger as our days become more artificial.
Conclusion
The conversation was about how software process changes when code becomes cheaper to write. CI, tests, and human judgment still matter. But feedback has to move earlier, review has to become more focused, and the green checkmark has to mean real confidence, not just that the pipeline finished.
Thanks to Rob Zuber for the conversation, and to the CircleCI team for making it happen.
You can find Rob’s team’s work at circleci.com, including Chunk Sidecars and the State of Software Delivery research we touched on.
This is the first in a series of Dev Shorts conversations with technical leaders shaping how software gets built. If there’s a leader you want me to sit down with next, reply to this email. I read everything.