🎙️ FDE Talks, Episode 2 - A conversation with Michael Levan, AI Architect and Forward Deployed Engineer at Solo.io, about discovery, agent sandboxing, custom demos, technical depth, and earning customer trust.
The cleanest failure is solving the wrong problem well
An AI agent does not have to crash to fail. It can return a polished, confident answer that happens to be wrong.
Customer-facing engineering can fail the same way.
The demo runs. The architecture looks sensible. The POC passes. Everyone walks away happy. Months later, the deployment struggles because nobody stopped to ask what the customer actually meant when they said, “We need a sandbox.”
Maybe they wanted to keep generated code away from the host filesystem. Maybe they needed tenant isolation. Maybe Agent A could never be allowed to reach Agent B. Or maybe their security team required a stronger isolation boundary altogether.
All of those requests can arrive in the same sentence.
That was the idea that surfaced with Michael Levan: the FDE is a human verification layer between customer intent and production architecture.
The product gives you capabilities. The customer gives you requirements. Neither automatically gives you the right system.
Someone has to interrogate what sits between them.
“We need a sandbox” is not a requirement
Michael used agent sandboxing as an example of where this becomes obvious.
A customer says they need a sandbox. Ask why, and the answer might be isolation. Ask isolation from what, and suddenly the architecture starts to change.
Do they want to prevent generated code from touching the host? Is the concern cross-tenant access? Could the workload be malicious? What credentials exist inside the environment? Can the workload talk to the network? What happens if it escapes?
Only after those questions do technologies become useful.
Something like gVisor provides a different isolation model from a microVM architecture such as Firecracker. Both may appear on a slide under “sandboxing,” but they create different boundaries and different tradeoffs.
The important FDE work therefore happens before the architecture diagram.
The useful sequence is not:
Customer wants sandbox → choose sandbox product.
It is:
Customer wants sandbox → why? → what must be isolated? → what are we protecting against? → what boundary is required? → architecture.
The first sentence gives you a category. The last question gives you a system design.
The customer often does not know the architecture either
This matters even more in AI because many enterprises are building these systems for the first time.
They know their constraints. They may know that security will not approve an unrestricted agent, that an auditor needs evidence, or that generated code cannot execute directly on production infrastructure.
What they often do not have is a settled architecture for satisfying those constraints.
How should the agent authenticate? Where should its code run? What can it call? Where does authorization happen? How do you trace a request across an agent, model, MCP server and tool? When does a human need to approve an action?
These are still moving targets.
That changes the FDE relationship. You are not simply implementing the architecture the customer already designed. Increasingly, you are helping them decide what that architecture should be.
Michael sees three questions appear repeatedly: security and governance, sandboxing, and observability.
The names sound like infrastructure categories, but underneath them are much simpler customer questions.
Can I control what this agent does?
Can I contain it when it executes something unsafe?
Can I explain what happened afterward?
Those are the production questions.
The products come later.
The FDE sits across the lifecycle
There is no single FDE job today. Some roles are almost entirely pre-sales. Others are deeply post-sales. Some engineers embed with one customer. Others work across several accounts as escalation points.
Michael prefers being close to the full lifecycle: discovery, demo, POC, implementation and eventually production.
There is a reason that matters.
Every handoff loses context.
The person doing discovery hears why the customer cares. The person building the POC learns which assumptions were wrong. The person deploying it finds the constraints nobody thought to mention. The person handling the production problem learns what actually mattered.
When those are completely separate conversations, an organization can collect enormous amounts of customer feedback while still misunderstanding the customer.
A strong FDE does more than forward requests back to product.
“Customer X wants feature Y” is not enough.
The better questions are: What are they actually trying to achieve? Is the request unique to this customer? Is the same constraint appearing elsewhere? Is the product missing a capability, or is the existing product being applied incorrectly?
The FDE compresses noisy customer requests into something product and engineering can act on.
That is a much higher-leverage job than simply implementing feature requests.
But the FDE cannot become the entire company
There is a trap here.
If the FDE understands the customer best, why not have them build everything?
Because eventually the FDE becomes the solutions engineer, architect, consultant, customer success manager, backend engineer, platform engineer and product manager at the same time.
AI coding tools make this even more tempting. An FDE can now prototype integrations, generate infrastructure, explore an unfamiliar codebase and build a custom demo dramatically faster than before.
That is valuable.
But the ability to prototype something does not mean the FDE should own it forever.
The FDE should help answer whether the capability should exist, how important it is, whether multiple customers need it and what an implementation could look like.
Product and engineering should then turn repeated customer signal into a reusable product. Otherwise the FDE becomes a permanent workaround layer around a product that never learns.
The business process should drive the POC
One of the most useful parts of my conversation with Michael was how he ties technical requirements back to business value.
Suppose a bank says it needs agent governance.
You could immediately show policy configuration, tracing, access controls and dashboards.
Or you could keep asking why.
Why governance? Because agents should not access unauthorized systems.
Why does that matter? Because security will not approve unrestricted agents.
What business process creates the constraint? Maybe auditors review the environment every four months and the organization needs to show exactly what its agents accessed and which controls were applied.
Now the requirement has changed. “We need governance” became:
We need provable control over agent activity because this workload needs to survive an audit.
That sentence can drive an architecture. It also tells you exactly what the POC should prove. Have an agent attempt a restricted action. Establish its identity. Apply policy. Block the action. Retain the trace. Show that the evidence can later be inspected.
The product did not change. The meaning of the product did.
That is often where an FDE creates the most value.
A canned demo is a clean wrong answer
This is where the FDE role starts to look surprisingly similar to harness engineering.
A production agent is rarely just a model. Around it we build tools, permissions, evaluation, retries, validation, memory, observability and human approval.
The harness exists because the model’s first output is not automatically trustworthy enough to affect the real world.
A canned demo has the same problem. It runs. The UI looks good. The agent calls something. The dashboard lights up. Everyone sees that the product works.
But it may be answering a question the customer never asked.
Michael strongly prefers building demos around the customer’s actual use case instead of relying on the same demo everywhere.
The deeper principle is worth keeping:
Design the demo backward from the constraint, not forward from the feature list.
Start with the audit, the security review, the production incident or whatever business process creates the need. From that, derive the production constraint.
From the constraint, derive the technical requirement. Then decide which architecture and product capability matter.
Only then build the demo.
A generic AI gateway demo might show routing, authentication, policies and traces. A bank-specific demo might show an agent attempting to access a regulated MCP tool under a user’s identity, getting blocked by policy, and leaving behind evidence that can be audited later.
Same technology.
One says, “Look what our product can do.”
The other says, “Look what happens to your problem when this product exists.”
AI can build the demo. You still own the explanation.
AI coding agents make custom demos much cheaper to build.
FDEs should use them. But they introduce a subtle problem: the artifact can arrive faster than the understanding.
The demo works. Then the customer asks why a component was configured a certain way, what happens when a dependency goes down, or why you chose one isolation mechanism instead of another.
If the answer is effectively “the coding agent generated it,” the customer has found the boundary of your expertise. AI changes how quickly an engineer can create something. It does not change who is accountable for understanding it.
The useful workflow is no longer simply build, understand, explain. It becomes generate, inspect, understand, challenge, modify, explain.
That extra verification step is what turns AI-generated work into engineering.
Trust is a production dependency
Michael kept coming back to trust.
Customers expect the FDE to be the subject-matter expert. That does not mean having every answer. In fact, “I don’t know, let me verify that with engineering” is often a stronger answer than confidently guessing.
The customer is not only asking you to write code. They are asking for judgment.
Should we design it this way? Is this secure enough? Is this production ready? Is this a real platform limitation? Are we making the wrong tradeoff? The quality of those answers eventually determines whether the customer asks for your opinion before making the next architecture decision.
Michael recommends The Trusted Advisor alongside going deep on the technology. That combination captures the FDE role unusually well. Technical depth gets you into the conversation.
Trust keeps you there.
Become the SME. Then become the people harness.
Near the end of our conversation, Michael gave two pieces of advice to engineers who want to become FDEs.
First, go deep on whatever technology you represent. Read the documentation. Read the code. Understand what sits below the demo.
Second, learn how to communicate, ask good questions and build trust.
During the conversation, that turned into a phrase I like:
Become the SME. Then become the people harness.
A good software harness supplies context, constraints, execution and verification around a model. A strong FDE does something similar around an enterprise deployment.
The customer provides intent. The FDE clarifies it, identifies the constraints, applies technical judgment, helps shape the architecture, tests it through the POC and carries what was learned back into the product.
The product alone is not enough. The customer’s first request is not enough.
The value is in reconciling the two. That may be the most interesting part of forward deployed engineering in AI. You stand at the point where new technology meets a messy production environment, and you decide which assumptions are allowed to pass through.
Try this before your next POC
Take one sentence from your current customer.
Maybe it is:
“We need governance.”
“We need agent sandboxing.”
“We need observability.”
“We need an AI gateway.”
Then answer these questions in order:
What did they ask for? Why do they want it? What happens if they do not have it? Which business process creates that constraint? What must technically be true for the constraint to be satisfied? What should the POC prove?
Do not stop at the feature.
For example, “We need governance” may eventually become:
Every agent action needs an attributable identity, policy evaluation and retained trace because the workload needs to survive a regulatory audit.
Now you have something you can architect.
If you cannot explain the business process that creates the requirement, you have probably found the discovery call you still need to have.
And it is much cheaper to have that conversation now than to discover at renewal that you solved the wrong problem beautifully.
The AI Runtime explores the engineering decisions behind production AI systems. FDE Talks is our interview series with the engineers deploying them.