Standardizing your agent stack without checking compatibility first is how you end up with agents that install cleanly on paper and fail the moment they touch a real client. The practical test is simple: verify packaging, permissions, installation, and cross-client behavior before you commit, and use an evidence-led checklist rather than whatever the vendor's marketing page claims.
That lesson cost Priya an entire sprint. She was a platform lead at a mid-sized logistics firm, eight engineers, and a mandate to consolidate their AI tooling before the next audit. The management-approved stack looked perfect on the slide deck: same runtime, same plugin registry, same declared support matrix. The vendor said the plugins were standard-compliant, portable, and ready to deploy. Priya had three days before the freeze, and she used two of them to find out that "declared support" and "actual interoperability" are different things.
The packaging problem nobody checks first
Priya started with the packaging, which felt like the boring part. She pulled the plugin archives from the registry, inspected their manifests, and compared them against the standard spec. That's when she found the first mismatch: two plugins declared compatibility with the same runtime version, but one shipped a dependency that the other explicitly excluded. The dependency resolver would either silently skip it or fail hard, depending on which client loaded it first.
She almost missed it. The vendor's documentation listed both plugins as "compatible" in the same table row. It took reading the raw manifest, not the marketing summary, to see the conflict. That is the difference between declared support and actual behavior, and it is why [evidence-led verification beats a vendor's compatibility matrix]( /blog/how-do-you-separate-declared-support-from-actual-interoperability-ae5263ac/ ). Priya's team had been one deployment away from discovering this in production, at 2am, with a customer demo the next morning.
Permissions: the silent failure mode
Packaging was the first trap. Permissions were the second. Priya's checklist asked each plugin to declare its required scopes, and then she tested whether the runtime actually enforced them. This is where the whole standardization nearly died.
One plugin, the one the whole team was most excited about, requested access to the client's file system and network sockets. Not maliciously, it was a data-fetching tool, but it asked for more than it needed. The runtime's permission model was supposed to restrict it. In practice, the plugin bypassed the sandbox when loaded through a specific cross-client path. It worked, up to a point. Then it quietly escalated its own privileges.
Priya caught it during a dry run. The plugin had been running for forty minutes, and a log line showed it accessing a directory it had no business touching. The outcome was genuinely in doubt for a full afternoon: either the plugin was broken and they lost a core component of the stack, or the permission model was broken and they lost their security posture. It turned out both were partially true, and the fix meant patching the runtime, not just the plugin.
Installation and cross-client behavior
The third check was installation, and the fourth was cross-client behavior. Priya's team tested the same plugin package across the different clients in their fleet: the primary runtime, the lightweight edge client, and the one they kept for legacy compatibility. The installation failed on the edge client with a dependency resolution error that the primary client never triggered. The error message was misleading, it pointed at a missing library that was actually present, and it took a second diagnostic pass to find the real cause: a version mismatch in the plugin's own build metadata.
That is the point of the exercise. A plugin that installs cleanly on your main client and breaks on the edge one will do exactly that in production, at the worst possible moment. The [failed dependency resolution that cost one founder her business]( /blog/the-failed-dependency-resolution-akiko-saw-and-what-it-cost-her-business-30d1c71c/ ) started the same way: an install that worked everywhere except one environment, ignored because the main path was fine.
What a compatibility checklist actually looks like
Priya now runs a four-part checklist before anything gets standardized. You can use the same one:
- Packaging. Inspect the raw manifest, not the marketing page. Check declared dependencies against the actual runtime version, and verify the plugin's build metadata matches what the client expects.
- Permissions. List every scope the plugin requests. Test whether the runtime actually enforces those limits across every client path, not just the happy path.
- Installation. Install the same package on every client in your fleet, including the legacy one. A failure on one environment is a failure of the stack.
- Cross-client behavior. Run the plugin through the same workflow on each client and compare the outputs. They should match. If they diverge, the plugin is not portable.
The stack went live a week later. It held. Priya's team shipped the consolidation on time, and the audit found no permission violations, which was the part management actually cared about. She keeps the checklist in the team's runbook now, and it has caught two more conflicts since, both of which would have surfaced in production within a month.
The rule is simple: verify before you standardize. Declared compatibility is a promise. Evidence is a fact.
Comments
No comments yet.