Containing Autonomous Agent Divergence: File System Overlays, Hardware Nested Paging, and Deterministic Syscall Traps

Autonomous AI agent architectures have shifted from simple string-in/string-out text generation to active execution loops. Modern multi-agent systems generate code on the fly, construct complex bash pipelines, write to relational databases, and query web services without human intervention.

However, live execution introduces a major failure mode: agent divergence. When an LLM-driven loop hallucinates a parameter, attempts an invalid file manipulation, or constructs an improper file path, the state of the host or workspace becomes polluted. Recent telemetry from BlogBuckett Intelligence highlights that when autonomous agents are granted direct state modification rights, over 60% of sequential tool execution chains accumulate unhandled side effects, leading to compromised file trees, memory degradation, or outright security containment breaches.

To build production-grade agent execution platforms, systems architects must establish strict, deterministic runtime constraints. This analysis explores how hardware-assisted nested paging, ephemeral Copy-on-Write (CoW) virtual file systems, and dynamic system call filtering prevent state divergence and secure host infrastructure against rogue tool execution.

The Root Mechanics of Agent State Pollution

Traditional microservice containers assume deterministic binaries: software is compiled, tested, packaged, and executed with predictable resource paths. Autonomous AI agents violate every one of these assumptions. An agent dynamically synthesizes code, attempts trial-and-error execution, and mutates environment variables based on stochastic model outputs.

State pollution occurs along three primary vectors:

1. Persistent File System Corruption: An agent executing sed, rm, or dynamic script generation alters shared workspace files during a failed iteration, leaving downstream agent steps with an inconsistent state.
2. Virtual Memory Boundary Overreach: Dynamic binaries generated inside the runtime attempt to allocate host-mapped memory buffers, leading to memory exhaust attacks or unintended address leakages across multi-tenant worker nodes.
3. Privilege Escalation via Syscall Composition: Even without root access inside a container, agents can leverage legal system calls (unshare, ptrace, io_uring_setup, process_vm_writev) in unexpected sequences to bypass namespace barriers.

Addressing these vectors requires isolating every tool invocation step into a completely disposable, non-polluting execution boundary.

Memory Boundary Defense: EPT/NPT vs. Software Fault Isolation

When evaluating memory safety for untrusted execution, platforms generally choose between two core paradigms: Hardware-Assisted Virtualization or Software Fault Isolation (SFI).

flowchart TD
    A["Agent Loop Orchestrator"] -->|Tool Invocation Request| B["Sandbox Runtime Manager"]
    B --> C{"Isolation Boundary Type"}
    C -->|High Security / Untrusted Code| D["Hardware MicroVM (KVM & EPT/NPT)"]
    C -->|Ultra-Low Latency / Pure Logic| E["Wasm Isolate (SFI & Linear Memory)"]
    D --> F["Ephemeral CoW VFS Overlay"]
    E --> F
    F --> G["Kernel LSM & eBPF Syscall Interceptor"]
    G -->|Permitted Operation| H["Host Physical Layer / Network"]
    G -->|Disallowed / State Mutation| I["Instant Trap & Ephemeral Rollback"]

Hardware-Assisted Nested Page Tables (EPT/NPT)

In hardware-assisted microVMs (such as KVM-backed hypervisors), host physical memory addresses are translated to guest physical addresses via Extended Page Tables (Intel EPT) or Nested Page Tables (AMD NPT).

When an agent-generated binary executes a pointer dereference inside a microVM, the CPU hardware performs a two-dimensional page walk:

$\text{Guest Virtual (GVA)} \longrightarrow \text{Guest Physical (GPA)} \longrightarrow \text{Host Physical (HPA)}$

If an agent process attempts to breach its allocated memory address space, the hardware MMU triggers an EPT Violation exit directly back to the hypervisor host loop. The hypervisor intercepts this exit in < 2 microseconds, halts the guest vCPU, and returns a sanitized runtime error to the agent orchestrator without exposing any host memory pages.

Software Fault Isolation (SFI) in WebAssembly Runtimes

For lightweight agent tools that execute pure logic or data parsing, WebAssembly runtimes (e.g., Wasmtime, V8 Isolates) rely on Software Fault Isolation. Instead of using CPU page tables, the runtime enforces memory boundaries through compiler-inserted boundary checks and sandbox-linear memory arrays.

Every memory access inside the Wasm module is constrained to an index offset off a base pointer:

$\text{Address} = \text{BasePtr} + (\text{Offset} \ \& \ \text{MemoryMask})$

This bitwise masking mathematically prevents the guest execution engine from reading or writing outside its allocated linear memory block. The trade-off is execution flexibility: SFI runtimes cannot natively execute arbitrary x86_64 or ARM64 Linux binaries created on the fly by an agent without specialized re-compilation or cross-compilation layers.

Ephemeral Copy-on-Write (CoW) VFS Overlays

Allowing an agent full write access to a workspace filesystem presents immediate risks. If an agent modifies a critical dependencies folder or deletes source code mid-loop, the workspace becomes unusable for subsequent reasoning steps.

To achieve non-destructive, high-throughput execution, modern agent engines utilize multi-layered Copy-on-Write (CoW) Virtual File Systems built on Linux overlayfs or block-level device mapper snapshots.

Ephemeral Overlay Geometry

The file system is divided into three distinct functional layers:

1. Lowerdir (Read-Only Base): Contains the immutable system root, global tool dependencies, interpreter runtimes, and baseline code repository.
2. Upperdir (Ephemeral CoW Layer): A temporary, memory-backed tmpfs layer where all write operations (file creations, mutations, log outputs) are directed during the tool's execution cycle.
3. Workdir (Internal Kernel State): Used by the kernel to prepare modifications before committing them to the upper layer.

When the agent attempts to alter a file in the workspace:

# Executing an unvetted script inside an isolated CoW session
mount -t overlay overlay \
  -o lowerdir=/var/sandboxes/base_root,/var/sandboxes/workspace_base \
  -o upperdir=/tmp/agent_sessions/sess_99a8f2/upper \
  -o workdir=/tmp/agent_sessions/sess_99a8f2/work \
  /mnt/agent_active_workspace

If the agent loop succeeds and passes validation tests, the files inside the upperdir are selectively merged back into the shared branch. If the agent loop fails, hallucinates, or pollutes the state, the orchestrator simply drops the upperdir mount point and unlinks the directory. The cost to completely purge mutated agent state and revert back to a clean baseline is reduced to a single kernel directory unlink (< 1 millisecond overhead).

Deterministic Syscall Filtering with eBPF and LSM

Isolating memory and file systems is insufficient if an agent process can abuse kernel interface boundaries. Containers share the host kernel; therefore, a unconfined system call can jeopardize host integrity.

While traditional container platforms rely on static seccomp-bpf profiles, autonomous agents require dynamic, contextual system call enforcement. For instance, an agent executing a git clone step requires socket creation privileges, but once code execution begins, networking capabilities should be immediately revoked.

Modern eBPF Security Enforcement

By coupling Linux Security Modules (LSM) with modern eBPF hooks, platform engineers can programmatically toggle syscall capabilities on a per-tool, per-second basis.

Below is an eBPF LSM probe program snippet illustrating how host engines dynamically intercept and block unauthorized process execution (execve) requests initiated by untrusted agent runtime processes:

#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 1024);
    __type(key, u32);   /* Sandbox Process Namespace ID */
    __type(value, u8);  /* 1 = Locked/Restricted, 0 = Privileged */
} agent_policy_map SEC(".maps");

SEC("lsm/bprm_check_security")
int BPF_PROG(restrict_agent_exec, struct linux_binprm *bprm)
{
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    u32 ns_id = bpf_get_current_ns_pid_tgid() >> 32;

    u8 *status = bpf_map_lookup_elem(&agent_policy_map, &ns_id);
    if (status && *status == 1) {
        /* Agent process is in restricted mode: deny execve of non-whitelisted binaries */
        bpf_printk("Blocked execution attempt in sandbox NS: %u, PID: %u", ns_id, pid);
        return -EPERM;
    }

    return 0;
}

char _license[] SEC("license") = "GPL";

Through this mechanism, the agent orchestrator dynamically populates agent_policy_map via eBPF host utilities. When an agent transitions from the "planning/fetch" phase to the "code execution" phase, the orchestrator sets the namespace status bit to 1. Any subsequent attempt by the agent process to invoke unapproved binaries triggers an immediate hardware level -EPERM error code, neutralizing prompt-injection-driven command execution.

Performance & Overhead Benchmark Analysis

Selecting the right sandboxing geometry requires evaluating the operational trade-offs between hardware virtualization, container isolation, and Wasm isolates. The table below presents real-world production metrics across execution styles:

Architectural Guidelines for Infrastructure Engineers

When designing production orchestrators for untrusted agent workflows, consider the following rules:

1. Decouple Logic from State Generation: Treat every agent invocation as entirely stateless. Force all persistent state writes to pass through an ephemeral CoW layer that requires explicit post-execution assertion checks before committing changes to main storage.
2. Apply Multi-Layered Enforcement: Use SFI (Wasm) for deterministic, high-throughput parsing and data transformation tasks where boot latency must remain sub-millisecond. Fall back to hardware microVMs (KVM-backed) whenever the agent generates or executes arbitrary native shell commands or binary payloads.
3. Implement Contextual Syscall State Machines: Do not rely on static seccomp profiles. Utilize eBPF-driven LSM enforcement to dynamically toggle network access and file write permissions depending on the specific task phase of the agent loop.

By enforcing rigid isolation at the memory, file system, and system call boundaries, engineering teams can safely harness autonomous AI agent execution loops at scale - eliminating state divergence and insulating host infrastructure from untrusted code execution.