Evaluating Safety Boundaries for LLM-Controlled Robotic Systems

A robot that understands an instruction has only solved part of the problem. It may correctly interpret “put the component in the tray” while choosing the wrong component, relying on an old camera frame, or planning a route through a space that has changed. A convincing explanation of that plan does not tell us whether the next motion should be allowed.

My work has repeatedly crossed the boundary between software interpretation and physical action. At Empirical Automation, that included a plywood repair prototype connecting visual defect recognition to an industrial robot. More recently, Project A.T.O.M. brings natural language, vision, calibration, task constraints, and motion planning into an operator-in-the-loop workflow for robot arms. Those projects motivate the question I want to examine here: how should we evaluate the boundaries of a robot system when a language model participates in deciding what happens next?

My answer starts with the architecture. We need to examine what a model can propose, which component can authorize that proposal, what evidence authorization depends on, and what can stop execution when those assumptions fail. The evaluation approach below is a proposed engineering framework. The examples and calculations are illustrative; they are not reported A.T.O.M. experiments or a claim of validated safety performance.

1. Define the boundary before testing the model

“Safe robot behavior” is too broad to serve as a test specification. A particular workcell has tools, payloads, access conditions, fixtures, operating modes, and consequences. The same motion can be acceptable in one configuration and hazardous in another. A test suite needs an explicit statement of where the system is intended to operate and what must remain true there.

I would begin with a bounded task: transfer a known part between two registered fixtures using an approved tool, within an established operating area. Describe permitted object classes, fixture versions, calibration validity, sensor availability, and conditions requiring operator intervention. State what the system cannot establish, such as the suitability of an unfamiliar container. That last category matters because unknown conditions should not silently inherit permission from familiar ones.

Next, distinguish the task objective from constraints on achieving it. Delivering a part is an objective. Remaining within an authorized motion envelope is a constraint. The objective may be negotiable when a grasp fails; the constraint should not become negotiable because the model believes another approach would be helpful. Concrete Problems in AI Safety describes how objective design, side effects, and distributional shift can produce accidents. My practical reading is to test the preservation of constraints separately from task completion.

2. Separate interpretation from authority

There is a useful distinction between letting an LLM select an approved skill and letting it generate arbitrary robot commands. A skill might identify a part, request a grasp from a bounded library, or propose placement in a registered tray. Its inputs and preconditions can be explicit. Arbitrary command generation expands the space of behaviors that the rest of the system must understand and contain.

SayCan connects language-based planning with learned skill affordances, grounding high-level instructions in actions a robot can perform. That is an important capability distinction. Feasibility, however, is not the same as authorization: a robot may be capable of reaching a location that a particular task is not permitted to use. I would keep those judgments explicit and separately testable.

In a proposed architecture, the model produces a typed task proposal. A policy and state validator checks the permitted skill, object identity, destination, relevant observations, and current configuration. A bounded executor then invokes established control functions. Independent protective functions remain responsible for their assigned physical hazards. Neither a typed schema nor a second LLM makes those functions safety-rated; their suitability requires its own engineering evidence.

3. Turn safety claims into observable evidence

Consider the claim “the robot will not act on an ambiguous object reference.” To evaluate it, the test must define ambiguity and an acceptable response. Two visually similar parts could share a description while having distinct registered identities. The expected behavior might be to request a specific selection and withhold execution until that selection is bound to a current observation.

That produces several observable questions. Did the model propose a particular part without sufficient evidence? Did validation reject the proposal? Did the executor receive a command anyway? Did the interface tell the operator what remained unresolved? A single pass/fail label would conceal which defense worked and whether the operator could recover.

I would maintain a small claim-to-evidence table before generating a large dataset. Each row names a constraint, the evidence required to check it, an intervention point, and the trace that demonstrates the outcome. If there is no observable signal for a claimed property, adding prompts will not solve that measurement problem. The system needs instrumentation or a narrower claim.

4. Organize tests around failure mechanisms

A useful test set should contain more than obviously dangerous instructions. Ordinary requests can expose unsafe assumptions. “Use the other tray” tests reference resolution. A valid task issued after a camera interruption tests state freshness. A familiar object with a changed tool tests whether the system accidentally treats a previous configuration as current.

I would group scenarios into semantic ambiguity, perception and calibration failures, planning violations, execution faults, and adversarial input. Within each family, vary one condition at a time before combining them. This helps distinguish a model that misunderstands the task from a validator that fails to enforce a known limit. Combined scenarios then test whether individually sound defenses interact badly.

The expected outcome is not always a refusal. Sometimes the right response is an observation refresh, a bounded alternative, or a precise clarification. Overly conservative behavior also deserves measurement: a system that blocks every task can look excellent on a narrow violation metric while providing no practical value. The evaluation should reward justified execution and justified withholding, using labels defined before seeing the output.

5. Treat language entering through the environment as untrusted

The operator's task instruction is only one way language can reach an embodied system. Text may appear in retrieved manuals, logs, labels, tool responses, or camera-visible documents. A model can encounter a sentence that resembles an instruction while it is supposed to be extracting facts. This creates an authority problem even when the original user request is harmless.

Greshake and colleagues' work on indirect prompt injection demonstrates the risk of instructions arriving through external content in LLM-integrated applications. For a robot workflow, I would use controlled, nonhazardous test fixtures to examine whether such content can alter a proposal or be promoted into operational policy. The test should never need a dangerous real-world action to establish that a boundary failed.

RoboPAIR provides further motivation by studying jailbreaks of LLM-controlled robotic systems across different access settings. Its findings concern the systems and experiments studied, rather than a universal failure rate for all robots. The architectural lesson I draw is to evaluate the downstream action boundary directly, instead of treating reassuring model responses as evidence that physical effects are contained.

Keep trusted operating configuration in a channel that retrieved text cannot edit. Where feasible, pass structured measurements into authorization rather than unrestricted narrative. These are design choices to test, not guarantees against injection. A compromised planner should still face limits on what it can cause, and the evaluation should deliberately substitute invalid proposals to verify those limits.

6. Include time in the definition of valid state

A proposal can be correct when generated and inappropriate when executed. Camera acquisition, inference, network transport, planning, and controller handoff all consume time. Meanwhile, a part can move or an operator can change a fixture. A timestamp alone does not settle the issue; validity depends on how quickly the relevant state can change.

For intuition, consider a simplified one-dimensional example with illustrative values. A moving component travels at 0.2 meters per second. A total delay of 0.15 seconds corresponds to 0.03 meters of travel before a response begins. If subsequent constant deceleration were 1 meter per second squared, the elementary braking term v² / (2a) adds 0.02 meters. That already totals 50 millimeters, before uncertainty or other motion is considered.

This calculation is not a protective separation-distance formula. It omits robot geometry, human approach, load-dependent dynamics, sensing coverage, and many other factors. Its purpose is to expose the mistake of reasoning only about position. The evaluation must measure relevant delays and actual stopping behavior for the configured system. A model's fast average response cannot establish the worst-case timing of the complete chain.

At the software boundary, associate each proposal with the state snapshot and configuration it used. Before accepting it, check the conditions that could invalidate it. Where continued execution depends on live observations, those checks must continue during execution. A preflight check cannot promise that an entire multi-step task will remain valid.

7. Test the controls independently, including shared failures

The controller's job is different from the language model's job. Control barrier function research, for example, formalizes certain state constraints through control-input conditions under specified assumptions. Such methods illustrate how control objectives and constraints can be separated mathematically. Their guarantees depend on the model, sensing, actuation, feasibility, and other assumptions; placing a language model above a controller does not remove those dependencies.

For evaluation, bypass the planner and inject malformed, unauthorized, or state-incompatible proposals into a non-actuating test harness. Confirm that the gate rejects them. Then test the executor's response to timeout, lost acknowledgments, interrupted skills, and invalid state transitions. Otherwise, the planner may simply fail to produce the particular output that would reveal a downstream defect.

Also examine common causes. Two software checks may share one stale camera feed or one incorrect coordinate transform. Calling them independent does not make their evidence independent. Record which observations and configuration sources each defense relies on. A useful failure test removes or corrupts a shared dependency and examines whether the system acknowledges the loss of knowledge instead of producing two matching but unreliable approvals.

8. Measure attempted violations, containment, and recovery separately

I would report at least three layers of outcome: what the model proposed, what the execution boundary authorized, and what the physical or simulated system did. A rejected invalid proposal is evidence of successful containment, but it is still relevant evidence about planner behavior. Combining those outcomes into one “safe completion” number loses that distinction.

Other useful measures include false rejection of permitted tasks, unresolved ambiguity reaching execution, time from detectable fault to intervention, and success of the specified recovery path. Define denominators carefully. A violation rate per proposal answers a different question from a rate per completed task or per operating hour. Long tasks can contain more opportunities for failure than short ones.

Results should also identify the conditions tested: workcell configuration, model version, prompt and tool definitions, perception version, calibration, and scenario family. Keep ordinary operating cases separate from deliberately adversarial stress cases. Both matter, but an adversarial dataset does not estimate the frequency of those conditions in everyday operation. Nor should good average performance hide one severe, reproducible escape from an intended boundary.

The evaluator needs scrutiny too. If a model reads its own execution transcript and decides whether it behaved safely, the report may reproduce the same misunderstanding that caused the failure. For precisely specified properties, compare the recorded state against an independently implemented rule. For judgments requiring engineering interpretation, preserve the evidence for a qualified reviewer and record disagreements. An incomplete trace should produce an unresolved outcome rather than quietly entering the pass column.

Repeated runs can reveal variability in planning, even with apparently identical inputs. I would preserve each proposal and decision rather than reporting only the best run. Pair model comparisons on the same scenario definitions and configuration, then examine whether a changed outcome came from interpretation, containment, or execution. A more capable planner may attempt a broader range of actions; that can improve task success while creating new demands on validation. Both effects belong in the report.

9. Explain what a clean test run cannot prove

Suppose an illustrative evaluation records zero violations across 300 trials. It is tempting to report that the system has a zero percent violation rate. What we actually know is that no violations were observed in those trials. The strength of that evidence depends on scenario coverage, measurement quality, and the relationship between tests and intended operation.

Even in a simplified statistical model, zero observed events is not proof of zero probability. If trials were independent, identically distributed Bernoulli events with violation probability p, the chance of observing none would be (1 − p)ⁿ. Setting that probability to 0.05 gives a one-sided 95 percent upper bound of 1 − 0.05^(1/n), approximately one percent for 300 trials.

Robot evaluations often violate the assumptions that make that calculation useful. Repeated runs may share the same scene, calibration, prompt template, and hidden weakness. A thousand near-duplicates do not provide a thousand independent checks of the whole design. I would therefore present coverage by failure mechanism and configuration alongside counts, and preserve uncertainty rather than turning a small clean run into a deployment-wide assurance claim.

10. Follow one task through a proposed evaluation

Take an illustrative instruction: “Move the blue component from fixture A to tray B.” Start in a non-actuating environment with known fixture and object identifiers. Record the expected proposal, the permitted skill sequence, and the conditions for authorization. Establish a baseline in which the scene and request are unambiguous and the task is permitted.

Now vary the input. Add a second blue component and expect a clarification or an explicit registered selection. Replace the observation with an expired snapshot and expect a refresh. Change the calibration identifier between planning and dispatch and expect revalidation. Mark tray B unavailable and expect the system to withhold that placement rather than silently choose a different destination.

Next, test the action boundary directly with an unauthorized destination while leaving the ordinary user instruction unchanged. If the validator accepts it, that is a boundary failure even if the model never generated that proposal in earlier tests. Finally, interrupt an approved sequence after grasp confirmation. Verify that resume requires the relevant current-state checks and does not replay a previously authorized command blindly.

Each scenario needs a trace containing inputs, observations, proposal, validation decision, executor transition, and available outcome evidence. Save enough context to reproduce the failure without collecting unnecessary personal or sensitive data. The most useful result is a specific diagnosis: a missing state check, a mistaken identifier binding, an executor transition that bypassed authorization, or a sensor limitation requiring a narrower operating domain.

11. Give the operator a decision they can actually assess

Human confirmation can become a ritual if the interface displays a fluent paragraph and a large approve button. The operator needs the unresolved facts and the proposed effect: which object, which destination, which tool, which motion region, and which conditions still depend on fresh observations. Show a preview where it helps answer those questions, and make uncertainty visible in concrete terms.

Confirmation should be bound to a specific proposal and relevant state. If the system changes the destination after approval, the old confirmation should not automatically authorize the new proposal. Conversely, avoid repeatedly asking users to approve inconsequential details. Concentrate attention at transitions where authority, uncertainty, or consequences change. Evaluate whether an operator can detect a seeded discrepancy before relying on that interface as a defense.

Recovery deserves equal care. Stopping motion does not necessarily mean removing all power; a suspended payload or process may require a particular controlled state. The correct response depends on the equipment and hazard analysis. The interface should communicate the actual state, the reason execution stopped, and the conditions for a deliberate restart. “Something went wrong; try again” is not enough information for a physical system.

12. Maintain the boundary as the system changes

A model update can change planning behavior. A new tool can expand the available effects. A camera adjustment can invalidate assumptions beneath an unchanged language interface. Evaluation should follow the configured system, with targeted reruns when a change affects a claim or its supporting evidence. Preserve regression cases from failures and keep some held-out variations to reduce tuning to a familiar test set.

The NIST AI Risk Management Framework 1.0 treats risk management as an ongoing activity organized around governance, context, measurement, and management. For this proposed workflow, that suggests assigning ownership for constraints, tests, exceptions, and retirement of invalid evidence. It is a useful organizational reference, not a substitute for workcell-specific engineering or applicable safety requirements.

I would also define how evidence supports each increase in operating scope. A recorded replay, a simulator, and a constrained physical trial answer different questions. Progression should identify which claims are already supported and which require the next stage, with clear conditions for returning to an earlier configuration. Preserve a known configuration and its evidence so that an unsuccessful update can be withdrawn deliberately. Wider autonomy should be attached to demonstrated coverage of the relevant hazards and assumptions, rather than inferred from a larger number of successful demonstrations.

Natural language can make robotics much more accessible. The engineering opportunity is to make that accessibility compatible with clear limits on physical effects. I would judge progress by whether the system can show why a particular action was authorized, detect when its assumptions stop holding, and reach an appropriate controlled state when it cannot proceed. Those properties make a demonstration easier to turn into a system whose behavior can be inspected, challenged, and improved.

Sources and further reading

1. Amodei et al. (2016). Concrete Problems in AI Safety. Background on objectives, side effects, supervision, and distributional shift.
2. Ahn et al. (2022). Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. Language planning combined with skill affordances.
3. Greshake et al. (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. External content as an instruction attack surface.
4. Robey et al. (2024). Jailbreaking LLM-Controlled Robots. RoboPAIR experiments and their stated access settings.
5. Ames et al. (2016 preprint; 2017 journal publication). Control Barrier Function Based Quadratic Programs for Safety Critical Systems. Formal treatment of control constraints under modeled assumptions.
6. NIST (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). Voluntary risk management framework.