agent-tool-design
Design tool systems for AI agents — core/extended/deferred tiers, eval compute, batch tools, trust metadata. Use when building agent tools, reviewing tool architecture, or reducing tool count.
Install
npx skills add https://github.com/fortunto2/solo-factory/tree/main/templates/principles/agent-tool-design
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install fortunto2-solo-factory@llmmart
git clone https://github.com/fortunto2/solo-factory.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole fortunto2/solo-factory collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Tool Design Principles for AI Agents
Practical guide to designing tool systems for LLM-powered agents, distilled from building a PAC1 benchmark agent (Rust, 16 tools reduced to 12, tested across 7 models, 40+ tasks) and studying Codex CLI and Claude Code architectures.
See references/ for code patterns, comparison tables, and a quick checklist.
1. Tool Count Sweet Spot
Every tool in the schema is a token cost and a new failure mode. Models degrade on long tool lists.
Industry reference points:
| Agent | Core | Extended/Deferred | Total in Schema |
|---|---|---|---|
| Claude Code | 7 | 33 (deferred via ToolSearch) | 7-40 |
| Codex CLI | 7 | 0 | 7 |
| mini-SWE-agent | 1 (bash) | 0 | 1 |
| PAC1 agent | 14 | 8 deferred | 14 active + 8 deferred |
| SGR Python | 3-6 + reasoning | 0 | Union schema (structured output) |
See references/comparison.md for full architecture comparison.
Rules:
- Start with 7 core tools. Add only when you can measure round-trip savings.
- Track tool usage rate per task. Remove tools with <5% usage across benchmark.
- Every tool added must justify itself: "saves N round-trips per task" or "prevents failure mode X."
- Test with your weakest target model first -- if it can't handle the tool count, the design is wrong.
Anti-pattern: Adding a tool "just in case." We added mkdir, move_file, find -- usage was <3%. Disabled them. Zero regression.
2. Three-Tier Organization
CORE (always in schema)
Universal agent capabilities: observe (read, search, list, tree), act (write, delete, eval), report (answer, context).
Codex CLI uses exactly 7: shell, apply_patch, read_file, list_dir, grep_files, search_bm25, js_repl. Our PAC1 agent uses 9 core tools. Both converge on the same categories.
EXTENDED (batch operations)
Justified only when saving 3+ round-trips per task. Example: read_all saved 44 round-trips on our hardest task (48 to 4 tool calls).
DEFERRED (loaded on demand)
Claude Code's ToolSearch pattern: model sees only tool names. When it needs one, it calls ToolSearch("select:mkdir") to load the full JSON schema. Then it can call mkdir({path: "/new/dir"}).
This keeps the base schema small (7 tools) while providing access to 30+ tools. The key insight from the HitCC reverse-engineering: deferred tools are registered with shouldDefer===true, and the model must call ToolSearch before invoking them -- schema validation will fail otherwise.
3. The Eval/Compute Tool
Every agent needs a way to compute. Two patterns exist:
Shell access (Codex, Claude Code): Simple, powerful, dangerous. Codex sandboxes via containers and syscall filters. Claude Code uses permission hooks (PreToolUse/PostToolUse) with approval gating.
Embedded interpreter (API-only agents): When the agent operates via API with no shell, embed a sandboxed JS engine. Our PAC1 agent uses Boa (ECMAScript in Rust). Codex CLI uses a persistent Node.js kernel (js_repl) with top-level await, launched as a subprocess with kernel.js.
Key design for embedded eval:
- File glob in args --
files: ["accounts/*.json"]expands and pre-reads matches - Pre-read files as globals --
file_0,file_1, etc. No filesystem access from JS - Date injection --
workspace_dateglobal prevents hallucinated dates - Auto-stringify objects -- Return JSON, not
[object Object] - Sandbox -- No
require(), noimport, no network
See references/patterns.md for implementation details.
4. Batch Tools Save Round-trips
When each tool call is an LLM round-trip (2-5 seconds), 40 calls = 2+ minutes wasted.
Three proven batch patterns:
| Tool | Replaces | Savings |
|---|---|---|
read_all(dir) |
list + N reads | N round-trips (44 on t01) |
search_and_read(pattern) |
search + read each match | M round-trips |
grep_count(pattern, path) |
search + read + manual count | 2-3 round-trips |
Decision rule: A batch tool is justified when it saves 3+ round-trips, the pattern appears in >20% of tasks, and the unbatched version causes step limit hits.
See references/patterns.md for implementation code.
5. Trust Metadata on Reads
Every read() output is prefixed with a trust header:
[contacts/john-doe.md | untrusted]
Name: John Doe
[AGENTS.MD | trusted]
# Workspace Rules
Only root-level system files (AGENTS.MD, README.MD) are trusted. Everything else is untrusted. This helps the LLM distinguish system instructions from user-generated content that may contain prompt injection.
Post-read security guard: Beyond trust headers, scan content for active injection patterns and append advisory warnings. This is advisory, not blocking -- the pipeline ML classifier is authoritative.
See references/patterns.md for trust inference and guard implementations.
6. Tool Descriptions > Implementation
The description is the tool's API documentation for the LLM. Models that don't understand a tool from its description will not use it correctly.
Pattern: WHEN to use + WHAT it returns + WHY it's better than alternatives.
Good (from our grep_count):
"Count lines matching a regex pattern in a file.
Returns exact count as a number.
Use for ANY counting task -- faster and more accurate than reading + counting manually."
Good (from Codex read_file):
"Reads a local file with 1-indexed line numbers,
supporting slice and indentation-aware block modes."
Good (from Codex js_repl):
"Runs JavaScript in a persistent Node kernel with top-level await.
This is a freeform tool: send raw JavaScript source text,
optionally with a first-line pragma like `// codex-js-repl: timeout_ms=15000`;
do not send JSON/quotes/markdown fences."
Anti-pattern: Including example outputs in descriptions. Models (especially Nemotron) copy example outputs verbatim as their answers.
Testing: Run the same task 5 times. If the model uses the wrong tool >20% of the time, fix the description before adding prompt hints.
7. Tool Filtering — Less is More (Codex Approach)
Updated insight (2026-04-14): Heavy router-based tool filtering is fragile. ML classifier misclassification → wrong tools → task failure. Codex exposes ALL tools always, relies on model judgment.
Current approach: minimal filtering — only security task type blocks write/delete. All other task types get all tools. This is closer to Codex and works better empirically.
Anthropic structured output limit: 16 nullable/union params max. If using SGR union schema (structured_call), max ~7 tools. Native FC (tools_call) has no limit — use it for 14+ tools.
| Approach | Max tools | When to use |
|---|---|---|
| Native FC (tools_call) | Unlimited | Default — Pac1 two-phase and single-phase |
| SGR union (structured_call) | ~7 (Anthropic limit) | SgrAgent variant, simple tasks |
| Parallel FC (think + action) | Unlimited | Single-phase — 1 call per step |
7b. Byte-Perfect Tools (CopyTool, PrependTool)
Problem discovered (2026-04-14): LLMs cannot reproduce files >1KB verbatim. Even 1-byte difference (extra newline) fails harness validation. OCR/migration tasks scored 0% before this fix.
Solution: Tools that bypass LLM context for file content:
| Tool | What | When |
|---|---|---|
copy_file(src, dst) |
read → write through backend, content never enters LLM | NORA migration, file rewrite in place |
prepend_to_file(path, header) |
read body → prepend header → write. LLM generates only header (~400 bytes) | OCR: add YAML frontmatter to existing files |
Impact: OCR tasks t016, t018, t064, t091 went from 0→1.00 on Haiku.
Design rule: If task requires preserving existing file content, use byte-perfect tools. LLM generates only NEW content (frontmatter, metadata), never re-types body.
7c. Single-Phase Agent Architecture
Problem: Two-phase agent = 2 LLM calls per step = slow (116s/task avg on Haiku).
Solution: Parallel think+action in ONE tools_call. Model calls think() AND action tool together:
tools_call([think, search, read, write, delete, answer, ...])
→ Model returns: think({task_type, security, plan}) + search({pattern: "hello"})
= 1 LLM call, structured reasoning + action
Key findings:
- All models support parallel tool calls (Haiku, Sonnet, Opus, Nemotron)
completed = falsealways — let agent_loop execute answer() tool, then complete naturallyReasoningToolBuilderfrom sgr-agent creates think tool schema (extensible, not hardcoded)- Anthropic
parallel_tool_callsfield NOT supported via OpenRouter (use default which is parallel-enabled)
Performance: 2.5-3x faster, same score, 50% fewer tokens.
8. Hooks as Tool Augmentation
Hooks inject workflow guidance into tool output. The model follows tool output more reliably than system prompt instructions buried in 7K of context.
The pattern:
- Parse hooks from workspace rules (AGENTS.MD) at trial start
- Register in shared
HookRegistry(Arc - On every tool call, match against registered hooks
- Append matched messages to tool output
Why tool output, not system prompt? The model processes tool results with high attention (it just asked for this data). System prompt instructions 7K tokens back get less attention, especially on weaker models.
See references/patterns.md for hook implementation.
9. Testing Tools Without LLM
Every tool has logic that can break independently of the model. Unit test:
- Argument parsing edge cases -- missing optional fields, wrong types
- JSON auto-repair -- LLMs produce broken JSON (trailing commas, unquoted keys)
- Trust metadata -- root vs nested path inference
- Tool filtering -- router task type restrictions
- Auto-expand thresholds -- batch tool cutoffs
- Sandbox safety -- eval cannot access filesystem/network
- Guard content -- security scanning on read output
Do NOT test: "Does the model call the right tool?" (integration test), "Does output look good?" (subjective).
See references/patterns.md for test examples.
10. Middleware Pattern (sgr-agent-tools)
When you use sgr-agent-tools crate, extend tools via middleware wrappers — not forks:
struct MyReadTool<B: FileBackend> {
inner: sgr_agent_tools::ReadTool<B>, // base: trust metadata, line numbers
workflow: Arc<Mutex<WorkflowState>>, // your addition: phase tracking
}
impl<B: FileBackend> Tool for MyReadTool<B> {
fn name(&self) -> &str { self.inner.name() } // delegate
async fn execute(&self, args, ctx) {
let result = self.inner.execute(args, ctx).await?; // base
let output = security_scan(result.content); // middleware
self.workflow.post_action("read", &path); // middleware
Ok(ToolOutput::text(output))
}
}
When to use: pre/post hooks, project-specific annotations, policy guards, content scanning. When NOT to use: completely different schema → build custom tool instead.
Real-world split (PAC1 agent, 22 tools):
- 9 direct from sgr-agent-tools: List, Tree, ReadAll, MkDir, Move, Find, Eval, CopyTool, PrependTool
- 3 middleware: Read (+security scan), Write (+hooks/outbox), Delete (+workflow guards)
- 4 PAC1-only: Answer (harness submit), Context (workspace date), DateTool, LookupContactTool
- 3 local: Search (CRM annotations), ListSkills, GetSkill
- ML infra: sgr-agent-ml (OnnxEncoder, CentroidClassifier, KnnStore) — separate crate
11. Quick Reference
See references/checklist.md for the complete tool system design checklist.
References
references/patterns.md-- Code patterns + examples from Codex/Claude Code/PAC1references/comparison.md-- Architecture comparison tablereferences/checklist.md-- Quick-reference design checklistscripts/scaffold-tool.sh-- Generate Rust tool boilerplate- Codex CLI source:
codex-rs/core/src/tools/(7 tools: shell, apply_patch, read_file, list_dir, grep_files, search_bm25, js_repl) - Claude Code architecture: HitCC reverse-engineering docs (tool execution core, deferred tools, permission hooks)
- PAC1 agent:
agent-bit/src/tools.rs(16 tools),src/hooks.rs,src/workflow.rs - Boa JS engine: https://boajs.dev/
Files (solo-factory)
-
references
-
checklist.md 2.5 KB
# Tool System Design Checklist Quick reference for designing or reviewing an agent tool system. Score each item 0 (missing), 1 (partial), or 2 (done). --- ## Foundation (must-have) - [ ] **7 core tools max** -- observe (read, search, list), act (write, delete, eval), report (answer) - [ ] **Descriptions follow WHEN+WHAT+WHY** -- no example outputs in descriptions - [ ] **No example outputs in descriptions** -- models copy examples verbatim as answers - [ ] **`additionalProperties: false`** on all tool schemas -- prevent LLM from inventing fields - [ ] **Required fields minimal** -- only truly mandatory params in `required` array ## Efficiency - [ ] **Batch tools justified** -- each saves 3+ round-trips, used in >20% of tasks - [ ] **Search auto-expand** -- search results include file content when matches are few - [ ] **Tool usage tracking** -- remove tools with <5% usage across benchmark - [ ] **Deferred loading** for rarely-used tools -- model sees names only, loads schema on demand ## Safety - [ ] **Trust metadata on reads** -- `[path | trusted/untrusted]` header on all read outputs - [ ] **Post-read security guard** -- advisory warning for injection patterns - [ ] **Task-type filtering** -- router restricts tool set based on classifier, not model self-report - [ ] **Permanent restrictions** for dangerous combos -- `delete` task cannot access `write` - [ ] **Step-based unlock** for read-then-act -- step 0 read-only, step 1+ full toolkit ## Guidance - [ ] **Hooks from workspace rules** -- parse AGENTS.MD into HookRegistry, not hardcoded - [ ] **Hooks in tool output** -- append after normal result, not injected into system prompt - [ ] **Built-in hooks for universal patterns** -- e.g. "update seq.json after writing to outbox/" ## Testing - [ ] **Argument parsing unit tests** -- missing optionals, wrong types, edge cases - [ ] **JSON auto-repair tests** -- trailing commas, unquoted keys, missing brackets - [ ] **Trust inference tests** -- root vs nested, case sensitivity - [ ] **Tool filtering tests** -- each task type blocks/allows correct tools - [ ] **Sandbox safety tests** -- eval cannot access filesystem/network - [ ] **Weakest model test** -- tool system works on your least capable target model ## Architecture - [ ] **Tool handler separate from spec** -- definition in one file, implementation in another - [ ] **Read-only tools marked parallel-safe** -- framework can batch parallel calls - [ ] **Error messages actionable** -- tell the model what to do next, not just "error" -
comparison.md 4.3 KB
# Architecture Comparison Side-by-side analysis of tool architectures from Codex CLI, Claude Code, and PAC1 agent. --- ## Tool Count & Organization | Dimension | Codex CLI | Claude Code | PAC1 Agent | |-----------|-----------|-------------|------------| | Core tools | 7 | 7 | 9 | | Extended/batch | 0 | 0 | 3 (read_all, search_and_read, grep_count) | | Deferred | 0 | 33 (via ToolSearch) | 5 (mkdir, move, find, list_skills, get_skill) | | Total available | 7 | 40 | 17 (12 active) | | Schema size | Fixed | Dynamic (grows on demand) | Fixed per task type | --- ## Tool-by-Tool Mapping | Capability | Codex CLI | Claude Code | PAC1 Agent | |------------|-----------|-------------|------------| | **Execute** | `shell` / `shell_command` / `exec_command` | `Bash` | -- (API-only) | | **Compute** | `js_repl` (Node kernel, freeform grammar) | `Bash` (Python/Node via shell) | `eval` (Boa JS engine, sandboxed) | | **Read** | `read_file` (slice + indentation modes) | `Read` (offset + limit) | `read` (line numbers, range, trust metadata) | | **Write** | `apply_patch` (unified diff or freeform) | `Edit` (exact string replace) / `Write` | `write` (full or ranged overwrite + hooks) | | **Search** | `grep_files` (ripgrep wrapper, 30s timeout) | `Grep` (ripgrep, multiple output modes) | `search` (auto-expand, smart retry) | | **Semantic search** | `search_bm25` (BM25 + app connectors) | -- | `query_crm` (petgraph + ONNX embeddings) | | **List/tree** | `list_dir` (depth + pagination) | `Glob` (pattern matching) | `list` + `tree` | | **Delete** | via `shell` | via `Bash` | `delete` (policy-gated, batch paths) | | **Image** | `view_image` | `Read` (multimodal) | -- | | **Sub-agents** | `spawn_agent` / `send_input` | `Agent` (sub-agent tool) | -- | | **Deferred lookup** | -- | `ToolSearch` (name -> full schema) | `list_skills` / `get_skill` | | **Batch read** | -- | -- | `read_all` (dir -> all files) | | **Batch search** | -- | -- | `search_and_read` (search + auto-read) | | **Count** | -- | `Grep` (count mode) | `grep_count` | --- ## Architectural Patterns ### Permission Model | Agent | Model | How | |-------|-------|-----| | Codex CLI | Approval-based | `ExecApprovalRequest` checks `is_known_safe_command()`, unknown commands prompt user | | Claude Code | Hook-based | `PreToolUse` / `PostToolUse` hooks gate execution, permission merge | | PAC1 Agent | Policy + Workflow SM | `policy.rs` for file protection, `workflow.rs` for phase-based guards | ### Tool Output Schema | Agent | Pattern | |-------|---------| | Codex CLI | Only `exec_command` defines `output_schema` (JSON with exit_code, output, session_id). Others return plain text | | Claude Code | Tools return `tool_result` blocks. Some return `contextModifier` attachments | | PAC1 Agent | All tools return `ToolOutput::text()`. Trust metadata + hook messages appended | ### Error Handling | Agent | Pattern | |-------|---------| | Codex CLI | `FunctionCallError` type with structured error messages | | Claude Code | `is_error: true` flag on tool_result, with retry hints ("select tool via ToolSearch first") | | PAC1 Agent | `ToolError` enum, JSON auto-repair on parse failure, retry on empty response | --- ## Key Design Trade-offs ### Codex: Minimal + Freeform - 7 tools total, never changes - `js_repl` uses freeform grammar (Lark) instead of JSON -- unique approach for code-as-input - `apply_patch` also supports freeform (unified diff format) - No batch tools needed -- shell can do anything in one call - Pro: simple schema, reliable tool selection - Con: requires shell access (not suitable for API-only environments) ### Claude Code: Dynamic + Deferred - 7 core tools visible, 33+ available via ToolSearch - Schema grows dynamically as model discovers tools - MCP tools (`isMcp===true`) auto-deferred - Built-in tools can also be deferred via `shouldDefer===true` - Pro: scales to unlimited tools without schema bloat - Con: extra round-trip per new tool discovery ### PAC1: Task-Filtered + Augmented - 12 active tools, filtered per task type by ML classifier - Router restricts available tools based on task (delete = no write) - Trust metadata + hook injection augment tool output - Batch tools replace multi-step patterns - Pro: structural safety guarantees, round-trip efficiency - Con: classifier errors can lock out needed tools (mitigated by step-based unlock) -
patterns.md 16.5 KB
# Code Patterns & Examples Concrete implementations from Codex CLI, Claude Code (HitCC), and PAC1 agent. --- ## Codex CLI Tool Architecture Codex CLI (codex-rs) defines tools in `core/src/tools/spec.rs` using `ToolSpec::Function` with typed JSON schemas. Each tool has a separate handler in `core/src/tools/handlers/`. ### Tool Definition Pattern (Codex) ```rust // From codex-rs/core/src/tools/spec.rs fn create_read_file_tool() -> ToolSpec { let properties = BTreeMap::from([ ("file_path".to_string(), JsonSchema::String { description: Some("Absolute path to the file".to_string()), }), ("offset".to_string(), JsonSchema::Number { description: Some("The line number to start reading from. Must be 1 or greater.".to_string()), }), ("limit".to_string(), JsonSchema::Number { description: Some("The maximum number of lines to return.".to_string()), }), ("mode".to_string(), JsonSchema::String { description: Some( "Optional mode selector: \"slice\" for simple ranges (default) or \ \"indentation\" to expand around an anchor line.".to_string(), ), }), ]); ToolSpec::Function(ResponsesApiTool { name: "read_file".to_string(), description: "Reads a local file with 1-indexed line numbers, supporting slice and \ indentation-aware block modes.".to_string(), strict: false, parameters: JsonSchema::Object { properties, required: Some(vec!["file_path".to_string()]), additional_properties: Some(false.into()), }, output_schema: None, }) } ``` Key observations: - `strict: false` -- allows optional parameters without listing all in `required` - `additional_properties: Some(false.into())` -- prevents LLM from inventing fields - `output_schema: None` -- most tools don't define output schema (only `exec_command` does) - Description is one sentence -- action + key feature ### Shell Tool (Codex) ```rust // From codex-rs/core/src/tools/handlers/shell.rs pub struct ShellHandler; pub struct ShellCommandHandler { backend: ShellCommandBackend, // Classic or ZshFork } // Shell tool description (Linux/macOS) "Runs a shell command and returns its output. Always set the `workdir` param when using the shell_command function. Do not use `cd` unless absolutely necessary." ``` The shell handler is the most complex tool -- it manages subprocess execution, output streaming, timeout handling, and permission checks. Key lesson: even the most powerful tool has a simple description. ### Freeform Tool Pattern (js_repl) ```rust // From codex-rs/core/src/tools/spec.rs ToolSpec::Freeform(FreeformTool { name: "js_repl".to_string(), description: "Runs JavaScript in a persistent Node kernel with top-level await. \ This is a freeform tool: send raw JavaScript source text, optionally with \ a first-line pragma like `// codex-js-repl: timeout_ms=15000`; \ do not send JSON/quotes/markdown fences.".to_string(), format: FreeformToolFormat { r#type: "grammar".to_string(), syntax: "lark".to_string(), definition: JS_REPL_FREEFORM_GRAMMAR.to_string(), }, }) ``` `js_repl` is the only freeform tool -- it accepts raw JS source text instead of JSON arguments. The grammar validates syntax without requiring JSON wrapping. This is a unique pattern: when the tool input IS code, don't wrap it in JSON. ### Tool Registration (Codex) ```rust // Codex registers tools with parallel execution support builder.push_spec_with_parallel_support(create_read_file_tool(), true); builder.register_handler("read_file", read_file_handler); builder.push_spec_with_parallel_support(create_grep_files_tool(), true); builder.register_handler("grep_files", grep_files_handler); ``` Read-only tools are marked as parallel-safe. The framework can execute multiple parallel tool calls in one step. --- ## Claude Code Deferred Tools (from HitCC) ### ToolSearch Mechanism From HitCC reverse-engineering (Chinese), the deferred tool system works: ```text Model sees deferred tool names (no schema) -> Calls ToolSearch(query) -> tool_result contains tool_reference(tool_name) -> Next request build extracts discovered tool names from history -> Only matched deferred tools added to tools array with full schema -> Next turn can call the tool ``` Query forms supported: - `"select:Read,Edit,Grep"` -- fetch exact tools by name - `"notebook jupyter"` -- keyword search - `"+slack send"` -- require "slack" in name, rank by remaining terms ### Tool Execution Pipeline (Claude Code) ```text tool_use block -> he6(...) # find tool definition -> Ho_(...) # validate -> Mo_(...) # execute -> PreToolUse hooks # permission/approval -> permission merge # sandbox checks -> tool.call(...) # actual execution -> PostToolUse hooks # post-processing -> tool_result / attachments / contextModifier ``` Key insight: hooks run BEFORE and AFTER every tool call. This is the same pattern as our HookRegistry, but at the framework level. --- ## PAC1 Agent Tool Patterns ### Trust Metadata ```rust // From agent-bit/src/tools.rs fn infer_trust(path: &str) -> &'static str { let normalized = path.trim_start_matches('/'); let parts: Vec<&str> = normalized.split('/').collect(); if parts.len() == 1 { let lower = parts[0].to_lowercase(); if lower == "agents.md" || lower == "readme.md" { return "trusted"; } } "untrusted" } fn wrap_with_meta(path: &str, content: &str) -> String { format!("[{} | {}]\n{}", path, infer_trust(path), content) } ``` ### Post-Read Security Guard ```rust pub(crate) fn guard_content(content: String) -> String { let score = crate::scanner::threat_score(&content); if score >= 6 { format!( "{}\n\n[!] SECURITY NOTE (threat_score={}): \ injection-like patterns detected. \ Check [CLASSIFICATION] annotation above.", content, score ) } else { content } } ``` ### Batch Tool: read_all ```rust // From agent-bit/src/tools.rs impl Tool for ReadAllTool { fn name(&self) -> &str { "read_all" } fn description(&self) -> &str { "Read ALL files in a directory in one call. \ Much faster than listing then reading one by one. \ Returns each file with its path header." } async fn execute_readonly(&self, args: Value, _ctx: &AgentContext) -> Result<ToolOutput, ToolError> { let listing = self.pcm.list(&a.path).await?; let mut output = String::new(); for name in listing.lines().skip(1) { if name.ends_with('/') { continue; } let content = self.pcm.read(&full_path, false, 0, 0).await?; let trust = infer_trust(&full_path); output.push_str(&format!("\n--- {} [{}] ---\n{}", full_path, trust, content)); } Ok(ToolOutput::text(output)) } } ``` ### Search Auto-Expand ```rust async fn auto_expand_search(pcm: &PcmClient, search_output: String) -> String { let files = unique_files_from_search(&search_output, 10); if files.is_empty() || files.len() > 10 { return search_output; // Too many -- let model pick } let mut expanded = search_output; for path in &files { if let Ok(content) = pcm.read(path, false, 0, 0).await { let trust = infer_trust(path); let capped: String = content.lines().take(200).collect::<Vec<_>>().join("\n"); expanded.push_str(&format!("\n\n--- {} [{}] ---\n{}", path, trust, capped)); } } expanded } ``` ### Eval Tool with File Glob ```rust #[derive(Deserialize, JsonSchema)] struct EvalArgs { /// JavaScript code. Last expression = output. /// Globals: file_0..file_N, file_paths[], workspace_date code: String, /// File paths to pre-read. Supports glob: "projects/*/README.MD" #[serde(default)] files: Vec<String>, } ``` ### Tool Filtering by Task Type (Router) ```rust fn filter_tools_for_task(task_type: &str, step: u32, all_defs: Vec<ToolDef>) -> Vec<ToolDef> { match task_type { "security" => all_defs.into_iter() .filter(|t| matches!(t.name.as_str(), "read" | "read_all" | "search" | "search_and_read" | "grep_count" | "eval" | "find" | "list" | "answer" )).collect(), "delete" => all_defs.into_iter() .filter(|t| matches!(t.name.as_str(), "search" | "search_and_read" | "read" | "read_all" | "find" | "list" | "delete" | "answer" )).collect(), "analyze" if step == 0 => all_defs.into_iter() .filter(|t| matches!(t.name.as_str(), "read" | "read_all" | "search" | "find" | "list" | "tree" | "context" | "answer" )).collect(), _ => all_defs, } } ``` ### Hook Registry ```rust pub struct Hook { pub tool: String, pub path_contains: String, pub exclude: Vec<String>, pub message: String, } pub struct HookRegistry { hooks: Vec<Hook>, } impl HookRegistry { pub fn check(&self, tool_name: &str, path: &str) -> Vec<String> { self.hooks.iter() .filter(|h| h.tool == tool_name) .filter(|h| path.to_lowercase().contains(&h.path_contains)) .filter(|h| !h.exclude.iter().any(|ex| path.to_lowercase().contains(ex))) .map(|h| h.message.clone()) .collect() } } ``` ### Parsing Hooks from Workspace Rules ```rust pub fn from_agents_md(content: &str) -> HookRegistry { let mut registry = HookRegistry::new(); for line in content.lines() { let ll = line.to_lowercase(); // "when adding/writing to {path}, also {action}" if (ll.contains("when adding") || ll.contains("when writing")) && ll.contains("also") { if let Some(source_path) = extract_path_ref(line) { if let Some(action) = line.to_lowercase().split("also").nth(1) { registry.add(Hook { tool: "write".into(), path_contains: source_path.to_lowercase(), exclude: vec!["template".into()], message: format!("NEXT: {}", action.trim()), }); } } } // "keep files in {path} immutable" if ll.contains("immutable") || ll.contains("do not modify") { if let Some(path) = extract_path_ref(line) { registry.add(Hook { tool: "write".into(), path_contains: path.to_lowercase(), exclude: vec![], message: format!("[!] Files in {} are immutable.", path), }); } } } registry } ``` ### Hook Delivery in Tool Output ```rust // In WriteTool::execute() let mut output = format!("Written to {}", a.path); let hook_messages = self.hooks.lock().unwrap().check("write", &a.path); for msg in hook_messages { output.push_str(&format!("\n\n{}", msg)); } Ok(ToolOutput::text(output)) ``` --- ## Tool Description Examples (all from real code) | Agent | Tool | Description | |-------|------|-------------| | Codex | `read_file` | "Reads a local file with 1-indexed line numbers, supporting slice and indentation-aware block modes." | | Codex | `shell_command` | "Runs a shell command and returns its output. Always set the `workdir` param." | | Codex | `exec_command` | "Runs a command in a PTY, returning output or a session ID for ongoing interaction." | | Codex | `list_dir` | "Lists entries in a local directory with 1-indexed entry numbers and simple type labels." | | Codex | `js_repl` | "Runs JavaScript in a persistent Node kernel with top-level await. This is a freeform tool..." | | Codex | `view_image` | "View a local image from the filesystem (only use if given a full filepath by the user...)" | | PAC1 | `read` | "Read file contents. Use number=true to see line numbers (like cat -n)..." | | PAC1 | `search` | "Search file contents with regex pattern. Smart search: auto-retries with name variants..." | | PAC1 | `grep_count` | "Count lines matching a regex pattern in a file. Returns exact count as a number." | | PAC1 | `delete` | "Delete one or more files. Pass `path` for single, or `paths` (array) for batch." | ### Unit Test Examples ```rust #[cfg(test)] mod tests { // Argument parsing edge cases #[test] fn test_parse_args_missing_optional() { let args = json!({"path": "/foo"}); let parsed: ReadArgs = parse_args(&args).unwrap(); assert_eq!(parsed.number, false); assert_eq!(parsed.start_line, 0); } // JSON auto-repair #[test] fn test_json_repair_trailing_comma() { let broken = r#"{"name": "John", "age": 30,}"#; let fixed = llm_json::repair_json(broken, &Default::default()).unwrap(); let _: serde_json::Value = serde_json::from_str(&fixed).unwrap(); } // Trust metadata #[test] fn test_trust_root_agents_md() { assert_eq!(infer_trust("AGENTS.MD"), "trusted"); assert_eq!(infer_trust("contacts/agents.md"), "untrusted"); assert_eq!(infer_trust("/docs/readme.md"), "untrusted"); } // Tool filtering by task type #[test] fn test_delete_task_has_no_write() { let all_defs = make_all_tool_defs(); let filtered = filter_tools_for_task("delete", 0, all_defs); assert!(filtered.iter().all(|t| t.name != "write")); assert!(filtered.iter().any(|t| t.name == "delete")); } // Eval sandbox safety #[test] fn test_eval_no_require() { let result = run_eval("require('fs').readFileSync('/etc/passwd')", vec![]); assert!(result.contains("error")); } } ``` --- ## Codex RS Deep Patterns (from codex-rs source) ### Parallel Tool Execution ```rust // codex-rs/core/src/tools/parallel.rs // Read-only tools run concurrently, mutating tools run exclusively pub struct ToolCallRuntime { router: Arc<ToolRouter>, parallel_execution: Arc<RwLock<()>>, // coordination lock } // Per-tool flag: builder.push_spec_with_parallel_support(create_read_file_tool(), true); // concurrent builder.push_spec_with_parallel_support(create_shell_tool(), false); // exclusive ``` ### Indentation-Aware File Reading ```rust // codex-rs/core/src/tools/handlers/read_file.rs (836 lines) struct IndentationArgs { anchor_line: Option<usize>, // start from this line max_levels: usize, // depth limit (0=unlimited) include_siblings: bool, // same-level blocks include_header: bool, // comments above anchor max_lines: Option<usize>, // hard cap } // Smart: reads code block at anchor, follows indentation structure // Much better than raw offset/limit for code files ``` ### Sandbox Retry Escalation ```rust // codex-rs/core/src/tools/sandboxing.rs trait Sandboxable { fn sandbox_preference(&self) -> SandboxablePreference; // Auto|Always|Never fn escalate_on_failure(&self) -> bool; // retry without sandbox } // Flow: sandbox → fail → prompt user → retry without sandbox (cached approval) ``` ### JS REPL via Node.js Subprocess (NOT embedded engine) ```rust // codex-rs/core/src/tools/js_repl/mod.rs (3918 lines!) // - Spawns Node.js with --experimental-vm-modules // - JSON line protocol (stdin/stdout) // - VM context isolation per execution // - Previous module namespace imported into new cells (REPL semantics) // - Nested tool calls: JS can call back into Codex tools via async RPC // - Embeds meriyah parser for static binding analysis ``` ### Apply Patch (Diff-Based Edit) — saves tokens ```rust // codex-rs/core/src/tools/handlers/apply_patch.rs // Instead of rewriting full file (500 lines → 500 tokens): // Send diff (2 lines → 50 tokens) // Lark grammar parser for custom diff format // Fallback to shell: git apply ``` ### FileBackend Trait Pattern ```rust // Recommended for reusable tools: pub trait FileBackend: Send + Sync { async fn read(&self, path: &str) -> Result<String>; async fn write(&self, path: &str, content: &str) -> Result<()>; async fn search(&self, root: &str, pattern: &str) -> Result<String>; async fn list(&self, path: &str) -> Result<String>; } // Then tools are generic: pub struct ReadTool<B: FileBackend>(pub Arc<B>); pub struct SearchTool<B: FileBackend>(pub Arc<B>); // Each project provides backend: // agent-bit: FileBackend for PcmClient (BitGN API) // rc-cli: FileBackend for LocalFs (std::fs) ```
-
-
scripts
-
scaffold-tool.sh 2.1 KB
#!/bin/bash # Generate Rust tool boilerplate for an agent tool # # Usage: scaffold-tool.sh ToolName "Short description" # Example: scaffold-tool.sh GrepCount "Count lines matching a regex pattern" # # Output: prints Rust code to stdout. Redirect to file: # ./scaffold-tool.sh GrepCount "Count matching lines" > src/tools/grep_count.rs set -euo pipefail TOOL_NAME="${1:?Usage: scaffold-tool.sh ToolName \"description\"}" DESCRIPTION="${2:?Usage: scaffold-tool.sh ToolName \"description\"}" # Convert PascalCase to snake_case for function/module name SNAKE_NAME=$(echo "$TOOL_NAME" | sed 's/\([A-Z]\)/_\L\1/g' | sed 's/^_//') cat <<RUST use std::sync::Arc; use async_trait::async_trait; use schemars::JsonSchema; use serde::Deserialize; use serde_json::Value; use sgr_agent::{AgentContext, Tool, ToolError, ToolOutput}; use crate::pcm::PcmClient; pub struct ${TOOL_NAME}Tool(pub Arc<PcmClient>); #[derive(Deserialize, JsonSchema)] struct ${TOOL_NAME}Args { /// Primary argument (rename this) path: String, } #[async_trait] impl Tool for ${TOOL_NAME}Tool { fn name(&self) -> &str { "${SNAKE_NAME}" } fn description(&self) -> &str { "${DESCRIPTION}" } fn parameters_schema(&self) -> Value { schemars::schema_for!(${TOOL_NAME}Args).into() } fn is_readonly(&self) -> bool { true } async fn execute(&self, args: Value, _ctx: &AgentContext) -> Result<ToolOutput, ToolError> { let a: ${TOOL_NAME}Args = serde_json::from_value(args) .map_err(|e| ToolError::InvalidArgs(e.to_string()))?; // TODO: implement tool logic let result = format!("${TOOL_NAME} executed on {}", a.path); Ok(ToolOutput::text(result)) } } #[cfg(test)] mod tests { use super::*; use serde_json::json; #[test] fn test_parse_args() { let args = json!({"path": "/test"}); let parsed: ${TOOL_NAME}Args = serde_json::from_value(args).unwrap(); assert_eq!(parsed.path, "/test"); } #[test] fn test_name() { let tool = ${TOOL_NAME}Tool(Arc::new(PcmClient::mock())); assert_eq!(tool.name(), "${SNAKE_NAME}"); } } RUST
-
-
SKILL.md 12.7 KB
--- name: agent-tool-design description: Design tool systems for AI agents — core/extended/deferred tiers, eval compute, batch tools, trust metadata. Use when building agent tools, reviewing tool architecture, or reducing tool count. allowed-tools: Read, Write, Bash, Glob, Grep, Agent argument-hint: "[topic] e.g. 'eval tool', 'batch tools', 'tool count'" triggers: [tool-design, agent-tools, tool-architecture] priority: 10 keywords: [tool, agent, batch, deferred, trust, eval, hook, router] --- # Tool Design Principles for AI Agents Practical guide to designing tool systems for LLM-powered agents, distilled from building a PAC1 benchmark agent (Rust, 16 tools reduced to 12, tested across 7 models, 40+ tasks) and studying Codex CLI and Claude Code architectures. See `references/` for code patterns, comparison tables, and a quick checklist. --- ## 1. Tool Count Sweet Spot Every tool in the schema is a token cost and a new failure mode. Models degrade on long tool lists. **Industry reference points:** | Agent | Core | Extended/Deferred | Total in Schema | |-------|------|--------------------|-----------------| | Claude Code | 7 | 33 (deferred via ToolSearch) | 7-40 | | Codex CLI | 7 | 0 | 7 | | mini-SWE-agent | 1 (bash) | 0 | 1 | | PAC1 agent | 14 | 8 deferred | 14 active + 8 deferred | | SGR Python | 3-6 + reasoning | 0 | Union schema (structured output) | See `references/comparison.md` for full architecture comparison. **Rules:** - Start with 7 core tools. Add only when you can measure round-trip savings. - Track tool usage rate per task. Remove tools with <5% usage across benchmark. - Every tool added must justify itself: "saves N round-trips per task" or "prevents failure mode X." - Test with your weakest target model first -- if it can't handle the tool count, the design is wrong. **Anti-pattern:** Adding a tool "just in case." We added `mkdir`, `move_file`, `find` -- usage was <3%. Disabled them. Zero regression. --- ## 2. Three-Tier Organization ### CORE (always in schema) Universal agent capabilities: **observe** (read, search, list, tree), **act** (write, delete, eval), **report** (answer, context). Codex CLI uses exactly 7: `shell`, `apply_patch`, `read_file`, `list_dir`, `grep_files`, `search_bm25`, `js_repl`. Our PAC1 agent uses 9 core tools. Both converge on the same categories. ### EXTENDED (batch operations) Justified only when saving 3+ round-trips per task. Example: `read_all` saved 44 round-trips on our hardest task (48 to 4 tool calls). ### DEFERRED (loaded on demand) Claude Code's ToolSearch pattern: model sees only tool names. When it needs one, it calls `ToolSearch("select:mkdir")` to load the full JSON schema. Then it can call `mkdir({path: "/new/dir"})`. This keeps the base schema small (7 tools) while providing access to 30+ tools. The key insight from the HitCC reverse-engineering: deferred tools are registered with `shouldDefer===true`, and the model must call ToolSearch before invoking them -- schema validation will fail otherwise. --- ## 3. The Eval/Compute Tool Every agent needs a way to compute. Two patterns exist: **Shell access (Codex, Claude Code):** Simple, powerful, dangerous. Codex sandboxes via containers and syscall filters. Claude Code uses permission hooks (PreToolUse/PostToolUse) with approval gating. **Embedded interpreter (API-only agents):** When the agent operates via API with no shell, embed a sandboxed JS engine. Our PAC1 agent uses Boa (ECMAScript in Rust). Codex CLI uses a persistent Node.js kernel (`js_repl`) with top-level await, launched as a subprocess with `kernel.js`. Key design for embedded eval: 1. **File glob in args** -- `files: ["accounts/*.json"]` expands and pre-reads matches 2. **Pre-read files as globals** -- `file_0`, `file_1`, etc. No filesystem access from JS 3. **Date injection** -- `workspace_date` global prevents hallucinated dates 4. **Auto-stringify objects** -- Return JSON, not `[object Object]` 5. **Sandbox** -- No `require()`, no `import`, no network See `references/patterns.md` for implementation details. --- ## 4. Batch Tools Save Round-trips When each tool call is an LLM round-trip (2-5 seconds), 40 calls = 2+ minutes wasted. Three proven batch patterns: | Tool | Replaces | Savings | |------|----------|---------| | `read_all(dir)` | list + N reads | N round-trips (44 on t01) | | `search_and_read(pattern)` | search + read each match | M round-trips | | `grep_count(pattern, path)` | search + read + manual count | 2-3 round-trips | **Decision rule:** A batch tool is justified when it saves 3+ round-trips, the pattern appears in >20% of tasks, and the unbatched version causes step limit hits. See `references/patterns.md` for implementation code. --- ## 5. Trust Metadata on Reads Every `read()` output is prefixed with a trust header: ``` [contacts/john-doe.md | untrusted] Name: John Doe ``` ``` [AGENTS.MD | trusted] # Workspace Rules ``` Only root-level system files (AGENTS.MD, README.MD) are trusted. Everything else is untrusted. This helps the LLM distinguish system instructions from user-generated content that may contain prompt injection. **Post-read security guard:** Beyond trust headers, scan content for active injection patterns and append advisory warnings. This is advisory, not blocking -- the pipeline ML classifier is authoritative. See `references/patterns.md` for trust inference and guard implementations. --- ## 6. Tool Descriptions > Implementation The description is the tool's API documentation for the LLM. Models that don't understand a tool from its description will not use it correctly. **Pattern: WHEN to use + WHAT it returns + WHY it's better than alternatives.** Good (from our `grep_count`): ``` "Count lines matching a regex pattern in a file. Returns exact count as a number. Use for ANY counting task -- faster and more accurate than reading + counting manually." ``` Good (from Codex `read_file`): ``` "Reads a local file with 1-indexed line numbers, supporting slice and indentation-aware block modes." ``` Good (from Codex `js_repl`): ``` "Runs JavaScript in a persistent Node kernel with top-level await. This is a freeform tool: send raw JavaScript source text, optionally with a first-line pragma like `// codex-js-repl: timeout_ms=15000`; do not send JSON/quotes/markdown fences." ``` **Anti-pattern:** Including example outputs in descriptions. Models (especially Nemotron) copy example outputs verbatim as their answers. **Testing:** Run the same task 5 times. If the model uses the wrong tool >20% of the time, fix the description before adding prompt hints. --- ## 7. Tool Filtering — Less is More (Codex Approach) **Updated insight (2026-04-14):** Heavy router-based tool filtering is fragile. ML classifier misclassification → wrong tools → task failure. Codex exposes ALL tools always, relies on model judgment. Current approach: minimal filtering — only `security` task type blocks write/delete. All other task types get all tools. This is closer to Codex and works better empirically. **Anthropic structured output limit:** 16 nullable/union params max. If using SGR union schema (structured_call), max ~7 tools. Native FC (tools_call) has no limit — use it for 14+ tools. | Approach | Max tools | When to use | |----------|-----------|-------------| | Native FC (tools_call) | Unlimited | Default — Pac1 two-phase and single-phase | | SGR union (structured_call) | ~7 (Anthropic limit) | SgrAgent variant, simple tasks | | Parallel FC (think + action) | Unlimited | Single-phase — 1 call per step | --- ## 7b. Byte-Perfect Tools (CopyTool, PrependTool) **Problem discovered (2026-04-14):** LLMs cannot reproduce files >1KB verbatim. Even 1-byte difference (extra newline) fails harness validation. OCR/migration tasks scored 0% before this fix. **Solution:** Tools that bypass LLM context for file content: | Tool | What | When | |------|------|------| | `copy_file(src, dst)` | read → write through backend, content never enters LLM | NORA migration, file rewrite in place | | `prepend_to_file(path, header)` | read body → prepend header → write. LLM generates only header (~400 bytes) | OCR: add YAML frontmatter to existing files | **Impact:** OCR tasks t016, t018, t064, t091 went from 0→1.00 on Haiku. **Design rule:** If task requires preserving existing file content, use byte-perfect tools. LLM generates only NEW content (frontmatter, metadata), never re-types body. --- ## 7c. Single-Phase Agent Architecture **Problem:** Two-phase agent = 2 LLM calls per step = slow (116s/task avg on Haiku). **Solution:** Parallel think+action in ONE tools_call. Model calls `think()` AND action tool together: ``` tools_call([think, search, read, write, delete, answer, ...]) → Model returns: think({task_type, security, plan}) + search({pattern: "hello"}) = 1 LLM call, structured reasoning + action ``` **Key findings:** - All models support parallel tool calls (Haiku, Sonnet, Opus, Nemotron) - `completed = false` always — let agent_loop execute answer() tool, then complete naturally - `ReasoningToolBuilder` from sgr-agent creates think tool schema (extensible, not hardcoded) - Anthropic `parallel_tool_calls` field NOT supported via OpenRouter (use default which is parallel-enabled) **Performance:** 2.5-3x faster, same score, 50% fewer tokens. --- ## 8. Hooks as Tool Augmentation Hooks inject workflow guidance into tool output. The model follows tool output more reliably than system prompt instructions buried in 7K of context. **The pattern:** 1. Parse hooks from workspace rules (AGENTS.MD) at trial start 2. Register in shared `HookRegistry` (Arc<Mutex>) 3. On every tool call, match against registered hooks 4. Append matched messages to tool output **Why tool output, not system prompt?** The model processes tool results with high attention (it just asked for this data). System prompt instructions 7K tokens back get less attention, especially on weaker models. See `references/patterns.md` for hook implementation. --- ## 9. Testing Tools Without LLM Every tool has logic that can break independently of the model. Unit test: 1. **Argument parsing edge cases** -- missing optional fields, wrong types 2. **JSON auto-repair** -- LLMs produce broken JSON (trailing commas, unquoted keys) 3. **Trust metadata** -- root vs nested path inference 4. **Tool filtering** -- router task type restrictions 5. **Auto-expand thresholds** -- batch tool cutoffs 6. **Sandbox safety** -- eval cannot access filesystem/network 7. **Guard content** -- security scanning on read output **Do NOT test:** "Does the model call the right tool?" (integration test), "Does output look good?" (subjective). See `references/patterns.md` for test examples. --- ## 10. Middleware Pattern (sgr-agent-tools) When you use `sgr-agent-tools` crate, extend tools via **middleware wrappers** — not forks: ```rust struct MyReadTool<B: FileBackend> { inner: sgr_agent_tools::ReadTool<B>, // base: trust metadata, line numbers workflow: Arc<Mutex<WorkflowState>>, // your addition: phase tracking } impl<B: FileBackend> Tool for MyReadTool<B> { fn name(&self) -> &str { self.inner.name() } // delegate async fn execute(&self, args, ctx) { let result = self.inner.execute(args, ctx).await?; // base let output = security_scan(result.content); // middleware self.workflow.post_action("read", &path); // middleware Ok(ToolOutput::text(output)) } } ``` **When to use:** pre/post hooks, project-specific annotations, policy guards, content scanning. **When NOT to use:** completely different schema → build custom tool instead. Real-world split (PAC1 agent, 22 tools): - **9 direct** from sgr-agent-tools: List, Tree, ReadAll, MkDir, Move, Find, Eval, **CopyTool, PrependTool** - **3 middleware**: Read (+security scan), Write (+hooks/outbox), Delete (+workflow guards) - **4 PAC1-only**: Answer (harness submit), Context (workspace date), DateTool, LookupContactTool - **3 local**: Search (CRM annotations), ListSkills, GetSkill - **ML infra**: sgr-agent-ml (OnnxEncoder, CentroidClassifier, KnnStore) — separate crate --- ## 11. Quick Reference See `references/checklist.md` for the complete tool system design checklist. --- ## References - `references/patterns.md` -- Code patterns + examples from Codex/Claude Code/PAC1 - `references/comparison.md` -- Architecture comparison table - `references/checklist.md` -- Quick-reference design checklist - `scripts/scaffold-tool.sh` -- Generate Rust tool boilerplate - Codex CLI source: `codex-rs/core/src/tools/` (7 tools: shell, apply_patch, read_file, list_dir, grep_files, search_bm25, js_repl) - Claude Code architecture: HitCC reverse-engineering docs (tool execution core, deferred tools, permission hooks) - PAC1 agent: `agent-bit/src/tools.rs` (16 tools), `src/hooks.rs`, `src/workflow.rs` - Boa JS engine: https://boajs.dev/
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.