Project 4: Custom Agent Factory
Project 4: Custom Agent Factory
Creating Specialized Personas: Build AI agents with precisely scoped capabilities
Project Metadata
| Attribute | Value |
|---|---|
| Difficulty | Level 2: Intermediate |
| Time Estimate | 1-2 Weeks (20-25 hours) |
| Primary Language | JSON/TypeScript |
| Alternative Languages | Python |
| Prerequisites | Projects 1-3, JSON, basic security concepts |
| Main Reference | “Clean Architecture” by Robert C. Martin |
Learning Objectives
By completing this project, you will:
- Master agent configuration schema - understand every field in an agent JSON file
- Design permission models - implement principle of least privilege for AI agents
- Create specialized personas - build agents optimized for specific tasks
- Understand tool restrictions - control what actions an agent can perform
- Build reusable agent libraries - create templates for common use cases
Deep Theoretical Foundation
Why Custom Agents Matter
The default Kiro agent is a generalist - it can read, write, execute shell commands, and access any file. This flexibility is powerful but dangerous. Custom agents let you create specialists with precisely scoped abilities:
┌─────────────────────────────────────────────────────────────────────┐
│ DEFAULT AGENT vs CUSTOM AGENT │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ DEFAULT AGENT (Generalist) │
│ ───────────────────────── │
│ ┌───────────────────────────────────────────────────────────────┐ │
│ │ CAPABILITIES: │ │
│ │ ✓ Read any file ✓ Write any file │ │
│ │ ✓ Execute any command ✓ Access any API │ │
│ │ ✓ Modify any config ✓ Delete anything │ │
│ │ │ │
│ │ RISK: "Delete all files in /src" │ │
│ │ Agent CAN comply (with user approval) │ │
│ └───────────────────────────────────────────────────────────────┘ │
│ │
│ CUSTOM AGENT (Security Auditor) │
│ ───────────────────────────── │
│ ┌───────────────────────────────────────────────────────────────┐ │
│ │ CAPABILITIES: │ │
│ │ ✓ Read source files ✗ Write any file │ │
│ │ ✗ Execute commands ✓ Access security docs │ │
│ │ ✗ Modify configs ✗ Delete anything │ │
│ │ │ │
│ │ SAFETY: "Delete all files" → "I cannot delete files. │ │
│ │ My role is read-only security analysis." │ │
│ └───────────────────────────────────────────────────────────────┘ │
│ │
│ WHY THIS MATTERS: │
│ • Defense in depth: Errors can't cause damage │
│ • Role clarity: Agent knows its purpose │
│ • Audit trail: Actions match expected capabilities │
│ • Trust building: Users know agent limits │
│ │
└─────────────────────────────────────────────────────────────────────┘
The Principle of Least Privilege
This fundamental security principle states: every component should have only the permissions necessary to perform its function.
┌─────────────────────────────────────────────────────────────────────┐
│ LEAST PRIVILEGE IN ACTION │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ TASK: Audit code for security vulnerabilities │
│ │
│ MINIMUM REQUIRED PERMISSIONS: │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ ✓ Read source code files │ │
│ │ ✓ Read configuration files │ │
│ │ ✓ Search codebase (grep) │ │
│ │ ✓ Generate reports (stdout) │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ DANGEROUS IF GRANTED: │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ ✗ Write files (could hide vulnerabilities) │ │
│ │ ✗ Shell access (could exfiltrate secrets) │ │
│ │ ✗ Network access (could leak findings) │ │
│ │ ✗ Git operations (could commit malicious code) │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ AGENT CONFIGURATION: │
│ { │
│ "allowedTools": ["read", "grep", "glob"], │
│ "deniedTools": ["write", "shell", "git"] │
│ } │
│ │
└─────────────────────────────────────────────────────────────────────┘
Agent Anatomy
Every custom agent is defined by a JSON configuration file:
┌─────────────────────────────────────────────────────────────────────┐
│ AGENT CONFIGURATION ANATOMY │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ .kiro/agents/security-auditor.json │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ { │ │
│ │ // IDENTITY │ │
│ │ "name": "security-auditor", │ │
│ │ "description": "Read-only security analysis agent", │ │
│ │ "model": "claude-opus-4.5", // Force specific model │ │
│ │ │ │
│ │ // PERSONA │ │
│ │ "systemPrompt": "You are a security expert focused on...", │ │
│ │ │ │
│ │ // PERMISSIONS │ │
│ │ "allowedTools": ["read", "grep", "glob"], │ │
│ │ "toolsSettings": { │ │
│ │ "read": { │ │
│ │ "allowedPaths": ["src/**", "config/**"] │ │
│ │ } │ │
│ │ }, │ │
│ │ │ │
│ │ // KNOWLEDGE │ │
│ │ "resources": [ │ │
│ │ "file://docs/security_guidelines.md", │ │
│ │ "file://OWASP_TOP_10.md" │ │
│ │ ], │ │
│ │ │ │
│ │ // INTEGRATIONS │ │
│ │ "mcpServers": ["sonarqube"] │ │
│ │ } │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
Tool Permission Hierarchy
Kiro tools follow a hierarchy of restrictions:
┌─────────────────────────────────────────────────────────────────────┐
│ TOOL PERMISSION HIERARCHY │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ LEVEL 0: UNRESTRICTED │
│ ───────────────────── │
│ No allowedTools specified = all tools available │
│ │
│ │ │
│ ▼ │
│ │
│ LEVEL 1: TOOL ALLOWLIST │
│ ─────────────────────── │
│ "allowedTools": ["read", "grep", "glob"] │
│ Only listed tools are available │
│ │
│ │ │
│ ▼ │
│ │
│ LEVEL 2: TOOL DENYLIST (within allowed) │
│ ─────────────────────────────────────── │
│ "toolsSettings": { │
│ "shell": { │
│ "deniedCommands": ["rm", "sudo", "curl"] │
│ } │
│ } │
│ Specific commands blocked within allowed tools │
│ │
│ │ │
│ ▼ │
│ │
│ LEVEL 3: PATH RESTRICTIONS │
│ ───────────────────────── │
│ "toolsSettings": { │
│ "read": { │
│ "allowedPaths": ["src/**"], │
│ "deniedPaths": ["**/*.env", "**/secrets/**"] │
│ } │
│ } │
│ File system access scoped to specific paths │
│ │
│ │ │
│ ▼ │
│ │
│ LEVEL 4: APPROVAL REQUIRED │
│ ───────────────────────── │
│ Some actions always require user confirmation │
│ (destructive operations, external network calls) │
│ │
└─────────────────────────────────────────────────────────────────────┘
Agent Archetypes
Common patterns for specialized agents:
┌─────────────────────────────────────────────────────────────────────┐
│ AGENT ARCHETYPES │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ AUDITOR │ │ REVIEWER │ │ IMPLEMENTER │ │
│ │ │ │ │ │ │ │
│ │ Read: ✓ │ │ Read: ✓ │ │ Read: ✓ │ │
│ │ Write: ✗ │ │ Write: ✗ │ │ Write: ✓ │ │
│ │ Shell: ✗ │ │ Shell: ✗ │ │ Shell: ✓ │ │
│ │ │ │ │ │ │ │
│ │ Purpose: │ │ Purpose: │ │ Purpose: │ │
│ │ Find issues │ │ Review changes │ │ Make changes │ │
│ │ without fixing │ │ suggest fixes │ │ run tests │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ RESEARCHER │ │ DEVOPS │ │ API DESIGNER │ │
│ │ │ │ │ │ │ │
│ │ Read: ✓ │ │ Read: ✓ │ │ Read: ✓ │ │
│ │ Write: ✗ │ │ Write: ✓ │ │ Write: ✓ │ │
│ │ Web: ✓ │ │ Shell: ✓ │ │ Shell: ✗ │ │
│ │ │ │ (scoped) │ │ │ │
│ │ Purpose: │ │ Purpose: │ │ Purpose: │ │
│ │ Gather info │ │ Deploy, config │ │ Design APIs │ │
│ │ from docs │ │ approved ops │ │ write specs │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
Real-World Analogy: Hospital Staff
Think of custom agents like hospital staff with different roles:
| Role | Access | Cannot Do |
|---|---|---|
| Radiologist (Auditor) | View all scans | Prescribe treatment |
| Nurse (Reviewer) | View records, suggest actions | Perform surgery |
| Surgeon (Implementer) | Full OR access | Access pharmacy |
| Pharmacist (DevOps) | Dispense medications | Access medical records |
Each role has precisely scoped access to protect patients (your code).
Complete Project Specification
What You Are Building
An Agent Factory Toolkit that:
- Generates Agent Configs: Interactive CLI to create new agents
- Validates Configurations: Ensures agent configs are valid and safe
- Tests Agent Boundaries: Verifies agents can’t exceed permissions
- Provides Pre-built Library: Collection of ready-to-use agents
- Documents Agent Capabilities: Auto-generates agent documentation
Architecture Overview
┌─────────────────────────────────────────────────────────────────────┐
│ AGENT FACTORY ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ CLI Interface │ │
│ │ ┌───────────────────────────────────────────────────────┐ │ │
│ │ │ agent-factory [command] │ │ │
│ │ │ │ │ │
│ │ │ Commands: │ │ │
│ │ │ create Interactive agent creation wizard │ │ │
│ │ │ validate Check agent config for errors │ │ │
│ │ │ test Verify permission boundaries │ │ │
│ │ │ list Show all agents in project │ │ │
│ │ │ template Generate from pre-built archetype │ │ │
│ │ │ docs Generate agent documentation │ │ │
│ │ └───────────────────────────────────────────────────────┘ │ │
│ └──────────────────────────┬──────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Agent Generator │ │
│ │ ┌───────────────────────────────────────────────────────┐ │ │
│ │ │ TemplateEngine │ │ │
│ │ │ • Load archetype templates │ │ │
│ │ │ • Apply user customizations │ │ │
│ │ │ • Merge with project defaults │ │ │
│ │ └───────────────────────────────────────────────────────┘ │ │
│ │ ┌───────────────────────────────────────────────────────┐ │ │
│ │ │ Validator │ │ │
│ │ │ • JSON schema validation │ │ │
│ │ │ • Permission conflict detection │ │ │
│ │ │ • Path pattern validation │ │ │
│ │ └───────────────────────────────────────────────────────┘ │ │
│ └──────────────────────────┬──────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Permission Tester │ │
│ │ ┌───────────────────────────────────────────────────────┐ │ │
│ │ │ • Spawn agent with config │ │ │
│ │ │ • Attempt denied operations │ │ │
│ │ │ • Verify rejection messages │ │ │
│ │ │ • Report boundary violations │ │ │
│ │ └───────────────────────────────────────────────────────┘ │ │
│ └──────────────────────────┬──────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Agent Library │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ sec-auditor │ │code-reviewer│ │ devops-eng │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ api-designer│ │test-engineer│ │doc-writer │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
Expected Deliverables
agent-factory/
├── src/
│ ├── index.ts # CLI entry point
│ ├── commands/
│ │ ├── create.ts # Interactive creation
│ │ ├── validate.ts # Config validation
│ │ ├── test.ts # Boundary testing
│ │ ├── list.ts # List agents
│ │ └── template.ts # Generate from archetype
│ ├── generator/
│ │ ├── TemplateEngine.ts # Template processing
│ │ └── Validator.ts # Schema validation
│ ├── tester/
│ │ └── PermissionTester.ts # Boundary testing
│ └── templates/
│ ├── security-auditor.json
│ ├── code-reviewer.json
│ ├── devops-engineer.json
│ └── api-designer.json
├── schemas/
│ └── agent-config.schema.json
├── tests/
│ ├── validator.test.ts
│ └── templates.test.ts
├── package.json
└── README.md
# Also creates in target project:
.kiro/agents/
├── sec-auditor.json
├── code-reviewer.json
└── devops-engineer.json
Solution Architecture
Agent Configuration Schema
// schemas/agent-config.schema.json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"required": ["name"],
"properties": {
"name": {
"type": "string",
"pattern": "^[a-z][a-z0-9-]*$",
"description": "Agent identifier (lowercase, alphanumeric, hyphens)"
},
"description": {
"type": "string",
"description": "Human-readable description of agent purpose"
},
"model": {
"type": "string",
"enum": ["claude-haiku-4.5", "claude-sonnet-4.5", "claude-opus-4.5", "auto"],
"default": "auto"
},
"systemPrompt": {
"type": "string",
"description": "Custom system prompt for agent persona"
},
"allowedTools": {
"type": "array",
"items": {
"type": "string",
"enum": ["read", "write", "glob", "grep", "shell", "git", "aws", "mcp"]
}
},
"toolsSettings": {
"type": "object",
"properties": {
"shell": {
"type": "object",
"properties": {
"allowedCommands": { "type": "array", "items": { "type": "string" } },
"deniedCommands": { "type": "array", "items": { "type": "string" } }
}
},
"read": {
"type": "object",
"properties": {
"allowedPaths": { "type": "array", "items": { "type": "string" } },
"deniedPaths": { "type": "array", "items": { "type": "string" } }
}
},
"write": {
"type": "object",
"properties": {
"allowedPaths": { "type": "array", "items": { "type": "string" } },
"deniedPaths": { "type": "array", "items": { "type": "string" } }
}
}
}
},
"resources": {
"type": "array",
"items": { "type": "string" },
"description": "file:// URIs to load on agent start"
},
"mcpServers": {
"type": "array",
"items": { "type": "string" },
"description": "Names of MCP servers to connect"
}
}
}
Pre-built Agent Templates
┌─────────────────────────────────────────────────────────────────────┐
│ SECURITY AUDITOR TEMPLATE │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Purpose: Read-only security analysis of codebase │
│ │
│ { │
│ "name": "sec-auditor", │
│ "description": "Security vulnerability scanner (read-only)", │
│ "model": "claude-opus-4.5", │
│ "systemPrompt": "You are a security expert. Your role is to │
│ identify vulnerabilities, NOT to fix them. You cannot modify │
│ files or run commands. Report findings in a structured format. │
│ Focus on: SQL injection, XSS, auth bypasses, secrets exposure.",│
│ │
│ "allowedTools": ["read", "grep", "glob"], │
│ │
│ "toolsSettings": { │
│ "read": { │
│ "allowedPaths": ["**/*.ts", "**/*.js", "**/*.py", "**/*.go"],│
│ "deniedPaths": ["**/node_modules/**", "**/.git/**"] │
│ } │
│ }, │
│ │
│ "resources": [ │
│ "file://docs/security_guidelines.md" │
│ ] │
│ } │
│ │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ CODE REVIEWER TEMPLATE │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Purpose: Review code changes, suggest improvements (no writes) │
│ │
│ { │
│ "name": "code-reviewer", │
│ "description": "Code review specialist (suggest, don't modify)", │
│ "model": "claude-sonnet-4.5", │
│ "systemPrompt": "You are a senior code reviewer. Analyze code │
│ for: readability, performance, security, test coverage. │
│ Provide specific, actionable feedback. Use diff format for │
│ suggestions. Never modify files directly.", │
│ │
│ "allowedTools": ["read", "grep", "glob", "git"], │
│ │
│ "toolsSettings": { │
│ "git": { │
│ "allowedCommands": ["git diff", "git log", "git show"] │
│ } │
│ }, │
│ │
│ "resources": [ │
│ "file://docs/code_style_guide.md", │
│ "file://.kiro/steering/tech.md" │
│ ] │
│ } │
│ │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ DEVOPS ENGINEER TEMPLATE │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Purpose: Execute approved deployment and config operations │
│ │
│ { │
│ "name": "devops-engineer", │
│ "description": "DevOps automation (scoped shell access)", │
│ "model": "claude-sonnet-4.5", │
│ "systemPrompt": "You are a DevOps engineer. You can run │
│ approved commands for deployment, testing, and configuration. │
│ Always explain what you're about to do before running. │
│ Never run commands that could damage production.", │
│ │
│ "allowedTools": ["read", "write", "shell", "aws"], │
│ │
│ "toolsSettings": { │
│ "shell": { │
│ "allowedCommands": [ │
│ "npm test", "npm run build", "npm run lint", │
│ "docker build", "docker-compose up", │
│ "terraform plan", "terraform apply" │
│ ], │
│ "deniedCommands": [ │
│ "rm -rf", "sudo", "> /dev/", "curl | bash" │
│ ] │
│ }, │
│ "write": { │
│ "allowedPaths": ["docker-compose.yml", "Dockerfile", │
│ ".github/workflows/**", "terraform/**"] │
│ } │
│ } │
│ } │
│ │
└─────────────────────────────────────────────────────────────────────┘
Permission Validation Flow
┌─────────────────────────────────────────────────────────────────────┐
│ PERMISSION VALIDATION FLOW │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Agent attempts: write("src/config.ts", content) │
│ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ STEP 1: Check allowedTools │ │
│ │ │ │
│ │ Is "write" in allowedTools? │ │
│ │ • Yes → Continue to Step 2 │ │
│ │ • No → REJECT: "Tool 'write' not allowed for this agent" │ │
│ └────────────────────────┬────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ STEP 2: Check toolsSettings.write.deniedPaths │ │
│ │ │ │
│ │ Does "src/config.ts" match any deniedPaths pattern? │ │
│ │ • Yes → REJECT: "Path denied by security policy" │ │
│ │ • No → Continue to Step 3 │ │
│ └────────────────────────┬────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ STEP 3: Check toolsSettings.write.allowedPaths │ │
│ │ │ │
│ │ Does "src/config.ts" match any allowedPaths pattern? │ │
│ │ • Yes → Continue to Step 4 │ │
│ │ • No allowedPaths defined → Continue to Step 4 │ │
│ │ • Has allowedPaths but no match → REJECT: "Path not in │ │
│ │ allowed list" │ │
│ └────────────────────────┬────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ STEP 4: User Approval (if required) │ │
│ │ │ │
│ │ Prompt user: "Agent wants to write to src/config.ts. Allow?" │ │
│ │ • User approves → EXECUTE │ │
│ │ • User denies → REJECT: "User denied operation" │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
Phased Implementation Guide
Phase 1: Schema and Validation (4-5 hours)
Goal: Create robust agent config validation
What to Build:
- JSON Schema for agent configs
- Validator class with clear error messages
- CLI
validatecommand
Hint 1: Use Ajv for JSON Schema validation:
import Ajv from 'ajv';
import addFormats from 'ajv-formats';
const ajv = new Ajv({ allErrors: true });
addFormats(ajv);
const validate = ajv.compile(agentSchema);
export function validateAgentConfig(config: unknown): ValidationResult {
const valid = validate(config);
if (!valid) {
return {
valid: false,
errors: validate.errors.map(e => ({
path: e.instancePath,
message: e.message
}))
};
}
return { valid: true, errors: [] };
}
Hint 2: Add semantic validation beyond schema:
function semanticValidation(config: AgentConfig): string[] {
const errors: string[] = [];
// Check for permission conflicts
if (config.allowedTools?.includes('write') && !config.toolsSettings?.write?.allowedPaths) {
errors.push('Warning: write allowed without path restrictions');
}
// Check that resources exist
for (const resource of config.resources || []) {
if (resource.startsWith('file://')) {
const path = resource.replace('file://', '');
if (!fs.existsSync(path)) {
errors.push(`Resource not found: ${path}`);
}
}
}
return errors;
}
Validation Checkpoint: Running agent-factory validate config.json shows clear errors or success.
Phase 2: Template System (4-5 hours)
Goal: Create agent templates and generation wizard
What to Build:
- Pre-built agent templates
- Interactive creation wizard
- Template customization
Hint 1: Use Inquirer for interactive prompts:
import inquirer from 'inquirer';
async function createAgentWizard(): Promise<AgentConfig> {
const answers = await inquirer.prompt([
{
type: 'list',
name: 'archetype',
message: 'Select agent archetype:',
choices: [
{ name: 'Security Auditor (read-only)', value: 'security-auditor' },
{ name: 'Code Reviewer (read + git)', value: 'code-reviewer' },
{ name: 'DevOps Engineer (scoped shell)', value: 'devops-engineer' },
{ name: 'Custom (start from scratch)', value: 'custom' }
]
},
{
type: 'input',
name: 'name',
message: 'Agent name:',
validate: (input) => /^[a-z][a-z0-9-]*$/.test(input) || 'Use lowercase letters, numbers, hyphens'
},
{
type: 'editor',
name: 'systemPrompt',
message: 'Custom system prompt (optional):',
when: (answers) => answers.archetype === 'custom'
}
]);
return generateFromAnswers(answers);
}
Hint 2: Merge template with customizations:
function mergeWithTemplate(
template: AgentConfig,
customizations: Partial<AgentConfig>
): AgentConfig {
return {
...template,
...customizations,
toolsSettings: {
...template.toolsSettings,
...customizations.toolsSettings
},
resources: [
...(template.resources || []),
...(customizations.resources || [])
]
};
}
Validation Checkpoint: Running agent-factory create guides through wizard and creates valid config.
Phase 3: Permission Testing (4-5 hours)
Goal: Build system to verify agent boundaries
What to Build:
- Test harness that spawns agent
- Attempts denied operations
- Reports results
Hint 1: Create test cases for each restriction:
interface PermissionTest {
name: string;
tool: string;
input: object;
expectation: 'allow' | 'deny';
}
const testsForAgent = (config: AgentConfig): PermissionTest[] => {
const tests: PermissionTest[] = [];
// If write is not allowed, test that writes fail
if (!config.allowedTools?.includes('write')) {
tests.push({
name: 'Cannot write files',
tool: 'write',
input: { path: 'test.txt', content: 'test' },
expectation: 'deny'
});
}
// If shell has denied commands, test they fail
const deniedCommands = config.toolsSettings?.shell?.deniedCommands || [];
for (const cmd of deniedCommands) {
tests.push({
name: `Cannot run: ${cmd}`,
tool: 'shell',
input: { command: cmd },
expectation: 'deny'
});
}
return tests;
};
Hint 2: Run tests against actual Kiro agent:
async function runPermissionTests(
agentName: string,
tests: PermissionTest[]
): Promise<TestResult[]> {
const results: TestResult[] = [];
for (const test of tests) {
// Start agent session
const session = await startAgentSession(agentName);
try {
// Attempt the operation
const prompt = buildPromptForTest(test);
const response = await session.send(prompt);
// Check if operation was allowed or denied
const wasAllowed = !response.includes('not allowed') &&
!response.includes('cannot') &&
!response.includes('denied');
const passed = (wasAllowed && test.expectation === 'allow') ||
(!wasAllowed && test.expectation === 'deny');
results.push({
test: test.name,
passed,
expected: test.expectation,
actual: wasAllowed ? 'allowed' : 'denied'
});
} finally {
await session.close();
}
}
return results;
}
Validation Checkpoint: Running agent-factory test sec-auditor shows pass/fail for each boundary.
Testing Strategy
Unit Tests
// tests/validator.test.ts
import { validateAgentConfig } from '../src/generator/Validator';
describe('Agent Config Validator', () => {
it('accepts valid minimal config', () => {
const config = { name: 'test-agent' };
const result = validateAgentConfig(config);
expect(result.valid).toBe(true);
});
it('rejects invalid name format', () => {
const config = { name: 'Test Agent' };
const result = validateAgentConfig(config);
expect(result.valid).toBe(false);
expect(result.errors[0].message).toContain('pattern');
});
it('rejects invalid model', () => {
const config = { name: 'test', model: 'gpt-4' };
const result = validateAgentConfig(config);
expect(result.valid).toBe(false);
});
it('warns about write without path restrictions', () => {
const config = {
name: 'test',
allowedTools: ['write']
};
const warnings = semanticValidation(config);
expect(warnings.some(w => w.includes('path restrictions'))).toBe(true);
});
});
Template Tests
// tests/templates.test.ts
describe('Agent Templates', () => {
const templates = ['security-auditor', 'code-reviewer', 'devops-engineer'];
templates.forEach(name => {
it(`${name} template is valid`, () => {
const template = loadTemplate(name);
const result = validateAgentConfig(template);
expect(result.valid).toBe(true);
});
it(`${name} has appropriate restrictions`, () => {
const template = loadTemplate(name);
// Security auditor should not have write
if (name === 'security-auditor') {
expect(template.allowedTools).not.toContain('write');
expect(template.allowedTools).not.toContain('shell');
}
});
});
});
Integration Tests
// tests/integration.test.ts
describe('Agent Factory Integration', () => {
it('creates agent file in correct location', async () => {
await createAgent({
name: 'test-agent',
archetype: 'security-auditor'
});
const path = '.kiro/agents/test-agent.json';
expect(fs.existsSync(path)).toBe(true);
const config = JSON.parse(fs.readFileSync(path, 'utf8'));
expect(config.name).toBe('test-agent');
});
it('agent can be started in Kiro', async () => {
// This test requires Kiro CLI installed
const result = await exec('kiro-cli --agent test-agent --test');
expect(result.exitCode).toBe(0);
});
});
Common Pitfalls and Debugging
Pitfall 1: Permission Conflicts
Symptom: Agent behaves unexpectedly (allows or denies wrong operations)
Cause: Conflicting allowedPaths and deniedPaths
Debug:
function analyzePermissions(config: AgentConfig): PermissionAnalysis {
const analysis: PermissionAnalysis = {};
// Check for conflicts
if (config.toolsSettings?.read) {
const allowed = config.toolsSettings.read.allowedPaths || ['**/*'];
const denied = config.toolsSettings.read.deniedPaths || [];
// Find paths in both
for (const ap of allowed) {
for (const dp of denied) {
if (minimatch(ap, dp) || minimatch(dp, ap)) {
analysis.conflicts = analysis.conflicts || [];
analysis.conflicts.push(`${ap} conflicts with ${dp}`);
}
}
}
}
return analysis;
}
Solution: Ensure deniedPaths are always more specific than allowedPaths.
Pitfall 2: Resource Not Found
Symptom: Agent starts but can’t access expected resources
Cause: File paths in resources are relative, but need to be project-relative
Debug:
# Check if resources exist
cat .kiro/agents/sec-auditor.json | jq -r '.resources[]' | xargs -I {} ls -la {}
Solution: Always use project-relative paths in resources:
{
"resources": [
"file://docs/security.md", // Correct: project-relative
"file:///Users/me/docs/sec.md" // Absolute paths also work
]
}
Pitfall 3: Shell Command Matching
Symptom: Commands blocked that should be allowed (or vice versa)
Cause: Pattern matching is substring-based
Debug:
// Log all command checks
function isCommandAllowed(command: string, settings: ShellSettings): boolean {
console.log(`Checking: "${command}"`);
console.log(`Allowed patterns: ${settings.allowedCommands}`);
console.log(`Denied patterns: ${settings.deniedCommands}`);
for (const denied of settings.deniedCommands || []) {
if (command.includes(denied)) {
console.log(`BLOCKED by: ${denied}`);
return false;
}
}
return true;
}
Solution: Use explicit patterns:
{
"shell": {
"deniedCommands": [
"rm -rf", // Blocks "rm -rf /"
"sudo ", // Note trailing space to avoid blocking "sudoers"
"curl | bash" // Block pipe to bash
]
}
}
Extensions and Challenges
Extension 1: Agent Composition
Create agents that can delegate to other agents:
{
"name": "lead-engineer",
"canDelegate": ["code-reviewer", "sec-auditor", "test-engineer"],
"delegationRules": {
"security_concerns": "sec-auditor",
"code_quality": "code-reviewer",
"test_coverage": "test-engineer"
}
}
Extension 2: Conditional Permissions
Permissions that change based on context:
{
"conditionalPermissions": {
"production_branch": {
"when": { "git_branch": "main" },
"deny": ["write", "shell"]
},
"test_files": {
"when": { "path_pattern": "**/*.test.ts" },
"allow": ["write"]
}
}
}
Extension 3: Audit Logging
Log all agent actions for compliance:
interface AgentAction {
timestamp: Date;
agentName: string;
tool: string;
input: object;
outcome: 'allowed' | 'denied' | 'error';
userId: string;
}
function logAction(action: AgentAction): void {
const logPath = '.kiro/audit/agent-actions.jsonl';
appendFileSync(logPath, JSON.stringify(action) + '\n');
}
Challenge: Visual Permission Editor
Build a UI for editing agent permissions:
┌─────────────────────────────────────────────────────────────────────┐
│ Agent: sec-auditor │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ TOOLS │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ [x] read [x] grep [x] glob [ ] write [ ] shell │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ READ PATHS │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Allowed: [src/**/*.ts] [config/**] [+Add] │ │
│ │ Denied: [**/*.env] [**/secrets/**] [+Add] │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ [Save] [Cancel] [Test Permissions] │
└─────────────────────────────────────────────────────────────────────┘
Real-World Connections
How Professionals Use This
- Security Teams: Create read-only auditors for compliance reviews
- DevOps: Scoped agents for deployment automation
- Code Review: Automated first-pass reviewers with consistent standards
- Onboarding: Restricted agents for junior developers
Industry Patterns
Role-Based Access Control (RBAC): Agent permissions mirror RBAC patterns used in IAM, database access, and application security.
Principle of Least Privilege: Fundamental security principle applied to AI agents.
Separation of Concerns: Different agents for different tasks, similar to microservices architecture.
Self-Assessment Checklist
Understanding Verification
- Can you explain why a read-only agent is safer for security audits?
- Cannot hide vulnerabilities by modifying code
- Cannot exfiltrate data via shell commands
- Audit trail is clean (only read operations)
- How do allowedTools and toolsSettings interact?
- allowedTools is the first gate (tool must be listed)
- toolsSettings provides fine-grained control within allowed tools
- Both must permit the action
- What is the principle of least privilege?
- Grant only permissions necessary for the task
- Reduce blast radius of errors or malicious actions
- Review and revoke unused permissions
- How would you test that an agent respects its boundaries?
- Attempt denied operations and verify rejection
- Check error messages are informative
- Test edge cases (path patterns, command variations)
Skill Demonstration
- I can create custom agent configs from scratch
- I can validate configs for errors and security issues
- I can explain permission hierarchy to others
- I can debug permission problems
- I can design agents for specific use cases
Interview Preparation
Be ready to answer:
- “How would you design a permission system for AI agents?”
- “What’s the principle of least privilege and how does it apply to AI?”
- “How do you prevent AI from executing destructive commands?”
- “How would you audit AI agent actions?”
Recommended Reading
| Topic | Resource | Why It Helps |
|---|---|---|
| Permission Design | “Foundations of Information Security” Ch. 5 | Access control fundamentals |
| Clean Architecture | “Clean Architecture” by Martin, Ch. 22 | Component boundaries |
| Schema Validation | JSON Schema specification | Config validation patterns |
| Security Patterns | OWASP guides | Real-world security concerns |
| TypeScript | “Programming TypeScript” by Cherny | Type-safe config handling |
What Success Looks Like
When you complete this project, you will have:
- Agent Library: Collection of pre-built, tested agent configs
- Creation Tool: CLI for generating new agents interactively
- Validation System: Robust checking for config errors
- Testing Framework: Verify agents respect boundaries
- Security Mindset: Understanding of permission design
Next Steps: Move to Project 5 (Steering Rules Engine) to learn how to encode project standards as persistent AI guidance.