Warning
🚧 This repository is under active development. Watch the repo, monitor branches and issues, and check the Changelog for the latest updates.
🗺️ Navigation:
🔵 Home | Vision LLM Theory | UI | Deployment | CDK Stacks | Runtime | S3 Files | Lambda Specialists | Prompting System
Broad Agentic Document Generative Extraction & Recognition System
BADGERS transforms document processing through vision-enabled AI and deep layout analysis. Unlike traditional text extraction tools, BADGERS understands document structure and meaning by recognizing visual hierarchies, reading patterns, and contextual relationships between elements.
Traditional document processing tools extract text but lose context. They can't distinguish a header from body text, understand table relationships, or recognize that a diagram explains the adjacent paragraph. BADGERS solves this by:
- 🏗️ Preserving semantic structure - Maintains document hierarchy and element relationships
- 👁️ Understanding visual context - Recognizes how layout conveys meaning
- 📚 Processing diverse content - Handles 21+ element types from handwriting to equations
- 🤖 Automating complex workflows - Orchestrates multiple specialized specialists via an AI agent
Use cases: research acceleration, compliance automation, content management, accessibility remediation.
A single React + Express app is both the testing workbench and the deployment/ops console — the same code runs locally via npm run dev or on ECS behind Cognito OIDC. Tabs are role-gated: the Testing row below is visible to all users, while an admin-only Deploy row (Stacks, Specialists, S3 Configs, Deploy Tags) is not pictured here. See the UI README for the full tab and role breakdown.
| Home | Chat |
|---|---|
 |
 |
| Landing view with per-page navigation and the resolved environment (region, runtime ARN, gateway ID, config bucket). | Streams messages to the AgentCore Runtime over WebSocket, with extended-thinking blocks, the live gateway tool list, and audit / dynamic-token toggles. |
| Create Specialist | Evaluations |
|---|---|
 |
 |
| Four-step wizard — basic info, generated prompt review, examples, deploy — including primary model and two fallbacks. | Pages through a session's specialist output and scores accuracy, element identification, and contextual understanding 1–5. |
| Observability | Edit Specialist |
|---|---|
 |
 |
Pulls traces and spans for a session from the CloudWatch aws/spans log group, with token usage and an event timeline. |
Edit an existing specialist's prompt XML configuration, model selections, and deployment settings. |
| Report Overview | Page Reader — Original vs Enhanced |
|---|---|
 |
 |
| Per-page analysis summary with element counts, specialist results, and structured XML output for each processed page. | Side-by-side original and enhanced page images — the image enhancer preprocesses pages for improved specialist accuracy. |
| Page Reader — Audit Trail | Report Overview — Multi-Document |
|---|---|
 |
 |
| Specialist execution times, token usage, and model selections per page — useful for performance tuning and cost analysis. | The same report viewer on a different document type (textbook with diagrams), showing how analysis adapts to content. |
The deployment menu tracks the eight ordered steps — Lambda layers, foundational infrastructure, prompts/manifests/schemas, specialist Lambdas, gateway, runtime, UI image, UI ECS service — per deployment ID and stack suffix. Steps can be run individually, all at once, or resumed from wherever the last run stopped.
┌─────────────────────────────────────────────────────────────────────────────┐
│ AgentCore Runtime │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ PDF Analysis Agent (Strands) │ │
│ │ - Claude Opus 4.6 with Adaptive Thinking │ │
│ │ - Session state management │ │
│ │ - MCP tool orchestration │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ AgentCore Gateway │
│ - MCP Protocol (2025-03-26) │
│ - Cognito JWT Authentication │
│ - Semantic tool search │
└─────────────────────────────────────────────────────────────────────────────┘
│
┌──────────────────┼──────────────────┐
│ │ │
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Lambda │ │ Lambda │ │ Lambda │
│ Specialist │ │ Specialist │ │ Specialist │
│ (27 tools) │ │ │ │ │
└─────────────┘ └─────────────┘ └─────────────┘
│ │ │
└──────────────────┼──────────────────┘
▼
┌─────────────┐
│ Bedrock │
│ Claude │
└─────────────┘
- 📄 User submits a document with analysis instructions
- 🧠 Strands Agent (running in AgentCore Runtime) interprets the request
- 🔧 Agent selects tools from a library of specialists via MCP Gateway
- 📋 Agent opens a job on the first specialist call, tagging every invocation in the run with the job and document it belongs to
- ⚡ Lambda specialists (standardized and domain-specific functions, including container-based) process document elements using Claude vision models, each recording its own outcome
- 📊 Results aggregate with preserved structure and semantic relationships
| Component | Technology |
|---|---|
| 🤖 Agent Framework | Strands Agents |
| 🏠 Agent Hosting | Amazon Bedrock AgentCore Runtime |
| 🚪 Tool Gateway | Amazon Bedrock AgentCore Gateway (MCP Protocol) |
| 🧠 Foundation Model | Claude Opus 4.6 for the agent; eight models for specialists (see below) |
| ⚡ Compute | AWS Lambda (modular specialist functions, including container-based) |
| 📦 Storage | Amazon S3 (configs, prompts, outputs) |
| 📋 Job Tracking | Amazon DynamoDB (document → job → subtask state) |
| 🖥️ UI Hosting | Amazon ECS Express Gateway service (in a VPC) |
| 🔐 Auth | Amazon Cognito (OIDC + PKCE for the UI, OAuth 2.0 M2M for the Gateway) |
| 🏗️ IaC | AWS CDK (Python) |
| 📈 Observability | CloudWatch Logs, X-Ray Transaction Search |
| 📊 Cost Tracking | Bedrock Application Inference Profiles |
| Specialist | Purpose |
|---|---|
📸 pdf_to_images_converter |
Convert PDF pages to images |
🏷️ classify_pdf_content |
Classify document content type |
📝 full_text_specialist |
Extract all text content |
📊 table_specialist |
Extract and structure tables |
📈 charts_specialist |
Analyze charts and graphs |
🔀 diagram_specialist |
Process diagrams and flowcharts |
📐 layout_specialist |
Document structure analysis |
🏥 decision_tree_specialist |
Medical/clinical document analysis |
🔬 scientific_specialist |
Scientific paper analysis |
✍️ handwriting_specialist |
Handwritten text recognition |
🔢 handwriting_math_specialist |
Handwritten mathematical notation recognition |
💻 code_block_specialist |
Extract code snippets |
🗂️ metadata_generic_specialist |
Generic metadata extraction |
🗂️ metadata_mads_specialist |
MADS metadata format extraction |
🗂️ metadata_mods_specialist |
MODS metadata format extraction |
🔑 keyword_topic_specialist |
Extract keywords and topics |
🔧 remediation_specialist |
PDF accessibility remediation (container, content stream tagging + structure tree builder) |
📄 page_specialist |
Single page content analysis |
🧱 elements_specialist |
Document element detection |
🧱 robust_elements_specialist |
Enhanced element detection with fallbacks |
👁️ general_visual_analysis_specialist |
General-purpose visual content analysis |
✏️ editorial_specialist |
Editorial content and markup analysis |
🗺️ war_map_specialist |
Historical war map analysis |
🎓 edu_transcript_specialist |
Educational transcript analysis |
🔗 correlation_specialist |
Correlate multi-specialist results per page |
🖼️ image_enhancer |
Image enhancement and preprocessing |
- ☁️ AWS CLI configured with credentials
- 📦 AWS CDK v2 (
npm install -g aws-cdk) - 🐳 Docker (running)
- 🐍 Python 3.12+
- ⚡ uv
./deploy.sh
That is the whole command. deploy.sh asks which deployment to work on — listing anything it finds in .deploy-state/, or offering to start a new one — and then presents a menu. It is resumable and every step is idempotent, so re-run it after a failure and it picks up where it stopped.
DEPLOYMENT_IDis not read from the environment. It is always chosen interactively, because a value left exported in your shell silently targets another deployment's stacks.
Pick option 9 for a full deployment, or 12 to run only what is still outstanding. You can jump straight to one option — ./deploy.sh 6 — and the deployment is still chosen interactively first.
The eight steps:
| # | Step | What it does |
|---|---|---|
| 1 | Lambda Layers | foundation, PDF processing, Poppler/qpdf |
| 2 | Foundational Infra | S3, Cognito, DynamoDB, IAM, ECR, Inference Profiles, X-Ray, Memory, VPC |
| 3 | Upload Config | prompts, manifests and schemas to the config bucket |
| 4 | Specialist Lambdas | container images, then the Lambda stack (12 of 27 specialists enabled by default; deployment/deployment_config.json selects which) |
| 5 | Gateway | AgentCore MCP Gateway, records the Gateway URL |
| 6 | Runtime | builds and pushes the agent image, then deploys the Runtime |
| 7 | UI — Build | generates ui/.env from Cognito, builds the bundle and image |
| 8 | UI — Deploy | ECS Express Gateway service, forces the rollout, waits for it |
Plus 9 full deployment, 12 resume, 10 status, 11 reset state (deletes nothing in AWS), 0 exit.
Step 8 asks once whether the UI should be publicly reachable. That answer is fixed for the life of the VPC — see Network exposure.
For the full procedure, prerequisites in depth, and every environment variable, see the Deployment Guide.
DEPLOYMENT_ID is a short label you choose — lowercase, starting with a letter, 16 characters or fewer. deploy.sh generates a three-character random STACK_SUFFIX once and persists both in .deploy-state/{DEPLOYMENT_ID}.json:
- Stack names are
BADGERS-{Name}-{DEPLOYMENT_ID}-{suffix}— for exampleBADGERS-S3-dev-a1b - Resource names carry both parts — for example
badgers-config-dev-a1b, and SSM parameters under/badgers-dev-a1b/
Because both are unique per deployment, several deployments can coexist in one account and region. Stack names include the deployment id as well as the suffix so each stack is self-describing — tooling reads a deployment's identity off the stack name, and a mistyped id matches no stacks instead of resolving someone else's. The state file also tracks which steps completed, which is what makes the script resumable.
./destroy.sh
Like deploy.sh it asks what to tear down, but it discovers deployments from CloudFormation rather than from .deploy-state/ — a state file can be deleted while the stacks are still live. You are then required to type the DEPLOYMENT_ID to confirm.
It empties the S3 buckets, deletes the ECS Express service and the AgentCore runtime before the VPC (CloudFormation cannot delete a VPC while any ENI is still attached), sweeps leftover ENIs, destroys every stack in reverse dependency order, verifies they are gone, and only then schedules the KMS key for deletion so its alias is freed for redeployment. A teardown that leaves stacks standing exits non-zero and says so rather than reporting success.
If a VPC stack still gets stuck on a lingering ENI:
DEPLOYMENT_ID=dev STACK_SUFFIX=a1b ./destroy.sh --vpc-cleanup-only
To tear down by hand when the script cannot run, follow Manual Teardown in the Console — the stack deletion order matters, and two resources have to be removed before any stack.
├── deployment/
│ ├── app.py # CDK app entry point
│ ├── stacks/ # CDK stack definitions
│ ├── lambdas/code/ # Specialist Lambda functions
│ ├── runtime/ # AgentCore Runtime container
│ ├── s3_files/ # Prompts, schemas, manifests
│ └── badgers-foundation/ # Shared specialist framework
├── ui/ # BADGERS UI (React + Express, runs locally or deployed via Docker)
│ ├── src/ # React components (testing + admin tabs, role-gated)
│ ├── server/ # Express API server (testing + admin routes, OIDC auth)
│ └── Dockerfile # Container image for AWS deployment
└── pyproject.toml
BADGERS uses Lambda layers shared across specialist functions:
🏗️ Foundation Layer (layer.zip)
- Built via
deployment/lambdas/build_foundation_layer.sh - Contains the specialist framework (7 Python modules)
- Includes dependencies: boto3, botocore
- Includes core system prompts used by all specialists
layer/python/
├── foundation/
│ ├── specialist_foundation.py # 🎯 Main orchestration class
│ ├── bedrock_client.py # 🔄 Bedrock API with retry/fallback
│ ├── configuration_manager.py # ⚙️ Config loading/validation
│ ├── image_processor.py # 🖼️ Image optimization
│ ├── message_chain_builder.py # 💬 Claude message formatting
│ ├── prompt_loader.py # 📜 Prompt file loading (local/S3)
│ └── response_processor.py # 📤 Response extraction
├── config/
│ └── config.py
└── prompts/core_system_prompts/
└── *.xml
📄 Poppler Layer (poppler-qpdf-layer.zip)
- PDF rendering library for
pdf_to_images_converter - Built via
deployment/lambdas/build_poppler_qdf_layer.sh
Each specialist follows the same pattern using SpecialistFoundation:
# Lambda handler (simplified)
def lambda_handler(event, context):
# 1️⃣ Load config from S3 manifest
config = load_manifest_from_s3(bucket, "full_text_specialist")
# 2️⃣ Initialize foundation with S3-aware prompt loader
specialist = SpecialistFoundation(...)
# 3️⃣ Run analysis pipeline
result = specialist.analyze(image_data)
# 4️⃣ Save result to S3 and return
save_result_to_s3(result, session_id)
return {"result": result}The analyze() method orchestrates:
- 🖼️ Image processing - Resize/optimize for Claude's vision API
- 📜 Prompt loading - Combine wrapper + specialist prompts from S3
- 💬 Message building - Format for Bedrock Converse API
- ⚡ Dynamic token estimation - Score image complexity and set token budget (when enabled)
- 🤖 Model invocation - Call Claude with retry/fallback logic
- ✅ Response processing - Extract and validate result
Prompts are modular XML files composed at runtime:
s3://config-bucket/
├── core_system_prompts/
│ ├── prompt_system_wrapper.xml # 🎁 Main template with placeholders
│ ├── core_rules/rules.xml # 📏 Shared rules for all specialists
│ └── error_handling/*.xml # ⚠️ Error response templates
├── prompts/{specialist_name}/
│ ├── {specialist}_job_role.xml # 👤 Role definition
│ ├── {specialist}_context.xml # 🌍 Domain context
│ ├── {specialist}_rules.xml # 📏 Specialist-specific rules
│ ├── {specialist}_tasks.xml # ✅ Task instructions
│ └── {specialist}_format.xml # 📋 Output format spec
└── wrappers/
└── prompt_system_wrapper.xml
The PromptLoader composes the final system prompt:
<!-- prompt_system_wrapper.xml -->
<system_prompt>
{core_rules} <!-- 📏 Injected from core_rules/rules.xml -->
{composed_prompt} <!-- 🧩 Injected from specialist prompt files -->
{error_handler_general}
{error_handler_not_found}
</system_prompt>
Placeholders like [[PIXEL_WIDTH]] and [[PIXEL_HEIGHT]] are replaced with actual image dimensions at runtime.
Each specialist has a manifest file in S3:
// s3://config-bucket/manifests/full_text_specialist.json
{
"tool": {
"name": "analyze_full_text_tool",
"description": "Extracts text content maintaining reading order...",
"inputSchema": {
"type": "object",
"properties": {
"image_path": { "type": "string" },
"session_id": { "type": "string" },
"audit_mode": { "type": "boolean" }
},
"required": ["image_path", "session_id"]
}
},
"specialist": {
"name": "full_text_specialist",
"enhancement_eligible": true,
"model_selections": {
"primary": "us.anthropic.claude-sonnet-4-6",
"fallback_list": [
"us.openai.gpt-5.6-terra",
"us.amazon.nova-2-lite-v1:0"
]
},
"max_retries": 3,
"prompt_files": [
"full_text_job_role.xml",
"full_text_context.xml",
"full_text_rules.xml",
"full_text_tasks_extraction.xml",
"full_text_format.xml"
],
"max_examples": 0,
"analysis_text": "full text content",
"expected_output_tokens": 6000,
"output_extension": "xml"
}
}
Key configuration features:
- 🔄 Model fallback chain - Primary model with ordered fallbacks
- 🔁 Retry logic - Configurable retry count per specialist
- 🧩 Prompt composition - List of XML files to combine
- 📋 Tool schema - MCP-compatible input schema for Gateway
- 🖼️ Enhancement eligible - Flag indicating specialist benefits from image preprocessing (used by
image_enhancertool)
Global settings (from environment or defaults):
{
"max_tokens": 8000,
"temperature": 0.1,
"max_image_size": 20971520, # 20MB
"max_dimension": 2048,
"jpeg_quality": 85,
"throttle_delay": 1.0,
"aws_region": "us-west-2"
}When enabled, BADGERS estimates the optimal max_tokens per image based on visual complexity, reducing cost on simple documents and avoiding truncation on dense ones. The scorer runs on the already-processed image bytes — no extra I/O.
Four metrics are combined into a complexity score: text pixel ratio, grayscale entropy, edge density, and color standard deviation. The score maps to a token budget (8K / 12K / 16K / 24K).
Enabling: Toggle "Dynamic Token Estimation" in the chat UI, or set the Lambda environment variable DYNAMIC_TOKENS_ENABLED=true.
Tuning: Add a dynamic_tokens block to an specialist manifest to customize weights and thresholds:
"dynamic_tokens": {
"weights": {
"text_ratio": 0.2,
"entropy": 0.3,
"edge_density": 0.3,
"color_std": 0.2
},
"thresholds": [
{"max_score": 0.20, "max_tokens": 8000},
{"max_score": 0.30, "max_tokens": 12000},
{"max_score": 0.45, "max_tokens": 16000},
{"max_score": 1.00, "max_tokens": 24000}
]
}
Observability: When active, logs report the estimated budget, actual token usage, and utilization percentage for calibration.
BADGERS uses Application Inference Profiles to enable cost allocation and usage monitoring. Every model in the registry gets one, and specialists resolve a model ID to its profile ARN at runtime:
┌─────────────────────────────────────────────────────────────────────────────┐
│ Inference Profile Flow │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ 1. CDK reads deployment/s3_files/config/model_registry.json at synth │
│ └─> One ApplicationInferenceProfile per model, in a loop │
│ badgers-claude-sonnet-4-6-{id}, badgers-claude-opus-4-6-{id}, │
│ badgers-gpt-5-6-terra-{id}, badgers-nova-2-lite-{id}, ... │
│ │
│ 2. The same loop writes one SSM parameter │
│ └─> /badgers-{id}/model-profiles = { model_id: profile_arn, ... } │
│ │
│ 3. Lambdas and the Runtime receive the parameter NAME, not the ARNs │
│ └─> MODEL_PROFILES_PARAM │
│ │
│ 4. At invocation, bedrock_client.py reads the parameter once, caches it, │
│ and looks up the model ID │
│ └─> Falls back to the raw model ID if no profile is configured │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
There are no per-model *_PROFILE_ARN environment variables. Adding or retiring a model is one registry edit plus a deploy — no stack, Lambda, or UI file lists models by hand.
The model list the UI offers comes from GET /api/models, which joins the SSM parameter against the registry, so the dropdown cannot offer a model this deployment has no profile for.
Important
Model inference does not stay in your deployment Region. All eight models are invoked through US geo cross-Region inference profiles (us.*), so Bedrock routes each request to a Region within the US geography. See Inference Profiles and Regions.
Option 1: Use the Wizard (Recommended)
cd ui
npm run dev
The Specialist Creation Wizard is the 🧙 Create Specialist tab in the UI. It generates the six prompt sections with Bedrock, writes the manifest, schema, prompts, and registry entry under deployment/custom_specialists/, then deploys the CustomSpecialists stack. Save and Deploy are separate steps — see Specialist Wizard Endpoints.
Option 2: Manual Creation
- 📜 Create prompt files in
deployment/s3_files/prompts/{specialist_name}/ - 📋 Create manifest in
deployment/s3_files/manifests/{specialist_name}.json - 📐 Create schema in
deployment/s3_files/schemas/{specialist_name}.json - ⚡ Create Lambda code in
deployment/lambdas/code/{specialist_name}/lambda_handler.py - 📝 Register in
deployment/stacks/lambda_stack.py - 🚀 Redeploy:
cdk deploy BADGERS-Lambda-{id}-{suffix} BADGERS-Gateway-{id}-{suffix}
If your AWS organization uses strict SCPs that deny cross-Region Bedrock operations, you may see:
AccessDeniedException: ... is not authorized to perform: bedrock:InvokeModelWithResponseStream
on resource: arn:aws:bedrock:us-east-2::foundation-model/anthropic.claude-... with an
explicit deny in a service control policy
BADGERS invokes US geo cross-Region inference profiles (us.*), which route each request to a destination Region inside the US geography. That is cross-Region routing — the us. prefix restricts which Regions can be used, not whether Regions are crossed. An SCP that allows only your deployment Region will block it.
Fix it on the SCP side: allow Bedrock inference actions in all US destination Regions for the profiles in use, or add an inference-profile exception. A denied ARN naming a Region you did not deploy into is the signature of this problem.
A denial on arn:aws:bedrock:::foundation-model/... — no Region, no account — is a different failure. That ARN form belongs to global cross-Region inference, and BADGERS does not use global.* profiles. If you see it, something is invoking a global. model ID.
See Inference Profiles and Regions for what crosses Regions, what does not, and why the IAM policy wildcards the Region field.
After a fresh deployment, the first model invocation may fail with:
AccessDeniedException: Model access is denied due to IAM user or service role is not authorized
to perform the required AWS Marketplace actions (aws-marketplace:ViewSubscriptions,
aws-marketplace:Subscribe)
The IAM stack now includes aws-marketplace:ViewSubscriptions and aws-marketplace:Subscribe permissions. If you see this error on an older deployment, redeploy the IAM stack. As a workaround, manually invoke the model once in the Bedrock console playground to trigger the Marketplace subscription.
Customers are responsible for making their own independent assessment of the information in this Guidance. This Guidance: (a) is for informational purposes only, (b) represents AWS current product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS products or services are provided "as is" without warranties, representations, or conditions of any kind, whether express or implied. AWS responsibilities and liabilities to its customers are controlled by AWS agreements, and this Guidance is not part of, nor does it modify, any agreement between AWS and its customers.
- Randall Potter
- Amazon Bedrock Developer Experience - Foundation model choice and customization
- Anthropic's Claude in Amazon Bedrock - Claude Opus 5, Opus 4.6, and Sonnet 4.6 hybrid reasoning models
- Claude Sonnet 4.6 model card - The default specialist workhorse: pricing, APIs, Regional availability
- Claude Opus 4.6 model card - Agent runtime and heavy-reasoning specialists
- Claude Opus 5 model card - Anthropic's newest Opus, same price as 4.6; adaptive thinking on by default
- OpenAI models in Amazon Bedrock - GPT-5.6 Terra; the GPT models are Converse-only on
bedrock-runtimeand needbedrock:InvokeModelonproject/default - Amazon Nova 2 Lite model card - The cheapest fallback in the chain
- Using Amazon Nova in AI Agents - Nova as foundation model for agents
- Geographic cross-Region inference - How
us.*profiles route, and the IAM grants they require
- Amazon Bedrock AgentCore Overview - Build, deploy, and operate agents at scale
- AgentCore Gateway Guide - Set up unified tool connectivity
- AgentCore Gateway Blog - Transforming enterprise AI agent tool development
- AgentCore Runtime - Secure serverless hosting for AI agents
- Lambda Layers Overview - Managing dependencies with layers
- Python Lambda Layers - Working with layers for Python functions
- Adding Layers to Functions - Layer configuration and management
- OAuth 2.0 Grants - Authorization code, implicit, and client credentials
- M2M Authorization - Scopes, resource servers, and machine-to-machine auth
- M2M Security Best Practices - Monitor, optimize, and secure M2M authorization
- CloudWatch + X-Ray Integration - Enhanced application monitoring
- Cross-Account Tracing - Distributed tracing across accounts
- AWS Observability Best Practices - Logs, metrics, and traces
- S3 as Data Lake Storage - Central storage platform best practices
- S3 Performance Optimization - Design patterns for optimal performance
- Amazon Kiro Overview - Agentic IDE for spec-driven development
- Kiro with AWS Builder ID - Sign in and get started with Kiro
- Nova Act IDE Extension - Accelerate AI agent development in Kiro
- Production-Ready AI Agents at Scale - Kiro as part of the agent development ecosystem