Skip to main content

System Architecture

This guide provides a comprehensive overview of Agenta's system architecture, explaining each component's role, responsibilities, and how they work together to deliver AI application evaluation and management capabilities.

System Overview​

Agenta follows a modern microservices architecture with clear separation of concerns. The system is designed for scalability, maintainability, and ease of deployment using Docker containers.

                                    ┌─────────────────────────────────────┐
                                    │               Users                 │
                                    │    (Developers, AI Engineers)       │
                                    └─────────────────┬───────────────────┘
                                                      │
                                    ┌─────────────────▼───────────────────┐
                                    │        Load Balancer / Proxy        │
                                    │         (Traefik or Nginx)          │
                                    │      Handles SSL and routing        │
                                    └─────────────┬───────────────────────┘
                                                  │
                    ┌─────────────────────────────┼─────────────────────────────┐
                    │                             │                             │
                    ▼                             ▼                             ▼
          ┌─────────────────┐         ┌─────────────────┐         ┌─────────────────┐
          │    Frontend     │         │   API Backend   │         │  Services API   │
          │    (Web UI)     │◄────────►    (FastAPI)    │◄────────►   (FastAPI)     │
          │                 │         │                 │         │                 │
          │ • Next.js App   │         │ • REST API      │         │ • Completion    │
          │ • Playground    │         │ • Core logic    │         │ • Chat          │
          │ • Admin UI      │         │ • Persistence   │         │ • LLM adapters  │
          └─────────────────┘         └─────────┬───────┘         └─────────────────┘
                    │                           │
                    │                           ▼
                    │                 ┌─────────────────────────────┐
                    │                 │      Worker Pool            │
                    │                 │   (background processes)    │
                    │                 │                             │
                    │                 │ • worker-evaluations        │
                    │                 │ • worker-tracing            │
                    │                 │ • worker-webhooks           │
                    │                 │ • worker-events             │
                    │                 │ • cron                      │
                    │                 └──────────────┬──────────────┘
                    │                                │
                    ▼                                ▼
          ┌───────────────────────────────────────────────────────────────────┐
          │                       Infrastructure Layer                        │
          │                                                                   │
          │ ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ │
          │ │   PostgreSQL      │ │      Redis        │ │   SuperTokens     │ │
          │ │                   │ │                   │ │                   │ │
          │ │ • Core DB         │ │ • Task queues     │ │ • Authentication  │ │
          │ │ • Tracing DB      │ │ • Streams         │ │ • Session mgmt    │ │
          │ │ • Auth DB         │ │ • Caching         │ │                   │ │
          │ └───────────────────┘ └───────────────────┘ └───────────────────┘ │
          └───────────────────────────────────────────────────────────────────┘

Frontend Components​

Web UI (NextJS Application)​

  • Technology: React, TypeScript, Next.js
  • Port: 3000 (internal)
  • Purpose: Primary user interface for Agenta platform

Key Responsibilities:

  • User Interface: Provides intuitive web interface for application management
  • Playground: Interactive environment for testing and evaluating LLM applications
  • Evaluation Dashboard: Visualizations and metrics for application performance
  • Application Management: Create, configure, and deploy AI applications
  • User Authentication: Login, registration, and session management

Backend Components​

API Service (FastAPI)​

  • Technology: Python, FastAPI, SQLAlchemy
  • Port: 8000 (internal)
  • Purpose: Core business logic and API endpoints

Key Responsibilities:

  • REST API: Provides RESTful endpoints for frontend and external integrations
  • Business Logic: Implements core platform functionality
  • Data Management: Handles CRUD operations for applications, evaluations, experiments, etc
  • Authentication: Integrates with SuperTokens for user authentication
  • Application Orchestration: Manages application lifecycle and deployment
  • Evaluation Management: Coordinates evaluation runs and result collection

Worker Services (TaskIQ + Async Consumers)​

  • Technology: Python workers, TaskIQ, asyncio consumers, Redis, PostgreSQL
  • Purpose: Background processing for evaluations, tracing, events, and webhooks

Key Responsibilities:

  • Evaluation Execution: worker-evaluations runs asynchronous evaluation workloads
  • Tracing Ingestion: worker-tracing consumes OTLP tracing pipelines
  • Webhook Delivery: worker-webhooks dispatches outbound webhook notifications
  • Event Processing: worker-events processes internal event streams
  • Scheduled Tasks: cron runs periodic maintenance and cleanup jobs

TaskIQ Integration:

  • Broker: Uses Redis streams for queueing and task distribution
  • Task Registration: Evaluation tasks are registered at worker startup
  • Execution: Workers consume Redis-backed jobs and process them asynchronously

Services Backend​

Services API (FastAPI)​

  • Technology: Python, FastAPI
  • Port: 80 (internal)
  • Purpose: LLM-facing endpoints and service-layer APIs exposed under /services/*

Key Responsibilities:

  • LLM Integration: Connects to various LLM providers (OpenAI, Anthropic, etc.)
  • Prompt Processing: Handles prompt templates and variable substitution
  • Response Generation: Manages LLM API calls and response handling
  • Provider Abstraction: Unified interface across different LLM providers
  • Error Handling: Robust error handling for LLM API failures
  • Endpoint Groups: Includes /services/completion/* and /services/chat/*

Infrastructure Services​

PostgreSQL (Database)​

  • Technology: PostgreSQL 16
  • Port: 5432
  • Purpose: Primary data storage

Databases:

  • Core Database: Application data, Datasets, Evaluations, Users & Profiles, etc.
  • Tracing Database: Execution traces and performance metrics
  • SuperTokens Database: Authentication and user management data

Redis (Task Queue, Caching & Sessions)​

  • Technology: Redis
  • Ports: 6379 (volatile), 6381 (durable)
  • Purpose: Task queue, caching, pub/sub, streams

Use Cases:

  • Task Queue: TaskIQ broker for background job distribution and processing
  • Application Caching: Frequently accessed data
  • Session Storage: User sessions and temporary data
  • Task Results: TaskIQ task results and status
  • Real-time Data: Live updates and notifications
  • Rate Limiting: API rate limit counters

SuperTokens (Authentication)​

  • Technology: SuperTokens
  • Port: 3567
  • Purpose: Authentication and user management

Features:

  • User Authentication: Login/logout, password management
  • Session Management: Secure session handling with JWT
  • OAuth Integration: Google, and GitHub
  • User Management: User registration, profile management

Service Dependencies​

Frontend Dependencies​

Web UI depends on:
├── API Service (primary backend)
├── Services API (playground and model calls)
└── Authentication (SuperTokens via API)

Backend Dependencies​

API Service depends on:
├── PostgreSQL (data persistence)
├── Redis (task queue, caching, sessions)
├── SuperTokens (authentication)
└── Worker pool (async task execution)

Worker Dependencies​

Worker pool depends on:
├── Redis (queues and streams)
├── PostgreSQL (state and persistence)
├── API backend (coordination and config)
└── Services API / external endpoints (workload-specific processing)