Skip to main content
Run Laddr agents with local LLM models using Ollama. This guide covers configuration, Docker setup, and usage examples.

​
Configuration

​
Environment Variables

Add these to your .env file: 
# ===================================
# Ollama Configuration
# ===================================

# Ollama server URL
# For local development (Ollama running on host):
OLLAMA_BASE_URL=http://localhost:11434

# For Docker containers (see Docker Setup section):
# OLLAMA_BASE_URL=http://host.docker.internal:11434

# Default Ollama model (used if not specified per-agent)
OLLAMA_MODEL=gemma2:2b

# ===================================
# Backend Selection
# ===================================

# Set Ollama as default LLM backend for all agents
LLM_BACKEND=ollama

# Or configure per-agent:
LLM_BACKEND_RESEARCHER=ollama
LLM_BACKEND_COORDINATOR=openai  # Mix with cloud models

# ===================================
# Per-Agent Models
# ===================================

# Use different Ollama models for different agents
LLM_MODEL_RESEARCHER=gemma2:2b        # Fast, cheap model
LLM_MODEL_COORDINATOR=llama3.2:latest # Better reasoning
LLM_MODEL_WRITER=mistral:7b           # Good at writing

​
Configuration Priority

Laddr resolves Ollama configuration in this order (highest to lowest priority): For base URL:
  1. Per-agent base URL: LLM_BASE_URL_RESEARCHER=http://custom:11434
  2. Global Ollama URL: OLLAMA_BASE_URL=http://localhost:11434
  3. Default: http://localhost:11434
For models:
  1. Per-agent model: LLM_MODEL_RESEARCHER=gemma2:2b
  2. Global Ollama model: OLLAMA_MODEL=llama3.2:latest
  3. Global LLM model: LLM_MODEL=gemma2:2b
  4. Default: gemma2:2b

​
Docker Setup

When running Laddr in Docker containers, special network configuration is needed so containers can reach Ollama running on the host machine.

​
Problem: Container Network Isolation

By default, http://localhost:11434 inside a Docker container refers to the container itself, not the host machine where Ollama is running. Error you might see: 
LLM generation failed: Ollama endpoints not reachable at http://localhost:11434
Cannot connect to host localhost:11434

​
Solution: Use host.docker.internal

Docker provides a special hostname host.docker.internal that resolves to the host’s IP address.

​
Docker Compose Configuration

Update your docker-compose.yml: 
services:
  api:
    build:
      context: ..
      dockerfile: tester/Dockerfile
    ports:
      - 8000:8000
    environment:
      # Point to host's Ollama via special hostname
      OLLAMA_BASE_URL: "http://host.docker.internal:11434"
    # Map host.docker.internal to host gateway
    extra_hosts:
      - "host.docker.internal:host-gateway"
    # ... other config

  researcher_worker:
    build:
      context: ..
      dockerfile: tester/Dockerfile
    command: python -m agents.researcher
    environment:
      AGENT_NAME: researcher
      # Worker must also use host.docker.internal
      OLLAMA_BASE_URL: "http://host.docker.internal:11434"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    # ... other config

  coordinator_worker:
    build:
      context: ..
      dockerfile: tester/Dockerfile
    command: python -m agents.coordinator
    environment:
      AGENT_NAME: coordinator
      OLLAMA_BASE_URL: "http://host.docker.internal:11434"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    # ... other config

​
Key Configuration Points

1. Environment Variable 
environment:
  OLLAMA_BASE_URL: "http://host.docker.internal:11434"
Tells Laddr to connect to Ollama via the special hostname. 2. Extra Hosts Mapping 
extra_hosts:
  - "host.docker.internal:host-gateway"
Maps the hostname to the Docker host’s gateway IP. 3. All Services Need It Apply this configuration to all services that use Ollama:
  • API server
  • Worker containers
  • Any agent services

​
Verification in Docker

Test Ollama connectivity from inside a container: 
# Start your containers
docker compose up -d

# Exec into a worker container
docker compose exec researcher_worker bash

# Test connection
curl http://host.docker.internal:11434/api/generate -d '{
  "model": "gemma2:2b",
  "prompt": "test",
  "stream": false
}'
Expected: JSON response with generated text.

​
Usage Examples

​
Example 1: Simple Agent with Ollama

# my_agent.py
from laddr import Agent
from laddr.llms import ollama

researcher = Agent(
    name="researcher",
    role="Research Specialist",
    goal="Find and analyze information",
    backstory="You are an expert researcher.",
    
    # Use Ollama with gemma2:2b model
    llm=ollama(model="gemma2:2b"),
    
    tools=[],
    instructions="Research the given topic thoroughly."
)

​
Example 2: Custom Ollama Server

from laddr import Agent
from laddr.llms import ollama

# Connect to Ollama on custom port/host
researcher = Agent(
    name="researcher",
    role="Researcher",
    goal="Research information",
    backstory="Expert researcher",
    
    llm=ollama(
        model="llama3.2:latest",
        base_url="http://192.168.1.100:11434"  # Remote Ollama server
    ),
    
    tools=[]
)

​
Example 3: Mixed Backends

from laddr import Agent
from laddr.llms import ollama, gemini

# Coordinator uses cloud API (complex reasoning)
coordinator = Agent(
    name="coordinator",
    role="Task Coordinator",
    goal="Coordinate research tasks",
    backstory="Experienced coordinator",
    llm=gemini("gemini-2.0-flash"),  # Cloud API
    is_coordinator=True
)

# Researcher uses local Ollama (cost-effective)
researcher = Agent(
    name="researcher",
    role="Researcher",
    goal="Find information",
    backstory="Research specialist",
    llm=ollama("gemma2:2b"),  # Local model
    tools=[web_search]
)

​
Example 4: Environment-Based Configuration

# agents/researcher.py
from laddr import Agent
from laddr.llms import ollama

# Model and URL from environment variables
researcher = Agent(
    name="researcher",
    role="Researcher",
    goal="Research topics",
    backstory="Expert researcher",
    
    # Automatically uses:
    # - LLM_MODEL_RESEARCHER or OLLAMA_MODEL or default
    # - OLLAMA_BASE_URL or default
    llm=ollama(),
    
    tools=[]
)

​
Available Models

Popular Ollama models you can use:
  • gemma2:2b - Fast, lightweight (good for simple tasks)
  • llama3.2:latest - Balanced performance and quality
  • mistral:7b - Good for writing and analysis
  • llama3.1:8b - Strong reasoning capabilities
  • qwen2.5:7b - Multilingual support
Start with smaller models (2b-7b) for faster responses. Use larger models (13b+) for complex reasoning tasks.

​
Troubleshooting

​
Connection Errors

If you see connection errors:
  1. Verify Ollama is running: 
    curl http://localhost:11434/api/tags
  2. Check Docker network: 
    docker compose exec worker curl http://host.docker.internal:11434/api/tags
  3. Verify environment variables: 
    docker compose exec worker env | grep OLLAMA

​
Model Not Found

If a model is not found:
  1. Pull the model: 
    ollama pull gemma2:2b
  2. List available models: 
    ollama list

​
Performance Issues

If responses are slow:
  1. Use smaller models for simple tasks
  2. Increase Ollama’s context window if needed
  3. Check system resources (CPU, RAM)
  4. Consider using GPU acceleration

​
Next Steps