Ollama Integration

Run Laddr agents with local LLM models using Ollama. This guide covers configuration, Docker setup, and usage examples.

Configuration

Environment Variables

Add these to your .env file:

# ===================================
# Ollama Configuration
# ===================================

# Ollama server URL
# For local development (Ollama running on host):
OLLAMA_BASE_URL=http://localhost:11434

# For Docker containers (see Docker Setup section):
# OLLAMA_BASE_URL=http://host.docker.internal:11434

# Default Ollama model (used if not specified per-agent)
OLLAMA_MODEL=gemma2:2b

# ===================================
# Backend Selection
# ===================================

# Set Ollama as default LLM backend for all agents
LLM_BACKEND=ollama

# Or configure per-agent:
LLM_BACKEND_RESEARCHER=ollama
LLM_BACKEND_COORDINATOR=openai  # Mix with cloud models

# ===================================
# Per-Agent Models
# ===================================

# Use different Ollama models for different agents
LLM_MODEL_RESEARCHER=gemma2:2b        # Fast, cheap model
LLM_MODEL_COORDINATOR=llama3.2:latest # Better reasoning
LLM_MODEL_WRITER=mistral:7b           # Good at writing

Configuration Priority

Laddr resolves Ollama configuration in this order (highest to lowest priority): For base URL:

Per-agent base URL: LLM_BASE_URL_RESEARCHER=http://custom:11434
Global Ollama URL: OLLAMA_BASE_URL=http://localhost:11434
Default: http://localhost:11434

For models:

Per-agent model: LLM_MODEL_RESEARCHER=gemma2:2b
Global Ollama model: OLLAMA_MODEL=llama3.2:latest
Global LLM model: LLM_MODEL=gemma2:2b
Default: gemma2:2b

Docker Setup

When running Laddr in Docker containers, special network configuration is needed so containers can reach Ollama running on the host machine.

Problem: Container Network Isolation

By default, http://localhost:11434 inside a Docker container refers to the container itself, not the host machine where Ollama is running. Error you might see:

LLM generation failed: Ollama endpoints not reachable at http://localhost:11434
Cannot connect to host localhost:11434

Solution: Use host.docker.internal

Docker provides a special hostname host.docker.internal that resolves to the host’s IP address.

Docker Compose Configuration

Update your docker-compose.yml:

services:
  api:
    build:
      context: ..
      dockerfile: tester/Dockerfile
    ports:
      - 8000:8000
    environment:
      # Point to host's Ollama via special hostname
      OLLAMA_BASE_URL: "http://host.docker.internal:11434"
    # Map host.docker.internal to host gateway
    extra_hosts:
      - "host.docker.internal:host-gateway"
    # ... other config

  researcher_worker:
    build:
      context: ..
      dockerfile: tester/Dockerfile
    command: python -m agents.researcher
    environment:
      AGENT_NAME: researcher
      # Worker must also use host.docker.internal
      OLLAMA_BASE_URL: "http://host.docker.internal:11434"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    # ... other config

  coordinator_worker:
    build:
      context: ..
      dockerfile: tester/Dockerfile
    command: python -m agents.coordinator
    environment:
      AGENT_NAME: coordinator
      OLLAMA_BASE_URL: "http://host.docker.internal:11434"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    # ... other config

Key Configuration Points

1. Environment Variable

environment:
  OLLAMA_BASE_URL: "http://host.docker.internal:11434"

Tells Laddr to connect to Ollama via the special hostname. 2. Extra Hosts Mapping

extra_hosts:
  - "host.docker.internal:host-gateway"

Maps the hostname to the Docker host’s gateway IP. 3. All Services Need It Apply this configuration to all services that use Ollama:

API server
Worker containers
Any agent services

Verification in Docker

Test Ollama connectivity from inside a container:

# Start your containers
docker compose up -d

# Exec into a worker container
docker compose exec researcher_worker bash

# Test connection
curl http://host.docker.internal:11434/api/generate -d '{
  "model": "gemma2:2b",
  "prompt": "test",
  "stream": false
}'

Expected: JSON response with generated text.

Usage Examples

Example 1: Simple Agent with Ollama

# my_agent.py
from laddr import Agent
from laddr.llms import ollama

researcher = Agent(
    name="researcher",
    role="Research Specialist",
    goal="Find and analyze information",
    backstory="You are an expert researcher.",
    
    # Use Ollama with gemma2:2b model
    llm=ollama(model="gemma2:2b"),
    
    tools=[],
    instructions="Research the given topic thoroughly."
)

Example 2: Custom Ollama Server

from laddr import Agent
from laddr.llms import ollama

# Connect to Ollama on custom port/host
researcher = Agent(
    name="researcher",
    role="Researcher",
    goal="Research information",
    backstory="Expert researcher",
    
    llm=ollama(
        model="llama3.2:latest",
        base_url="http://192.168.1.100:11434"  # Remote Ollama server
    ),
    
    tools=[]
)

Example 3: Mixed Backends

from laddr import Agent
from laddr.llms import ollama, gemini

# Coordinator uses cloud API (complex reasoning)
coordinator = Agent(
    name="coordinator",
    role="Task Coordinator",
    goal="Coordinate research tasks",
    backstory="Experienced coordinator",
    llm=gemini("gemini-2.0-flash"),  # Cloud API
    is_coordinator=True
)

# Researcher uses local Ollama (cost-effective)
researcher = Agent(
    name="researcher",
    role="Researcher",
    goal="Find information",
    backstory="Research specialist",
    llm=ollama("gemma2:2b"),  # Local model
    tools=[web_search]
)

Example 4: Environment-Based Configuration

# agents/researcher.py
from laddr import Agent
from laddr.llms import ollama

# Model and URL from environment variables
researcher = Agent(
    name="researcher",
    role="Researcher",
    goal="Research topics",
    backstory="Expert researcher",
    
    # Automatically uses:
    # - LLM_MODEL_RESEARCHER or OLLAMA_MODEL or default
    # - OLLAMA_BASE_URL or default
    llm=ollama(),
    
    tools=[]
)

Available Models

Popular Ollama models you can use:

gemma2:2b - Fast, lightweight (good for simple tasks)
llama3.2:latest - Balanced performance and quality
mistral:7b - Good for writing and analysis
llama3.1:8b - Strong reasoning capabilities
qwen2.5:7b - Multilingual support

Start with smaller models (2b-7b) for faster responses. Use larger models (13b+) for complex reasoning tasks.

Troubleshooting

Connection Errors

If you see connection errors:

Verify Ollama is running:
curl http://localhost:11434/api/tags

Check Docker network:

docker compose exec worker curl http://host.docker.internal:11434/api/tags

Verify environment variables:

docker compose exec worker env | grep OLLAMA

Model Not Found

If a model is not found:

Pull the model:
ollama pull gemma2:2b
List available models:
ollama list

Performance Issues

If responses are slow:

Use smaller models for simple tasks
Increase Ollama’s context window if needed
Check system resources (CPU, RAM)
Consider using GPU acceleration

Next Steps

Agent Configuration - Configure agents
Local Runtime - Local development
Scaling & Operations - Production deployment

Introduction

Getting Started

Guides

Reference

Examples

Community

Configuration

Environment Variables

Configuration Priority

Docker Setup

Problem: Container Network Isolation

Solution: Use host.docker.internal

Docker Compose Configuration

Key Configuration Points

Verification in Docker

Usage Examples

Example 1: Simple Agent with Ollama

Example 2: Custom Ollama Server

Example 3: Mixed Backends

Example 4: Environment-Based Configuration

Available Models

Troubleshooting

Connection Errors

Model Not Found

Performance Issues

Next Steps

​Configuration

​Environment Variables

​Configuration Priority

​Docker Setup

​Problem: Container Network Isolation

​Solution: Use host.docker.internal

​Docker Compose Configuration

​Key Configuration Points

​Verification in Docker

​Usage Examples

​Example 1: Simple Agent with Ollama

​Example 2: Custom Ollama Server

​Example 3: Mixed Backends

​Example 4: Environment-Based Configuration

​Available Models

​Troubleshooting

​Connection Errors

​Model Not Found

​Performance Issues

​Next Steps

Configuration

Environment Variables

Configuration Priority

Docker Setup

Problem: Container Network Isolation

Solution: Use host.docker.internal

Docker Compose Configuration

Key Configuration Points

Verification in Docker

Usage Examples

Example 1: Simple Agent with Ollama

Example 2: Custom Ollama Server

Example 3: Mixed Backends

Example 4: Environment-Based Configuration

Available Models

Troubleshooting

Connection Errors

Model Not Found

Performance Issues

Next Steps