--- title: Building Self-Hosted AI Agents with Local LLMs description: A practical guide to running AI agents entirely on your own hardware using Ollama, Docker, and OpenClaw — covering hardware requirements, model selection, and when local beats cloud. date: 2026-01-30 location: San Francisco, CA – Jan 30th, 2026 image: /images/self-hosted-ai-agents-local-llms-hero.png keywords: local llm, self-hosted ai, ollama, openclaw, clawdbot, privacy, on-premise ai, docker, gpu requirements, enterprise ai faq: - question: "What hardware do I need to run a local LLM?" answer: "For hobbyist/dev use, a single RTX 4090 (24GB VRAM, ~$1,600) handles most local use cases including 7-13B models at full precision and 70B models quantized. For production, 2x A100 80GB ($160GB total VRAM) runs 70B models at full precision at ~80 tok/s. Apple M2/M3 Max with 32-96GB unified memory also works well." - question: "When should I use local AI vs cloud APIs?" answer: "Use local when compliance requires data to stay on your network, volume exceeds 500K tokens/day, latency matters, or you need predictable availability. Use cloud when you need frontier capabilities (Claude Opus, GPT-4), volume is under 100K tokens/day, you need burst capacity, or you want zero maintenance. Most people use a hybrid approach." - question: "How do I set up Ollama with OpenClaw for local AI agents?" answer: "Install Ollama (brew install ollama on macOS), pull a model (ollama pull llama3.3:70b-instruct-q4_K_M), start the server (ollama serve), then add Ollama as a provider in ~/.openclaw/openclaw.json with baseUrl 'http://localhost:11434/v1'. Apply config and switch with '/model llama-local'." --- > **Note:** Clawdbot is now called **OpenClaw** (website: [openclaw.ai](https://openclaw.ai)). This guide works with all versions. For new installs, use `npm install -g openclaw`. Running your AI agent locally means your data never leaves your network. No API calls to external servers, no compliance paperwork, no per-token billing. The tradeoff is hardware investment and setup complexity. Here's how to make it work. ## Why self-host? Three reasons keep coming up: **Privacy and compliance.** Healthcare, legal, finance — regulated industries can't always send data to cloud APIs. HIPAA, GDPR, SOC2 audits get a lot simpler when prompts stay on-premise. **Cost at scale.** If you're running thousands of requests per day, local inference can be 10-50x cheaper than cloud APIs. The break-even point is lower than most people think. **Latency control.** No network round-trips. No provider rate limits. Your GPU, your queue. ## Hardware requirements Local LLMs need serious compute. Here's what actually works: #### Consumer hardware (hobbyist/dev) | Setup | VRAM | Models | Performance | |-------|------|--------|-------------| | RTX 4090 | 24GB | 7B-13B at full precision, 70B quantized | ~40 tok/s on 7B | | RTX 4080 | 16GB | 7B at full precision, 13B quantized | ~30 tok/s on 7B | | M2/M3 Max | 32-96GB unified | Up to 70B with offloading | ~20 tok/s on 7B | A single RTX 4090 (~$1,600) handles most local use cases. For 70B+ models at reasonable speed, you need multiple GPUs or cloud rentals. #### Production hardware (enterprise/heavy use) | Setup | VRAM | Models | Performance | |-------|------|--------|-------------| | 2x RTX 4090 | 48GB | 70B quantized | ~25 tok/s | | A100 40GB | 40GB | 70B at 4-bit | ~50 tok/s | | 2x A100 80GB | 160GB | 70B at full precision | ~80 tok/s | | H100 80GB | 80GB | 70B at 8-bit | ~120 tok/s | #### GPU rental vs ownership **Buy** if you're running inference 8+ hours daily. A 4090 pays for itself in 3-6 months versus cloud API costs at moderate volume. **Rent** for burst capacity or experimenting. Options: - **RunPod**: $0.44/hr for RTX 4090, $1.99/hr for A100 - **Vast.ai**: Variable pricing, often cheaper - **Lambda Labs**: $1.25/hr for A100, good availability At 1000 requests/day with a 70B model, cloud APIs run $30-100/day. A rented A100 at $2/hr costs $48/day and handles the same load with room to spare. ## Best local models by use case Not all open-source models are equal. Here's what works for different tasks: #### General assistant **Llama 3.3 70B** — Meta's best open model. Strong reasoning, good instruction following. Needs 40GB+ VRAM for decent quantization. **Qwen3 Coder** — Alibaba's latest. Strong at coding and multilingual tasks with 262K context. Similar hardware requirements. **Mistral Large** — Good balance of capability and speed. The 123B version competes with GPT-4. #### Coding **DeepSeek Coder V3** — Purpose-built for code. Handles completions, debugging, and multi-file edits well. The 33B version runs on a single 4090. **CodeLlama 70B** — Meta's code-focused variant. Solid for general dev work but showing its age. **Qwen3 32B** — Punches above its weight. Good option if VRAM is tight. #### Document analysis **Llama 3.3 70B** with extended context — Handles up to 128K tokens. Good for legal docs, contracts, research papers. **Mixtral 8x22B** — Mixture-of-experts architecture. Only activates 39B parameters per forward pass, so it's faster than you'd expect. #### Small and fast **Llama 3.2 3B** — Runs on almost anything. Good for simple tasks, routing, and classification. **Phi-3 Mini** — Microsoft's small model. 3.8B parameters, surprisingly capable for its size. ## Setting up Ollama Ollama is the easiest path to local inference. It handles model downloads, quantization, and serves an OpenAI-compatible API. #### Install ```bash # macOS brew install ollama # Linux curl -fsSL https://ollama.com/install.sh | sh # Windows # Download from https://ollama.com/download ``` #### Pull models ```bash # General purpose ollama pull llama3.3:70b-instruct-q4_K_M # Coding ollama pull deepseek-coder-v2:33b # Small and fast ollama pull llama3.2:3b ``` The tag after the colon specifies quantization. `q4_K_M` is a good balance of quality and VRAM usage. Use `q8_0` for better quality if you have the memory. #### Start the server ```bash ollama serve ``` By default, it listens on `http://localhost:11434`. Test it: ```bash curl http://localhost:11434/api/generate -d '{ "model": "llama3.3:70b-instruct-q4_K_M", "prompt": "Hello!" }' ``` ## Docker setup for production For production deployments, containerize everything: ```yaml # docker-compose.yml version: '3.8' services: ollama: image: ollama/ollama:latest ports: - "11434:11434" volumes: - ollama_data:/root/.ollama deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] restart: unless-stopped openclaw: image: openclaw/openclaw:latest depends_on: - ollama environment: - OLLAMA_BASE_URL=http://ollama:11434 ports: - "3000:3000" volumes: - ./openclaw-config:/root/.openclaw restart: unless-stopped volumes: ollama_data: ``` Start it: ```bash docker-compose up -d ``` Pre-pull models in the container: ```bash docker exec -it ollama ollama pull llama3.3:70b-instruct-q4_K_M ``` ## Integrating with OpenClaw Add Ollama as a custom provider in `~/.openclaw/openclaw.json`: ```json { "models": { "mode": "merge", "providers": { "ollama": { "baseUrl": "http://localhost:11434/v1", "apiKey": "ollama", "api": "openai-completions", "models": [ { "id": "llama3.3:70b-instruct-q4_K_M", "name": "Llama 3.3 70B", "reasoning": false, "input": ["text"], "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }, "contextWindow": 128000, "maxTokens": 4096 }, { "id": "deepseek-coder-v2:33b", "name": "DeepSeek Coder V2", "reasoning": false, "input": ["text"], "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }, "contextWindow": 64000, "maxTokens": 4096 } ] } } }, "agents": { "defaults": { "models": { "ollama/llama3.3:70b-instruct-q4_K_M": { "alias": "llama-local" }, "ollama/deepseek-coder-v2:33b": { "alias": "coder-local" } } } } } ``` Apply the config: ```bash openclaw gateway config.apply --file ~/.openclaw/openclaw.json ``` Now switch models in OpenClaw: ``` /model llama-local ``` ## Performance benchmarks Real-world numbers from testing common agent tasks: #### Latency (time to first token) | Model | Local (4090) | Cloud API | |-------|--------------|-----------| | 7B | 50ms | 200-400ms | | 13B | 80ms | 300-500ms | | 70B | 200ms | 400-800ms | Local wins on latency every time. No network overhead. #### Throughput (tokens per second) | Model | Local (4090) | Local (A100) | Cloud API | |-------|--------------|--------------|-----------| | 7B | 40 tok/s | 80 tok/s | 50-100 tok/s | | 13B | 25 tok/s | 60 tok/s | 40-80 tok/s | | 70B (q4) | 12 tok/s | 50 tok/s | 30-60 tok/s | Cloud APIs can be faster for large models if you don't have top-tier hardware. But you're paying per token. #### Cost comparison (1M tokens/day) | Setup | Monthly cost | |-------|--------------| | Claude Sonnet 4 | ~$540 | | GPT-4o | ~$300 | | Local 4090 (owned) | ~$40 (electricity) | | Local 4090 (rented) | ~$320 | | A100 (rented) | ~$600 | The math favors local if you own the hardware and have consistent volume. Renting only makes sense for burst workloads or testing. ## When to use local vs cloud #### Use local when: - **Compliance requires it.** Data can't leave your network. Full stop. - **Volume is high.** 500K+ tokens/day makes local cost-effective. - **Latency matters.** Real-time applications benefit from no network hops. - **You need predictability.** No rate limits, no provider outages. #### Use cloud when: - **You need frontier capabilities.** Claude Opus 4.5 and GPT-4 are still ahead of open-source for complex reasoning. - **Volume is low.** Under 100K tokens/day, cloud is simpler. - **Burst capacity.** Spinning up GPUs for a one-time project isn't worth it. - **You want zero maintenance.** No driver updates, no OOM errors, no thermal throttling. #### Hybrid approach Most people land on a mix: local models for simple stuff, cloud APIs for the hard problems. OpenClaw makes this easy. Configure both your local Ollama instance and a cloud provider like [Haimaker](https://haimaker.ai), then switch between them as needed: ```json { "models": { "providers": { "ollama": { "baseUrl": "http://localhost:11434/v1", "apiKey": "ollama", "api": "openai-completions", "models": [ { "id": "llama3.3:70b-instruct-q4_K_M", "name": "Llama 3.3 70B Local" } ] }, "haimaker": { "baseUrl": "https://api.haimaker.ai/v1", "apiKey": "${HAIMAKER_API_KEY}", "api": "openai-completions" } } }, "agents": { "defaults": { "models": { "ollama/llama3.3:70b-instruct-q4_K_M": { "alias": "local" }, "haimaker/claude-sonnet-4": { "alias": "cloud" } } } } } ``` In chat, switch models with `/model local` or `/model cloud`. Use local for quick questions and drafts, cloud when you need stronger reasoning or tool use. Haimaker gives you access to Claude, GPT-4, Gemini, and a bunch of cheaper alternatives through one API. Useful when your local model hits its limits or you're away from your hardware. ## Troubleshooting #### Out of memory errors Reduce batch size or use a smaller quantization: ```bash # Switch from q4 to q3 ollama pull llama3.3:70b-instruct-q3_K_M ``` Or enable partial GPU offloading in Ollama's modelfile. #### Slow generation Check GPU utilization with `nvidia-smi`. If it's not maxed out, the bottleneck is probably CPU or memory bandwidth. On Apple Silicon, ensure you're using the Metal backend. #### Model not found in OpenClaw Make sure the model name in your config matches exactly what Ollama reports: ```bash ollama list ``` Use the full name including the tag. ## Getting started 1. Install Ollama and pull a model 2. Add it to your OpenClaw config 3. Test with simple prompts 4. Scale up hardware as needed Start small. A 7B model on consumer hardware is enough to validate your workflow before investing in bigger iron. EXPLORE HAIMAKER --- *For more on configuring custom providers in OpenClaw, see our guide on [integrating custom LLM providers](/blog/integrating-custom-llm-providers-with-openclaw). Visit [openclaw.ai](https://openclaw.ai) to get started with your own AI agent.*