LLM Setup

My Setup for LLMs

As of 9/12/2026

Cloud LLMs

To get the cloud side of things out of the way, I primarily use ChatGPT from OpenAI for cloud-based LLMs. This gives me frontier intelligence with reasonable rate limits, along with Codex (CLI/Extension). With this, I get more usage than I would with Claude, and it doesn’t refuse every little benign thing I ask it to do.

On top of that, I have the free tiers of the other labs, plus a Pro Google subscription (free for students) that I rarely use. Having used Gemini’s paid tier, I do not understand why anyone would pay money for this. At least Anthropic models are smart for the comparatively limited amount you can use them.

Local AI: Running Models

To run models locally, I use Ollama in Docker. It allows me to have the models I want with little maintenance overhead, and integrates nicely with my maintenance scripts. In Docker Compose, it looks something like this:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ./ollama:/root/.ollama
    environment:
      - OLLAMA_CONTEXT_LENGTH=131072
      - OLLAMA_NUM_PARALLEL=4
      - OLLAMA_MAX_LOADED_MODELS=1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

As I tend to run smaller models, I use the default f16 KV Cache type. This is because my experience with quantizing the KV Cache of small models tends to ruin their performance. However, for running larger models, I will sometimes bump that down to q8_0 to be able to use longer context windows.

Local AI: Chats

For my local AI chats, I use Open WebUI hosted in Docker. This is combined with Apache Tika (content extraction), Calibre (textbooks), and ForgeJo (notes) for my local RAG setup. To sync Calibre and the Open WebUI knowledge base, I run a Python script on a cron job to export content updates, which is then picked up by oikb to import into the knowledge base. On top of that, for internet searching, I run a SearXNG instance. Additionally, my local LLMs are given a tool which unifies Playwright and FlareSolverr for fetching the websites found by SearXNG and converting them into a readable format. I have found that Playwright often breaks in my setup, so the redundancy helps things “just work” for me. This is sort of a union of my configurations, as it comes from multiple machines:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    depends_on:
      - ollama
    ports:
      - "3000:8080"
    volumes:
      - ./open-webui:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - USER_AGENT="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/148.0 Safari/537.36"
      - WEB_LOADER_ENGINE=playwright
      - PLAYWRIGHT_WS_URL=ws://192.168.1.46:3002  # remote machine
      - ENABLE_PYODIDE_FILE_PERSISTENCE=true  # Fix for my proxy setup
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped

  tika:
    image: apache/tika:latest
    container_name: tika
    restart: unless-stopped

  oikb:
    image: ghcr.io/open-webui/oikb:latest
    container_name: oikb
    depends_on:
      - open-webui
    command: ["daemon", "--port", "8080"]
    env_file: ".env"
    volumes:
      - ./books:/sources/books:ro
      - ./notes:/sources/notes:ro
      - ./oikb.yaml:/app/.oikb.yaml:ro
    environment:
      - OPEN_WEBUI_URL=http://open-webui:8080
      - OPEN_WEBUI_API_KEY=${OPEN_WEBUI_API_KEY}
    restart: unless-stopped

  playwright:  # Remote
    image: mcr.microsoft.com/playwright:v1.60.0-noble
    container_name: playwright

    command:
      - /bin/sh
      - -c
      - >
        npx -y playwright@1.60.0
        run-server
        --port 3000
        --host 0.0.0.0
    ports:
      - "3002:3000"

  searxng-core:  # Remote
    container_name: searxng-core
    image: docker.io/searxng/searxng:latest
    restart: unless-stopped
    ports:
      - 8006:8080
    volumes:
      - ./searxng/core-config/:/etc/searxng/:Z
      - ./searxng/core-data:/var/cache/searxng/
      - ./searxng/useragents.json:/usr/local/searxng/searx/data/useragents.json:ro

  searxng-valkey:  # Remote
    container_name: searxng-valkey
    image: docker.io/valkey/valkey:9-alpine
    command: valkey-server --save 30 1 --loglevel warning
    restart: unless-stopped
    volumes:
      - ./searxng/valkey-data:/data/

  flaresolverr:  # Remote
    image: ghcr.io/flaresolverr/flaresolverr:latest
    container_name: flaresolverr
    environment:
      - LOG_LEVEL=info
      - LOG_FILE=none
      - LOG_HTML=false
      - CAPTCHA_SOLVER=none
      - TZ=America/New_York
      - LANGUAGE=en_US:en
    ports:
      - "8191:8191"
    volumes:
      - ./flaresolver:/config
    restart: unless-stopped

Local AI: Code

As I’m already running Ollama and have Codex CLI installed, I simply point my Codex CLI at my Ollama server for local AI agents. Thus far, I’ve yet to find a model that works particularly well for this setup and my needs. My profile is set up like this, with the context window following whatever I have configured in Ollama.

model = "qwen3.8:27b-mtp-q4_K_M"
model_provider = "sleepy_ollama"
model_reasoning_effort = "max"
model_context_window = 131072

# approval_policy = "on-request"
approval_policy = "never"  # Currently this for benchmarking

sandbox_mode = "workspace-write"
web_search = "live"
experimental_multi_agent = true

[sandbox_workspace_write]
network_access = true

[model_providers.sleepy_ollama]
name = "Sleepy Ollama"
base_url = "http://10.8.0.4:11434/v1"
wire_api = "responses"

So far, qwen3.6:35b-a3b-q4_K_M has been the closest to usable model for me. It’s fast enough to not be waiting forever for a response, but too dumb to not closely watch what it’s doing.