backend/stub-service
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity

docs: rework LLM adapter to use local CPU-only models via Ollama #1

Open
backend wants to merge 1 commits from feature/adr-001-local-llm-adapter into main
pull from: feature/adr-001-local-llm-adapter
merge into: :main
Conversation 0 Commits 1 Files Changed 1
+35 -19
backend commented 2026-05-08 07:18:13 +00:00
Owner
Copy Link
Copy Source

Replace cloud-based LLM providers (Anthropic/OpenAI) with a local Ollama sidecar running lightweight quantized models.

  • Primary model: qwen2.5-coder:1.5b (~1 GB RAM, 15-40s generation)
  • Fallback: phi3:mini (3.8B, ~2.3 GB RAM)
  • 60s timeout budget enforced
  • Data stays on-premise, zero API costs

Updated sections: architecture diagram, component description, technology choices, project layout, docker-compose, env vars, consequences.

Replace cloud-based LLM providers (Anthropic/OpenAI) with a local Ollama sidecar running lightweight quantized models. - Primary model: qwen2.5-coder:1.5b (~1 GB RAM, 15-40s generation) - Fallback: phi3:mini (3.8B, ~2.3 GB RAM) - 60s timeout budget enforced - Data stays on-premise, zero API costs Updated sections: architecture diagram, component description, technology choices, project layout, docker-compose, env vars, consequences.
backend added 1 commit 2026-05-08 07:18:14 +00:00
docs: rework LLM adapter section to use local CPU-only models via Ollama d7e007a4ea 
Replace cloud-based LLM providers (Anthropic/OpenAI) with a local Ollama
sidecar running lightweight quantized models (qwen2.5-coder:1.5b as primary,
phi3:mini as fallback). All generation stays under 60s on CPU, data never
leaves the network, and there are zero API costs.

Updated: architecture diagram, component description, technology choices,
project layout, docker-compose, env vars, and consequences sections.
You are not authorized to merge this pull request.
This pull request can be merged automatically.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feature/adr-001-local-llm-adapter:feature/adr-001-local-llm-adapter
git checkout feature/adr-001-local-llm-adapter
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
backend
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: backend/stub-service#1