Multi-model chat — local and cloud LLMs, one interface
A self-hosted chat that routes prompts across local and cloud models — proving out the inference-on-your-own-hardware pattern I bring to client work.
Architecture
The idea
There’s a real engineering decision hiding inside every LLM feature: which model, and where does it run? Local inference is private and cheap at steady state but slow to scale; cloud models are more capable but send data off-box and cost per token. Most teams pick one and stick with it — and get the worst of the tradeoff.
What I’m building
A self-hosted chat that makes the choice explicit instead of invisible. A router layer looks at each request and picks a backend: the local GPU node for privacy-sensitive or high-volume work, a cloud model when the task genuinely needs more capability. The interface is one consistent stream either way, and every exchange is logged so I can compare quality, latency, and cost per route.
The point of the exercise is having a measured opinion instead of a guess: you can’t give a sensible answer about “local vs. cloud” if you’ve never run both and logged the difference yourself.
Status
In progress — the routing and local-inference pieces are working; the evaluation dashboard is next. When it’s live it gets embedded right here as an interactive demo.