Menu
Nederlands

Self-hosted language models and private AI environments

Sometimes data simply can't go to an external AI provider. Then the models come to your data: on your own GPUs or in a private environment, as easy to use as a public API.

Tell me about your project

The problem

Open models have become good enough for many business tasks, but running them well is a job of its own. GPUs are expensive, a model that's loaded but idle still occupies memory, and everyone wants access from their own tools.

Without a shared setup, everyone runs their own experiments, API keys end up in scripts, and nobody knows what's being used or what it costs.

How I approach it

  1. Pick the right models

    We test open models on your actual tasks and choose the smallest one that does the job well.

  2. One API for everything

    An OpenAI-compatible gateway in front of all models, local and external, with a key per user or application.

  3. Use the hardware efficiently

    Models start when they're needed and stop when they're idle, and a job queue shares the GPUs between training runs and model servers.

  4. Logging and limits

    Usage per user and application, spend limits for external providers, and caching where it helps.

  5. Connect the tools

    Chat interfaces, coding assistants and your own applications all talk to the same endpoint.

What to watch out for

  • Self-hosting isn't automatically cheaper. At low volume, an EU provider under a processing agreement is often the better deal.
  • A gateway holds the keys to everything. Treat it as critical infrastructure: pin versions, limit keys and follow security advisories.
  • Open models differ a lot per task and per language. If your documents are in Dutch, test in Dutch.

Projects like this

  • Software company
  • 2026

Local language models, only when the team needs them

The team wanted to use language models without sending data to an outside provider. But a model on a local GPU holds on to a large chunk of GPU memory, even when nobody is using it. Now the team has one API for all local models. An agent on each GPU machine starts a model on the first request and stops it after an hour without requests, which frees the GPU for other work.

  • One OpenAI-compatible API for all local models, which team members connect their chat and coding tools to.
  • A dashboard shows which models are running, and admins can start or stop any of them.
  • The models run on GPU machines the company already owns. No data goes to an outside provider.
  1. Request from a chat or coding tool
  2. Shared API
  3. Start the model if needed
  4. Answer from the local model
  • Financial services company
  • 2025

A private AI workspace with many agents

The team wanted AI assistants that don't send data to a public chat app. Now they have a self-hosted chat environment with specialised agents, plus a separate API layer that logs usage and caches answers.

  • A self-hosted chat environment with agents for different tasks.
  • An API layer between the agents and the model providers that logs usage and caches answers.
  • Everything runs on infrastructure the client already had.
  1. Questions and documents from the team
  2. Specialised agents
  3. Layer for logs and cache
  4. Answers inside the client's own environment
  • Software company
  • 2026

A job queue for shared GPU machines

A team of engineers and researchers shared a handful of GPU machines and used a group chat to agree who got which one. Now everyone submits a Docker job to a queue, which starts it on a machine with free resources. A web interface shows the queue, the logs and how busy each GPU is.

  • Jobs are placed by the GPU memory they need, so small jobs can share a GPU and less GPU time goes to waste.
  • Each job runs as the person who submitted it and can only access the folders the rules allow.
  • Users can open a shell in their job. A job can also start a model server for other jobs, which shuts down once those jobs finish.
  • Coding agents can submit jobs and read the results too, through an MCP server.
  1. Docker job from a team memberFree resources on each machine
  2. Select a machine
  3. Run the job as its owner
  4. Logs and GPU use in the web interface

Frequently asked questions

What hardware do we need?

That depends on the models and the volume. Smaller models run on a single workstation GPU; larger ones need server GPUs or a private cloud. We size this after testing on your tasks.

Can we combine local and external models?

Yes. One gateway can send sensitive work to local models and everything else to an external provider.

Who maintains it?

I can set it up and hand it over to your team with documentation, or keep running it for you.