Open weights models are becoming more and more capable, and many organisations are wondering whether they should deploy and manage their own.
Earlier this year the CEO of IBM said that open weights models are getting close enough to closed models, for a fraction of the cost.
Running your own model also makes sense if you want your data to stay within your company.
The purpose of this article is to find out what it takes to deploy an open weights model behind an API. I’ll be using Kubernetes to manage the deployment and scaling. Once the model is deployed, I’ll use the Pi coding agent to test the model’s coding capabilities and speed.
The hardware
This project runs on my own GPU home lab running Ubuntu Server. The machine has an RTX 2060 with 6 GB of VRAM, which is a fairly old card by today’s standards, but good enough to run some of the smaller models. The rest of the box is a Ryzen 5 3600 (12 threads) and 15 GB of RAM.
For reference, the RTX 2060 is the same generation as the T4 in AWS’s g4dn.xlarge, which costs about $0.53/hour on-demand, or roughly $380/month if left running (for a single instance). The T4 has 16 GB of VRAM though, so it can fit a bigger model than my card.
The VRAM on my graphics card is what limits the model size. I also only have one GPU, which means I can’t scale to more than one instance. In a production environment you’d have several nodes available when you need to scale.
The model
For the model I’ll use Qwen3.5 with 4 billion parameters, quantised to 4 bits (Q4_K_M). It’s a small model that can write code and use tools, which is the kind of model you’d deploy internally for software engineers to use with a coding agent like Pi.
The reason for picking this model is that it’s as much as I can fit in my card’s VRAM. The model file is about 2.7 GB, which gets loaded into the 6 GB of memory. The rest of the VRAM is needed for the KV cache, which holds the context and is what makes responses fast. (I wrote about how KV caching works in a previous post.)
The next size up is the 9 billion parameter model, which sits at about 5.7 GB before accounting for any memory for the KV cache. That would leave no room for context, making it useless on my card.

Memory allocation for my 6GB VRAM GPU
The deployment
The model is deployed as a centralised API service in Kubernetes. Users (me) call it with a URL and an API key to use the model for coding tasks.
I use K3s, a lightweight version of Kubernetes, in my home lab. The deployment consists of a Deployment, a ConfigMap, a Secret, a Service and an HPA (Horizontal Pod Autoscaler), plus a PersistentVolumeClaim to keep the model files between restarts.

Kubernetes resources in this deployment
The Deployment defines the image, storage and configuration used by the pods. At the container level you define which image the pods run, and which environment variables and config to load. For the image I’m using llama.cpp’s llama-server (ghcr.io/ggml-org/llama.cpp:server-cuda), a ready-to-use HTTP server for open weights models. Each pod requests one GPU (nvidia.com/gpu: 1), and the model is downloaded on startup into a volume mounted at /models.
apiVersion: apps/v1
kind: Deployment
metadata:
name: llm
namespace: llm
labels:
app: llm
spec:
strategy:
type: Recreate # one GPU: old pod must release it before the new one starts
selector:
matchLabels:
app: llm
template:
metadata:
labels:
app: llm
spec:
runtimeClassName: nvidia # k3s-specific: use the NVIDIA container runtime
nodeSelector:
homelab/gpu: "true" # only schedule on GPU nodes
tolerations: # allowed onto the GPU node once it's tainted
- key: homelab/gpu
operator: Exists
effect: NoSchedule
containers:
- name: llama-server
image: ghcr.io/ggml-org/llama.cpp:server-cuda
envFrom:
- configMapRef:
name: llm-config
- secretRef:
name: llm-secrets
env:
- name: LLAMA_CACHE
value: /models
- name: LLAMA_ARG_HOST
value: 0.0.0.0
- name: LLAMA_ARG_PORT
value: "8080"
ports:
- name: http
containerPort: 8080
resources:
requests:
cpu: "1" # HPA % is measured against this
memory: 2Gi
limits:
memory: 4Gi
nvidia.com/gpu: 1 # schedule only where a GPU is free
volumeMounts:
- name: models
mountPath: /models
# /health returns 503 while the model loads, 200 when ready.
startupProbe: # allow up to 15 min for first download + load
httpGet: { path: /health, port: http }
periodSeconds: 10
failureThreshold: 90
readinessProbe: # only receive traffic when ready
httpGet: { path: /health, port: http }
periodSeconds: 10
livenessProbe: # restart if it hangs
httpGet: { path: /health, port: http }
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3
volumes:
- name: models
persistentVolumeClaim:
claimName: llm-models
During the first deploy, the node needs to pull the image and download the model. The 2.6 GB image took about 10 minutes to download. Then, once the container starts, llama-server downloads the model, which took almost another 13 minutes. This is why the startup probe needs a generous budget the first time.
The Deployment is coupled with a Service, which is the interface we use to talk to the pods. The Service gives us a single IP and port, and does basic load balancing across the pods running the LLM. In K3s, a Service of type LoadBalancer is exposed on the node’s IP, so the API is available on my home network at 192.168.1.149:8080.
apiVersion: v1
kind: Service
metadata:
name: llm
namespace: llm
spec:
type: LoadBalancer
selector:
app: llm
ports:
- name: http
port: 8080
targetPort: http
The environment variables are loaded from a ConfigMap. This is where the model parameters live: which model to use, the context size, how many GPU layers to offload and so on. llama-server reads any LLAMA_ARG_* environment variable as a command line argument, so the whole model configuration lives in one file. This is the initial setup, using the Qwen3.5 4B model with a context size of 32,768 tokens:
# apps/llm/config/model.env
LLAMA_ARG_HF_REPO=unsloth/Qwen3.5-4B-GGUF:Q4_K_M
# The model "id" clients (Pi) ask for
LLAMA_ARG_ALIAS=qwen3.5-4b
# Total context, split across slots (32768 / 2 = 16384 per request)
LLAMA_ARG_CTX_SIZE=32768
# Concurrent request slots
LLAMA_ARG_N_PARALLEL=2
# Offload all layers to the GPU
LLAMA_ARG_N_GPU_LAYERS=99
# Use the model's chat template (needed for tool calls)
LLAMA_ARG_JINJA=true
# Expose Prometheus metrics at /metrics
LLAMA_ARG_ENDPOINT_METRICS=true
Another interesting setting is LLAMA_ARG_N_PARALLEL, which defines how many requests the model can handle in parallel. Any concurrent requests on top of that wait until earlier ones are finished. The context is also divided between the slots, so each one gets a slice of the total context size: with 2 slots, each request gets 16,384 tokens. Depending on your specs, you can tweak this to fit your needs. Since this is also what sets the number of concurrent users per pod, you’d tune it with stress tests to find the sweet spot of max concurrency an instance can hold, and set up autoscaling based on that.
The next part is the horizontal autoscaling. I’ve set it up to scale up to 2 instances if CPU usage goes over 70% (of the 1 CPU the pod requests). But this is pointless for two reasons.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: llm
namespace: llm
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: llm
minReplicas: 1
maxReplicas: 2
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
behavior:
scaleUp:
stabilizationWindowSeconds: 180 # ignore the CPU spike while the model loads
scaleDown:
stabilizationWindowSeconds: 300 # avoid flapping; model loads are slow
First, CPU usage is not the right metric to track. A better metric would be GPU memory usage or the number of queued requests. However, out of the box, the HPA only scales on CPU and memory (RAM, not GPU memory). To track these more useful metrics we could use KEDA instead, but this is out of scope for this project because of the next reason.
Second, I only have one GPU. Even when autoscaling triggers (which it does), the new pod can’t start because there are no more GPUs available, so it stays in Pending. On the first deploy, the CPU spike from loading the model was enough to trigger it: the HPA scaled to 2 replicas, and scaled back down to 1 six minutes later. I set up the HPA for illustrative purposes, and to check that it does work with these settings.

The HPA in action: the running pod is at 100% of its 1 CPU request, so a second replica is created. With no second GPU, it stays in Pending.
One thing to note about autoscaling, is that the K,V cache is maintaned at the pod level. If you make a request and it goes to one pod, but the next request goes to a different pod you lose the K,V cache and your next response will be very slow. In the context of coding agents, this is important. There are different methods you can use for this that I won’t discuss in this post, such as sticky sessions by IP or session ID, or using a shared cache.
Finally, there’s the Secret, which holds the API key needed to call the API. As everyone using Kubernetes knows, secrets are not really secret: the values are only base64 encoded. In a production environment it’s better to load your secrets from a secrets vault instead.
apiVersion: v1
kind: Secret
metadata:
name: llm-secrets
namespace: llm
type: Opaque
data:
LLAMA_API_KEY: <base64-encoded API key>
Once the model is deployed, we can call the API using the home lab’s IP and port:
export LLM=http://192.168.1.149:8080
curl $LLM/health
# {"status":"ok"}
Testing the API
Before setting up the coding agent, we can test the model by calling the API directly:
curl -s $LLM/v1/chat/completions \
-H "Authorization: Bearer $LLM_KEY" -H "Content-Type: application/json" \
-d '{"model":"qwen3.5-4b","messages":[{"role":"user","content":"Write a Python function that reverses a string."}]}' \
| python3 -m json.tool

The first request to the model. It took 1 min 10 s, and 61 of those seconds were spent processing a 20 token prompt.
The API returns a JSON object that follows the OpenAI API standard. Apart from the answer in message.content, it returns:
- reasoning_content: Qwen3.5 is a thinking model, and this is its reasoning before answering.
- usage: the prompt and completion tokens. Here 20 tokens in and 708 tokens out.
- timings: a llama.cpp extra with the speed of each step. The model generates at about 80 tokens/second, which it does consistently on every request.
The very first request suffers from a cold start, usually taking a whole minute just to process the prompt. Every request after that processed its prompt in well under a second, so this is a one-off cost after the model loads.
We can also call the API with streaming, by adding "stream": true. This is a much more useful approach, since we don’t have to wait for the full answer to be ready. We can read it as it gets generated, as we’re already used to from every LLM app out there.
curl -N $LLM/v1/chat/completions \
-H "Authorization: Bearer $LLM_KEY" -H "Content-Type: application/json" \
-d '{"model":"qwen3.5-4b","stream":true,"messages":[{"role":"user","content":"Count to 10"}]}'
Streaming: the response arrives as server-sent events, one chunk per token. The last chunk carries the timings, followed by data: [DONE].
As we can see, the response is split into many small chunks, one per token: "6", ",", " ", "7". The final chunk has finish_reason: "stop" and the timings, and the stream ends with data: [DONE].
One thing worth knowing is that the API is stateless. Calling it again won’t remember the previous conversation unless we send it again in the next call. This is why context grows so large as conversations get longer and longer: every request carries the whole history.
{"model": "qwen3.5-4b", "messages": [
{"role": "user", "content": "Count to 10"},
{"role": "assistant", "content": "1, 2, 3, 4, 5, 6, 7, 8, 9, 10"},
{"role": "user", "content": "Now count backwards"}
]}
Resending the whole history every time sounds expensive, but the server keeps a prompt cache. If the start of the prompt matches what it has already processed, it only processes the new tokens.
In our case we don’t need to worry about managing the history, because the coding agent handles it for us.
The last piece of the API a coding agent relies on is tool calling. The model never runs anything itself. The agent sends a list of tools it can use (a name, a description and a JSON schema for the arguments) with every request, and the model replies with a structured request to call one of them:
"finish_reason": "tool_calls",
"message": {
"content": "",
"tool_calls": [{
"type": "function",
"function": {"name": "read_file", "arguments": "{\"path\":\"main.py\"}"}
}]
}
The agent runs the tool, sends the result back in the next request, and the loop continues until the model answers in plain text. Note that the tool definitions count towards the prompt: asking “What is in the file main.py?” with a single tool attached was already 273 prompt tokens.
Using the model in a coding agent
To make the model useful, I’ll use the Pi coding agent. Pi is an open source coding agent like Codex or Claude Code, and because our API follows the OpenAI standard, it can connect to it to handle our coding tasks.
To make this similar to a real production setup, I install Pi on my laptop rather than on the machine running the model. This is the same as if you deployed the model on EC2 and ran the agent from your local machine.
Once Pi is installed, add the model to ~/.pi/agent/models.json:
{
"providers": {
"homelab": {
"baseUrl": "http://192.168.1.149:8080/v1",
"api": "openai-completions",
"apiKey": "LLM_KEY",
"models": [
{ "id": "qwen3.5-4b", "contextWindow": 32768, "maxTokens": 8192 }
]
}
}
}
The contextWindow tells Pi when to compact the conversation, so I set it to match the server. apiKey is the name of the environment variable that holds the key, so don’t forget to set it. To make sure the env variable loads automatically, add it to your ~/.bashrc or ~/.zshrc:
export LLM_KEY=...
Then we can run Pi, and if everything went well, we’ll see our model loaded. We can start using it for coding tasks. In this example I ask it to build a simple “hello world” web page.
图表:Pi on my laptop, using the model served from the home lab.
Pi on my laptop, using the model served from the home lab.
When Pi is working, the GPU runs flat out. Each agent turn kept it at 97% utilisation and 158 of its 160 W for around 45 seconds.

One Pi request in nvtop: 97% GPU utilisation and 158 W, the card’s power limit.
Testing the performance
When I first started using the model, my sessions broke down after a few prompts. Coding agents need a generous context size, and a model isn’t very useful if it only allows a handful of prompts before it errors out.

Pi after a few prompts: a connection error, then every response is truncated. The status bar in the bottom left shows 72.2% of the 16k context used.
There were two problems here. The connection error came from the pod restarting mid-session, most likely because it ran out of memory.
The truncated responses were a context problem. The context wasn’t full yet, but there wasn’t enough room left for the model to finish its answers.
When I checked my settings, I found why the context was so small. The total context size was only 32,768 tokens, and LLAMA_ARG_N_PARALLEL was set to 2, which splits the context in half between two slots. So each request only got 16,384 tokens.
16k tokens sounds like a lot: it’s around 12,000 words. But a coding agent burns through it fast. The system prompt and tool definitions go in every request, and so does every file the model reads and every command output. In one session the context doubled from about 6,000 to 12,000 tokens in a few minutes, and the server logs showed all the requests going to slot 0. Slot 1 was never used.
In reality, splitting the context size between n parallel runs is a tradeoff you don’t want to make. You want to be able to run n_parallel runs with a shared K,V cache so you can make the most of the context size. To do this, you can set the number of parallel runs and use the LLAMA_ARG_KV_UNIFIED=True environment variable.
Qwen3.5 supports up to 262K tokens of context, but because the context is held in memory as the KV cache, I’m limited by the amount of VRAM I have. llama.cpp reserves all the memory it needs when the instance starts, so I can use nvtop to check how much memory the model needs. The memory used is the model size + the KV cache for the full context + some compute buffers. Once it’s loaded, it doesn’t grow: a longer conversation gets slower, not bigger.
This is nvtop with my initial setup (2 slots, 32,768 tokens of context):

The initial setup, idle: llama-server holds 4.8 GB of the 6 GB even when nobody is using it, while drawing only 10 W.
Since I’m the only user, I set LLAMA_ARG_N_PARALLEL to 1, which gives one slot with the full context size, and matched Pi’s context window to it. Pi went from 16k to 33k of context, and the session recovered. I can also use LLAMA_ARG_N_PARALLEL=auto to automatically enable the shared K,V cache. But I prefer to be explicit.

The same session after setting LLAMA_ARG_N_PARALLEL=1: the context goes from 16k to 33k, and the model gets back to work.
32k of context is still small for a coding agent, and my memory usage was already pretty close to the limit. What I did next was store the KV cache in 8 bits (q8_0) instead of the default 16 bits (f16), and turn on flash attention. This roughly halves the memory per token of context, so I could double the context size to 65,536, and then push it further to 81,920.
## Optimising context
LLAMA_ARG_CTX_SIZE=81920
LLAMA_ARG_N_PARALLEL=1
LLAMA_ARG_FLASH_ATTN=on
LLAMA_ARG_CACHE_TYPE_K=q8_0
LLAMA_ARG_CACHE_TYPE_V=q8_0
When I redeployed the pods, my changes didn’t show up. I made the rookie mistake of restarting the pods without making sure the ConfigMap was updated. It turned out the ConfigMap in the cluster still had the old settings, and the server logs still showed 2 slots of 16k each. I can’t count the number of times this same issue has happened to me before.

The ConfigMap in the cluster (top) doesn’t match model.env in the repo (middle), and the server log (bottom) confirms the pod is still running 2 slots of 16,384 tokens.
Once the new config was actually applied, nvtop showed it still fits in memory, with about 280 MB to spare:

81,920 tokens of context with an 8-bit KV cache: 5.7 GB of the 6 GB used.
| Setup | Context per request | GPU memory |
|---|---|---|
| 32,768 total, 2 slots, 16-bit KV cache | 16,384 | 5.1 GB |
| 81,920 total, 1 slot, 8-bit KV cache | 81,920 | 5.7 GB |
For an extra 0.6 GB of memory, each request gets 5x the context.
When I run Pi on my laptop with the new config, you can see the larger context size in the bottom left of the screen, which makes the model much more useful.

Pi’s status bar with the new 82k context window.
80k is about as far as I can push the model in my case. It is still not very much context size, about a third of the total context this model can take, but it still lets me run some coding tasks as long as I don’t make conversations too long or try to run very complex tasks. For a truly useful coding agent I’d want to use a larger model deployed in an instance that can handle the model and context size.
After using the model for some tasks, it feels fine. It is capable of doing most of the tasks I ask it, as long as they are not very complex. The main thing I’ve noticed is that it takes my prompts too literally. This one is tricky for models to understand, but just because I ask for something, it doesn’t mean I want that exactly. In some cases I want the model to take some liberties to do the right thing, rather than the exact thing mentioned. This is something I feel like frontier models such as Claude Sonnet or Opus are much better at.
Scaling this for production
With one GPU I couldn’t really scale anything, but the setup makes it clear what you’d need to think about for a real deployment.
Concurrency is slots x replicas. Each pod can serve N_PARALLEL requests at once, and anything above that queues. But the context is split between slots, so more users per pod means less context for each of them on the same GPU. For coding agents, which need a lot of context, it’s benefitial to use a shared cache across runs to maximise available context size.
Scale on queued requests, not CPU. llama-server already exposes Prometheus metrics at /metrics, including how many requests are processing and how many are waiting. That’s the signal to autoscale on, with something like KEDA. CPU tells you almost nothing: the only time it spiked on my setup was while loading the model.
Scaling up is slow. A new pod on a fresh node needs to pull a 2.6 GB image and download the model before it can serve a single request. On my first deploy that took over 20 minutes. In production you’d want images pre-pulled and models on shared storage, so a new replica only has to load the model into memory.
Idle replicas still cost a whole GPU. The memory is reserved at startup, so an idle replica holds its GPU whether anyone is using it or not. On AWS, every replica running 24/7 on a g4dn.xlarge is roughly $380/month. Size the number of replicas on actual usage (requests per second, concurrent users), not just on memory allocation.
Monitoring and access. You’d want to track queued requests, time to first token, tokens per second and GPU memory, and put a gateway in front of the API for per-user keys and rate limits.
Lessons learned from this project
Overall, I was limited by my hardware. With 6 GB of VRAM I’m not really able to deploy a very useful LLM for coding. When I started this project, I wanted to learn what it takes to deploy an open weights model behind an API. What I found out is that this is the easy part.
Like with most systems in production, the tricky part is getting the node size, model size, context size, number of instances and autoscaling right, so the system runs smoothly and efficiently for its users. That can only be achieved by testing these models with real usage and having good monitoring of resource usage and SLAs (e.g. are there queued requests?), which give us enough insight to improve the deployment over time.
Even though I wasn’t able to try the larger, more capable models, I can already see how capable open models are at tackling many coding tasks if used well. They are not as good as the frontier models from OpenAI or Anthropic, but that may not be needed for many everyday tasks.
About the author: I'm Pablo Lopez Santori, a senior Data Platform and AI engineer based in Edinburgh, UK. I build scalable, distributed platforms and AI systems in production, from data infrastructure to applied AI. If you have a problem I can help with, I'm available for contract work, mostly remote and open to hybrid in Edinburgh.
Get in touch: Email · LinkedIn · GitHub