The constraint
Everything on this site that thinks runs on one laptop in my home: an RTX 4060 Laptop GPU with 8 GB of memory, of which vLLM sees 7.62 GiB. Two models live on it all the time:
- Autocomplete: Qwen3.5-0.8B-Base, which suggests the next words while I write these notes in the admin editor.
- Ask: Qwen3.5-4B (4-bit AWQ), which answers visitors' questions about my work on the home page.
The plan only worked once I stopped thinking of the GPU as one thing and started treating its memory as a budget.
Time-slicing shares the clock, not the memory
Kubernetes normally hands out a GPU whole: one pod gets it, the next one waits. The NVIDIA device plugin can instead advertise one card as two "slices", and I turned that on. It is easy to read too much into that. Time-slicing only tells the scheduler that two pods may share the card. The two processes then take turns on the compute, but nothing divides the memory between them. Whichever one asks for memory first gets it, and the other runs out.
So the real work was making both models promise, up front, how much they would take.
Each vLLM gets a fixed slice
vLLM grabs memory at startup: weights, a peak activation estimate, CUDA graphs, then whatever is left of its share as KV cache, the memory that holds the context of the requests in flight.
Autocomplete was set to 45% of the card. I measured what it actually needs before it can hold a single token of context: 1.68 GiB for weights and CUDA, 0.57 GiB of peak activations, and 0.04 GiB of CUDA graphs, 2.29 GiB in all. At 30%, it would have had exactly zero bytes of KV cache. At 35% it keeps about 19,600 tokens of KV cache, far more than a few hundred tokens of context per suggestion need, and its real footprint dropped from 3,136 MiB to 2,322 MiB.
Later I gave it a fixed KV cache instead, 0.42 GiB, about 19,000 tokens. With a ratio, vLLM sizes the cache from whatever it finds free at startup: when both models restarted together, autocomplete measured while the ask model was still loading, found no room and crashed, then on the retry took 0.86 GiB and left the card 374 MiB. With a fixed cache its share is only a startup check that enough is free, now 31%.
Ask got the rest. Here, a ratio did not work: vLLM's startup estimate of the activation peak for a 16k-token prompt was too pessimistic, and it refused to start. Giving it a fixed KV cache of 0.65 GiB instead did work. Answering 40 test questions, the whole card peaked at 6,853 MiB with both models loaded.
When three new notes pushed the prompt to 92% of its budget, I raised the cache to 1.04 GiB and the limit to 24k tokens. Four 12.5k-token prompts and one of 17.9k sent at once all succeeded, with the card peaking at 7,243 MiB.
The architecture decided which model fit
I compared three candidates for the ask model. Two of them were both 4B and 4-bit, and on paper they looked the same:
| | KV cache per token | 16k context needs | |---|---|---| | Qwen3-4B-Instruct (standard attention, 36 layers) | 147 KB | 2.25 GiB: did not fit | | Qwen3.5-4B (8 of 32 layers full attention, the rest linear) | 32 KB | 0.61 GiB |
Qwen3.5 only keeps a growing KV cache in a quarter of its layers. On an 8 GB card, that is the difference between serving a 16k-token prompt and not fitting an 8k one. Another surprise: 4-bit quantization only covered the transformer layers. The 248k-token vocabulary's embedding stayed in BF16 and was a third of the weights.
When something needs the whole card
Now and then a batch job needs the entire GPU, like model training. Instead of stopping the services by hand, I gave every GPU workload a priority:
| PriorityClass | Value | Who | |---|---|---| | gpu-batch | −10 | training and other batch jobs | | gpu-autocomplete | −50 | autocomplete | | gpu-ask | −100 | ask, the first to make room |
When a batch job asks for both slices, the scheduler evicts the two vLLM pods. They exit, the memory comes free, and the job has the card to itself. When it finishes, the Deployments bring the models back on their own.
The values are negative on purpose. Every pod without a PriorityClass (the database, the API, Redis) has priority 0, and preemption picks any lower-priority pod that frees enough room, not only GPU pods. A batch job at +1000 that was short on system RAM could evict the database. At −10, it can only evict the two models.
In a test, the job started 3 seconds after I submitted it, ran alone at 5,565 GFLOP/s, and both models were back and ready 2 minutes and 10 seconds after it finished.
What it cost
The ask model's KV cache holds 29,676 tokens: one 24k-token prompt, or four questions sharing the cached documents. Everything on this site goes into its prompt (why not retrieval), so that room is spoken for: a conversation remembers only its last two turns, and every new note brings the prompt closer to the limit. I also stopped at time-slicing plus priorities. Tools that cap each pod's memory (HAMi) or split the card in hardware (MIG, which the 4060 doesn't have) are what you need when several tenants must truly share a GPU at the same time. Here, the rule is simpler: only one side uses the card at a time, or both stay inside budgets I measured.