VLLM on Intellectual Guides

Daily Graduate-Concept Fable: Speculative Decoding

Mon, 17 Aug 2026 00:00:00 +0000

Good evening! Here is today’s “Daily Graduate-Concept Fable.”

The Clever Apprentice of the Royal Scriptorium

At the heart of the empire’s capital, the Royal Scriptorium faced a crisis unlike any before.

The empire’s Grand Scholar, “Balthazar,” was the only person alive with the wisdom to draft the sacred legal code. He had a photographic memory, and his mind held ten million ancient texts. Yet he had one fatal flaw: he wrote far too slowly.

The scholar’s rigor bordered on obsession. For every single word he wrote, he would leaf through countless heavy tomes in the library of his mind, deliberating carefully before slowly setting the word to the page. Having written that word, he would then reconsider everything he had just written to work out the next one. As a result, he could produce only one word per minute. The front lines desperately needed new decrees, yet the scholar could only squeeze them out one word at a time.

At this point, the scriptorium had a young apprentice named “Leo.” Leo’s learning was nowhere near the scholar’s — the books in his head amounted to less than one percent of the scholar’s — but he had one strength: he was extremely quick, and he was very good at guessing the scholar’s mind.

After observing for a few days, Leo made a bold proposal to the scholar.

“Master, rather than agonizing over every word from scratch, why don’t we change how we work?” said Leo. “Let me be your ‘drafter.’ In one go, I’ll guess the next five words you’re likely to write and quickly jot them on a slate. Then you simply look at the slate.”

The scholar frowned. “Absurd! Your learning is shallow — if the words you write are wrong, they’ll ruin the dignity of the code!”

“Master, you misunderstand,” Leo explained with a smile. “Conceiving a word from nothing costs you enormous effort hauling tomes through your mind; but glancing over a few words and judging whether they match your intent is effortless and lightning-fast for you. If you find the words I’ve written are correct, you simply stamp them with your seal. The moment you spot a single word that differs from what you had in mind, you erase that word and everything after it, write in the correct word yourself, and then I resume guessing from the word you corrected.”

Half-doubting, the scholar agreed.

The work began. Leo swiftly wrote on the slate: “The empire / shall / levy / taxes / on the north.”

The scholar swept his eyes across it, his mind instantly completing the verification — the five words fit his line of thought perfectly! He immediately stamped his great seal. A sentence that would once have taken five agonizing minutes was now done in seconds.

Leo then quickly wrote: “the tax rate / is / annually / fifty / gold / coins.”

The scholar glanced at it, frowned, and pointed to the fourth word: “Wrong. Not fifty — ‘one hundred.’”

With a sweep of his brush, the scholar erased “fifty / gold / coins” and wrote in “one hundred” with his own hand.

Then, looking at the word “one hundred,” Leo once more began his rapid guessing and drafting.

Although Leo sometimes guessed wrong, he hit most ordinary phrasings precisely. Most remarkable of all, the final quality and wording of the code turned out to be “exactly identical” to what the scholar would have labored over one word at a time — with no compromise whatsoever — yet the overall drafting speed increased several times over.

The Reveal: Speculative Decoding

What this story describes is an exceptionally elegant and pivotal graduate-level algorithm from the field of large-language-model (LLM) inference acceleration: Speculative Decoding. It is also one of the core techniques used by modern high-performance inference engines (such as vLLM) to break through hardware performance bottlenecks.

In traditional LLM text generation, the model must generate autoregressively — one token after another. For large models with tens of billions of parameters (say, the Qwen 35B class), generating each single token requires hauling tens of gigabytes of weights from GPU memory to the compute cores. This means the bottleneck of large-model inference isn’t compute at all, but memory bandwidth — just as, in the story, the scholar has to laboriously haul heavy tomes to write a single word.

Speculative decoding cleverly breaks this deadlock:

It introduces a tiny, extremely fast draft model (the apprentice, Leo) to quickly generate several consecutive tokens in one go.
It then hands these generated tokens to the large, precise target model (the scholar) for a single round of parallel verification.
Because the target model, when verifying the given tokens, can fully exploit the GPU’s massive parallel compute (compute-bound), it can judge in an instant whether these tokens’ probability distribution matches its own output.
If accepted, this amounts to generating multiple tokens in a single forward pass; if a token is rejected, the model corrects it at that point, discards the subsequent guesses, and starts over.

The most fascinating mathematical property is this: through a specific verification rejection-sampling algorithm, speculative decoding can guarantee that the final output’s probability distribution is mathematically “100% equivalent” to the distribution the target model would have produced on its own, slow generation — with no loss of generation quality whatsoever.

Mapping the Metaphor

Element in the story	LLM inference concept	Explanation
Grand Scholar “Balthazar”	Target Model	The large, highly accurate language model with a huge parameter count.
Clever apprentice “Leo”	Draft Model / Proposer	A tiny, extremely fast small language model (or recent multi-head prediction architectures like MTP / Medusa).
Hauling heavy tomes to write	Memory-bandwidth bound	The latency caused by hauling model weights in traditional autoregressive generation.
Glancing over the sentence on the slate	Parallel Verification	The target model uses its GPU compute advantage to evaluate multiple tokens’ probabilities in a single forward pass.
Stamping the seal / erasing and rewriting	Acceptance/Rejection Criterion	If the draft model’s prediction falls within the target model’s probability tolerance, it is accepted; otherwise the model rolls back from the erroneous point and the target model supplies the correct token.
Quality unchanged	Lossless Generation	Speculative decoding’s most important mathematical guarantee: it does not alter the target model’s original output probability distribution.

Local AI Infrastructure Setup Guide: From Zero to vLLM + ComfyUI

Mon, 27 Jul 2026 00:00:00 +0000

Why Build Local AI Infrastructure?

In 2026, AI tools have become essential infrastructure for developers, creators, and enterprises. But most people still rely on cloud APIs — paying per call, risking data privacy, and hitting rate limits.

Local AI infrastructure gives you three things: full control, unlimited usage, and zero marginal cost.

Once set up, you can:

Use LLMs without limits, no API fees
Process sensitive data without leaving your machine
Run multiple services simultaneously, no rate limits
Integrate with frameworks like Cowork MCP for true multi-agent systems

This guide assumes you have a machine with a GPU (DGX GB10, RTX 4090, or better recommended). I’ll walk you through building a complete local AI environment from scratch.

What you’ll learn:

Install and configure local LLM serving (vLLM)
Deploy ComfyUI for image generation
Manage GPU memory and service startup order
Automate service management (systemd)
Integrate with Cowork MCP for multi-agent AI teams

Let’s get started.

Prerequisites

Before we begin, make sure you have:

Item	Version	Check Command
Linux	Ubuntu 22.04+ or Arch	`uname -a`
GPU	NVIDIA RTX 3090+/4090/GB10	`nvidia-smi`
CUDA	≥ 12.4	`nvcc --version`
Docker	≥ 24.0	`docker --version`
Node.js	≥ 20	`node --version`
Python	≥ 3.10	`python3 --version`

If your GPU has less than 24GB memory, consider closing unnecessary GUI apps or services before starting.

Step 1: Install Docker and NVIDIA Container Toolkit

Docker is the recommended way to deploy local AI services because it isolates dependencies, avoids version conflicts, and prevents system pollution.

# Install Docker
sudo apt update
sudo apt install -y docker.io docker-compose-v2

# Install NVIDIA Container Toolkit (enables GPU access in Docker)
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU availability:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If you see GPU info and memory usage, your environment is correctly configured.

Step 2: Deploy vLLM Local Language Model

vLLM is currently the most efficient local LLM inference framework, supporting multiple model formats (GGUF, GPTQ, AWQ).

2.1 Deploy with Docker

docker run -d \
 --name vllm-server \
 --gpus all \
 -p 8000:8000 \
 -v ~/.cache/huggingface:/root/.cache/huggingface \
 vllm/vllm-openai:latest \
 --model "nvidia/Llama-3.1-Nemotron-70B-Instruct" \
 --gpu-memory-utilization 0.9 \
 --max-model-len 8192

Parameter explanation:

--model: The model to deploy (replace with any HuggingFace model)
--gpu-memory-utilization: Ratio of GPU memory to use (0.9 means 90%)
--max-model-len: Maximum context length

Note: If your GPU has less than 80GB memory, start with smaller models (7B or 13B) before trying larger ones.

2.2 Test the API Endpoint

curl -X POST http://localhost:8000/v1/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "nvidia/Llama-3.1-Nemotron-70B-Instruct",
 "prompt": "Explain what a multi-agent AI system is",
 "max_tokens": 200
 }'

If you receive a response, vLLM is successfully deployed.

2.3 Setup as System Service (Auto-start)

Create a systemd service file:

sudo tee /etc/systemd/system/vllm.service << 'EOF'
[Unit]
Description=vLLM Local LLM Server
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/docker start vllm-server
ExecStop=/usr/bin/docker stop vllm-server

[Install]
WantedBy=multi-user.target
EOF

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable vllm.service
sudo systemctl start vllm.service

Step 3: Deploy ComfyUI Image Generation

ComfyUI is the most powerful local image generation tool, supporting Flux, Stable Diffusion, LTX, and various other models.

3.1 Deploy with Docker

docker run -d \
 --name comfyui \
 --gpus all \
 -p 8188:8188 \
 -v ~/.cache/huggingface:/root/.cache/huggingface \
 -v ~/comfyui-output:/output \
 ghcr.io/ai-forest/comfyui:latest

This starts ComfyUI at http://localhost:8188.

3.2 Manual Installation (Advanced Option)

If you need more control, you can install manually:

# Clone ComfyUI repository
cd ~/comfyui
git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI

# Install dependencies
pip install -r requirements.txt

# Install custom nodes (optional)
git clone https://github.com/Fannovel16/ComfyUI-Frame-Interpolation.git custom_nodes/
git clone https://github.com/laksjdjoy/deforum-comfy.git custom_nodes/

# Start the service
python main.py --listen 0.0.0.0 --port 8188

3.3 Download Common Models

# Use HuggingFace CLI to download models
pip install huggingface_hub
huggingface-cli download stabilityai/stable-diffusion-xl-base-1.0 --local-dir ~/models/sdxl
huggingface-cli download stabilityai/stable-diffusion-3.5-large --local-dir ~/models/sd3.5

Step 4: GPU Memory Management Strategies

When running AI services locally, GPU memory is the biggest bottleneck. Here are some battle-tested practices:

4.1 Check Available Memory

free -h
nvidia-smi --query-gpu=memory.used,memory.free,temperature.gpu --format=csv

4.2 Recommended Service Startup Order

Start vLLM first (language models typically have fixed memory requirements)
Start ComfyUI second (image generation can adjust memory usage dynamically)
Close unnecessary GUI apps (browsers, Discord, etc.)

4.3 Memory Monitoring Script

Create a simple monitoring script:

#!/bin/bash
# monitor-gpu.sh
while true; do
 echo "=== $(date) ==="
 nvidia-smi --query-gpu=memory.used,memory.free,temperature.gpu --format=csv
 docker ps --filter "name=vllm\|name=comfyui" --format "table {{.Names}}\t{{.Status}}"
 sleep 30
done

Step 5: Integrate Cowork MCP Multi-Agent Framework

Once your local AI infrastructure is ready, connect it to Cowork MCP to build multi-agent systems.

5.1 Register vLLM as a Brain

# If using Hermes Agent
hermes register-cowork

# Or manually set up MCP endpoints
# In your Agent config file:
{
 "mcpServers": {
 "cowork": {
 "url": "http://localhost:6868/mcp",
 "transport": "streamable-http"
 },
 "vllm": {
 "url": "http://localhost:8000/v1",
 "transport": "streamable-http"
 }
 }
}

5.2 Real Application: Local AI Content Pipeline

Local vLLM (Content Generation) → ComfyUI (Image Generation) → ffmpeg (Post-processing)

Every step of this pipeline can be coordinated by different AI agents:

Research Agent: Use vLLM to generate content outlines
Writing Agent: Use vLLM to expand into full articles
Image Agent: Use ComfyUI to generate corresponding illustrations
Integration Agent: Combine content and images into final output

Step 6: Automation and Backup

6.1 Container Backup

# Backup ComfyUI container state
docker commit comfyui comfyui-backup:$(date +%Y%m%d)

# Export image
docker save comfyui:latest | gzip > ~/backups/comfyui-$(date +%Y%m%d).tar.gz

6.2 Model Backup

# Regularly backup HuggingFace cache
rsync -avh ~/.cache/huggingface ~/backups/huggingface-cache/

# Or use rclone to backup to cloud
rclone sync ~/.cache/huggingface remote:backup-huggingface --progress

6.3 Health Check Script

Create health check cron jobs:

# Check service status every 5 minutes
*/5 * * * * curl -f http://localhost:8000/v1/models || echo "vLLM is down at $(date)" | mail -s "vLLM Alert" admin@example.com

Troubleshooting

Q1: Docker Cannot Use GPU

# Check if NVIDIA Container Toolkit is installed
nvidia-ctk runtime verify

# Restart Docker service
sudo systemctl restart docker

Q2: GPU Memory Insufficient

# Reduce vLLM memory utilization
# Add to docker run command:
--gpu-memory-utilization 0.7

# Or stop unnecessary services
sudo systemctl stop bluetooth

Q3: ComfyUI Fails to Start

# Check Docker logs
docker logs comfyui

# Try restarting
docker restart comfyui

Q4: Slow Model Downloads

# Use HuggingFace mirror (for users in mainland China)
export HF_ENDPOINT=https://hf-mirror.com

What’s Next

You now have a complete local AI infrastructure:

Today:

Confirm vLLM and ComfyUI are running
Test API endpoints to verify content and image generation work

This Week:

Set up systemd services for auto-start on boot
Create backup and monitoring mechanisms
Connect at least one LLM backend to Cowork MCP

This Month:

Explore the Cowork MCP framework for more AI agent teams
Experiment with different models to find the best configuration for your needs
Build your own AI content production pipeline

FAQ

Q: What level of GPU do I need to run these services? A: At least 24GB memory recommended (RTX 3090/4090). If memory is smaller, you can run smaller models (7B-13B) or use Docker memory limits.

Q: Can I run this on a machine without GPU? A: Yes, but it will be slow. CPU inference is possible, but not practical for large models (70B+).

Q: How do I update models? A: Use HuggingFace CLI: huggingface-cli download <model-name> --local-dir <path>. Docker containers will automatically use updated local models.

Q: What’s the difference between vLLM and Ollama? A: vLLM is optimized for high-throughput inference and supports more model formats and advanced features. Ollama is more lightweight and easier to use but has fewer features. For local AI infrastructure, vLLM is recommended.

Q: How does ComfyUI differ from Automatic1111? A: ComfyUI uses a node-based workflow, making it more flexible and efficient. Automatic1111 has a GUI that’s more beginner-friendly. Both work well, but ComfyUI is more popular in production environments.

Read time: ~15 minutes | Published: 2026-07-27

Find this guide helpful? Explore more guides on Slashman Tools or check out our AI Toolchain Template Library for ready-to-use configurations.

[- Back to Home]