Local AI Infrastructure Setup Guide: From Zero to vLLM + ComfyUI

Mon, 27 Jul 2026 00:00:00 +0000

Why Build Local AI Infrastructure?

In 2026, AI tools have become essential infrastructure for developers, creators, and enterprises. But most people still rely on cloud APIs — paying per call, risking data privacy, and hitting rate limits.

Local AI infrastructure gives you three things: full control, unlimited usage, and zero marginal cost.

Once set up, you can:

Use LLMs without limits, no API fees
Process sensitive data without leaving your machine
Run multiple services simultaneously, no rate limits
Integrate with frameworks like Cowork MCP for true multi-agent systems

This guide assumes you have a machine with a GPU (DGX GB10, RTX 4090, or better recommended). I’ll walk you through building a complete local AI environment from scratch.

What you’ll learn:

Install and configure local LLM serving (vLLM)
Deploy ComfyUI for image generation
Manage GPU memory and service startup order
Automate service management (systemd)
Integrate with Cowork MCP for multi-agent AI teams

Let’s get started.

Prerequisites

Before we begin, make sure you have:

Item	Version	Check Command
Linux	Ubuntu 22.04+ or Arch	`uname -a`
GPU	NVIDIA RTX 3090+/4090/GB10	`nvidia-smi`
CUDA	≥ 12.4	`nvcc --version`
Docker	≥ 24.0	`docker --version`
Node.js	≥ 20	`node --version`
Python	≥ 3.10	`python3 --version`

If your GPU has less than 24GB memory, consider closing unnecessary GUI apps or services before starting.

Step 1: Install Docker and NVIDIA Container Toolkit

Docker is the recommended way to deploy local AI services because it isolates dependencies, avoids version conflicts, and prevents system pollution.

# Install Docker
sudo apt update
sudo apt install -y docker.io docker-compose-v2

# Install NVIDIA Container Toolkit (enables GPU access in Docker)
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU availability:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If you see GPU info and memory usage, your environment is correctly configured.

Step 2: Deploy vLLM Local Language Model

vLLM is currently the most efficient local LLM inference framework, supporting multiple model formats (GGUF, GPTQ, AWQ).

2.1 Deploy with Docker

docker run -d \
 --name vllm-server \
 --gpus all \
 -p 8000:8000 \
 -v ~/.cache/huggingface:/root/.cache/huggingface \
 vllm/vllm-openai:latest \
 --model "nvidia/Llama-3.1-Nemotron-70B-Instruct" \
 --gpu-memory-utilization 0.9 \
 --max-model-len 8192

Parameter explanation:

--model: The model to deploy (replace with any HuggingFace model)
--gpu-memory-utilization: Ratio of GPU memory to use (0.9 means 90%)
--max-model-len: Maximum context length

Note: If your GPU has less than 80GB memory, start with smaller models (7B or 13B) before trying larger ones.

2.2 Test the API Endpoint

curl -X POST http://localhost:8000/v1/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "nvidia/Llama-3.1-Nemotron-70B-Instruct",
 "prompt": "Explain what a multi-agent AI system is",
 "max_tokens": 200
 }'

If you receive a response, vLLM is successfully deployed.

2.3 Setup as System Service (Auto-start)

Create a systemd service file:

sudo tee /etc/systemd/system/vllm.service << 'EOF'
[Unit]
Description=vLLM Local LLM Server
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/docker start vllm-server
ExecStop=/usr/bin/docker stop vllm-server

[Install]
WantedBy=multi-user.target
EOF

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable vllm.service
sudo systemctl start vllm.service

Step 3: Deploy ComfyUI Image Generation

ComfyUI is the most powerful local image generation tool, supporting Flux, Stable Diffusion, LTX, and various other models.

3.1 Deploy with Docker

docker run -d \
 --name comfyui \
 --gpus all \
 -p 8188:8188 \
 -v ~/.cache/huggingface:/root/.cache/huggingface \
 -v ~/comfyui-output:/output \
 ghcr.io/ai-forest/comfyui:latest

This starts ComfyUI at http://localhost:8188.

3.2 Manual Installation (Advanced Option)

If you need more control, you can install manually:

# Clone ComfyUI repository
cd ~/comfyui
git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI

# Install dependencies
pip install -r requirements.txt

# Install custom nodes (optional)
git clone https://github.com/Fannovel16/ComfyUI-Frame-Interpolation.git custom_nodes/
git clone https://github.com/laksjdjoy/deforum-comfy.git custom_nodes/

# Start the service
python main.py --listen 0.0.0.0 --port 8188

3.3 Download Common Models

# Use HuggingFace CLI to download models
pip install huggingface_hub
huggingface-cli download stabilityai/stable-diffusion-xl-base-1.0 --local-dir ~/models/sdxl
huggingface-cli download stabilityai/stable-diffusion-3.5-large --local-dir ~/models/sd3.5

Step 4: GPU Memory Management Strategies

When running AI services locally, GPU memory is the biggest bottleneck. Here are some battle-tested practices:

4.1 Check Available Memory

free -h
nvidia-smi --query-gpu=memory.used,memory.free,temperature.gpu --format=csv

4.2 Recommended Service Startup Order

Start vLLM first (language models typically have fixed memory requirements)
Start ComfyUI second (image generation can adjust memory usage dynamically)
Close unnecessary GUI apps (browsers, Discord, etc.)

4.3 Memory Monitoring Script

Create a simple monitoring script:

#!/bin/bash
# monitor-gpu.sh
while true; do
 echo "=== $(date) ==="
 nvidia-smi --query-gpu=memory.used,memory.free,temperature.gpu --format=csv
 docker ps --filter "name=vllm\|name=comfyui" --format "table {{.Names}}\t{{.Status}}"
 sleep 30
done

Step 5: Integrate Cowork MCP Multi-Agent Framework

Once your local AI infrastructure is ready, connect it to Cowork MCP to build multi-agent systems.

5.1 Register vLLM as a Brain

# If using Hermes Agent
hermes register-cowork

# Or manually set up MCP endpoints
# In your Agent config file:
{
 "mcpServers": {
 "cowork": {
 "url": "http://localhost:6868/mcp",
 "transport": "streamable-http"
 },
 "vllm": {
 "url": "http://localhost:8000/v1",
 "transport": "streamable-http"
 }
 }
}

5.2 Real Application: Local AI Content Pipeline

Local vLLM (Content Generation) → ComfyUI (Image Generation) → ffmpeg (Post-processing)

Every step of this pipeline can be coordinated by different AI agents:

Research Agent: Use vLLM to generate content outlines
Writing Agent: Use vLLM to expand into full articles
Image Agent: Use ComfyUI to generate corresponding illustrations
Integration Agent: Combine content and images into final output

Step 6: Automation and Backup

6.1 Container Backup

# Backup ComfyUI container state
docker commit comfyui comfyui-backup:$(date +%Y%m%d)

# Export image
docker save comfyui:latest | gzip > ~/backups/comfyui-$(date +%Y%m%d).tar.gz

6.2 Model Backup

# Regularly backup HuggingFace cache
rsync -avh ~/.cache/huggingface ~/backups/huggingface-cache/

# Or use rclone to backup to cloud
rclone sync ~/.cache/huggingface remote:backup-huggingface --progress

6.3 Health Check Script

Create health check cron jobs:

# Check service status every 5 minutes
*/5 * * * * curl -f http://localhost:8000/v1/models || echo "vLLM is down at $(date)" | mail -s "vLLM Alert" admin@example.com

Troubleshooting

Q1: Docker Cannot Use GPU

# Check if NVIDIA Container Toolkit is installed
nvidia-ctk runtime verify

# Restart Docker service
sudo systemctl restart docker

Q2: GPU Memory Insufficient

# Reduce vLLM memory utilization
# Add to docker run command:
--gpu-memory-utilization 0.7

# Or stop unnecessary services
sudo systemctl stop bluetooth

Q3: ComfyUI Fails to Start

# Check Docker logs
docker logs comfyui

# Try restarting
docker restart comfyui

Q4: Slow Model Downloads

# Use HuggingFace mirror (for users in mainland China)
export HF_ENDPOINT=https://hf-mirror.com

What’s Next

You now have a complete local AI infrastructure:

Today:

Confirm vLLM and ComfyUI are running
Test API endpoints to verify content and image generation work

This Week:

Set up systemd services for auto-start on boot
Create backup and monitoring mechanisms
Connect at least one LLM backend to Cowork MCP

This Month:

Explore the Cowork MCP framework for more AI agent teams
Experiment with different models to find the best configuration for your needs
Build your own AI content production pipeline

FAQ

Q: What level of GPU do I need to run these services? A: At least 24GB memory recommended (RTX 3090/4090). If memory is smaller, you can run smaller models (7B-13B) or use Docker memory limits.

Q: Can I run this on a machine without GPU? A: Yes, but it will be slow. CPU inference is possible, but not practical for large models (70B+).

Q: How do I update models? A: Use HuggingFace CLI: huggingface-cli download <model-name> --local-dir <path>. Docker containers will automatically use updated local models.

Q: What’s the difference between vLLM and Ollama? A: vLLM is optimized for high-throughput inference and supports more model formats and advanced features. Ollama is more lightweight and easier to use but has fewer features. For local AI infrastructure, vLLM is recommended.

Q: How does ComfyUI differ from Automatic1111? A: ComfyUI uses a node-based workflow, making it more flexible and efficient. Automatic1111 has a GUI that’s more beginner-friendly. Both work well, but ComfyUI is more popular in production environments.

Read time: ~15 minutes | Published: 2026-07-27

Find this guide helpful? Explore more guides on Slashman Tools or check out our AI Toolchain Template Library for ready-to-use configurations.

[- Back to Home]

Local AI on Intellectual Guides