Troubleshooting guide

This guide covers common issues you might encounter when using SmolVM and how to resolve them.

Diagnostics

SmolVM includes a built-in diagnostic tool to check your system configuration:

# Auto-detect backend and check prerequisites
smolvm doctor

# Check specific backend
smolvm doctor --backend firecracker
smolvm doctor --backend qemu

# CI-friendly JSON output
smolvm doctor --json --strict

The smolvm doctor command validates:

KVM availability (Linux/Firecracker)
Firecracker binary installation
QEMU installation and HVF support (macOS)
Network configuration (nftables, iproute2)
System permissions

Common issues

Docker image build fails with daemon errors

Problem: Image builds fail because Docker is not installed, the daemon is not running, or your user lacks permission to access it.SmolVM automatically diagnoses the specific Docker issue and returns a targeted error message. The three most common scenarios are:Docker not installed:

Docker is required to build images. Install Docker Desktop (macOS) or docker.io (Linux).

Daemon not running:

Docker is installed, but SmolVM could not reach the Docker daemon.
Start Docker Desktop or the Docker service and try again.

Permission denied:

Docker is installed, but this user cannot access the Docker daemon socket.
Make sure Docker Desktop is running or grant access to /var/run/docker.sock.

Solution:

Install Docker if missing:

# macOS
brew install --cask docker

# Debian/Ubuntu
sudo apt-get install docker.io

Start the Docker daemon:

# macOS
open -a Docker

# Linux
sudo systemctl start docker

Fix socket permissions (Linux):

sudo usermod -aG docker $USER
newgrp docker  # Or log out and back in

You can also check Docker status programmatically before building:

from smolvm.build import ImageBuilder

builder = ImageBuilder()
if not builder.check_docker():
    error = builder.docker_requirement_error()
    print(error)  # Specific diagnosis and fix suggestion

KVM not available: /dev/kvm not found

Problem: The Firecracker backend requires KVM hardware virtualization.Solution:

Verify KVM is available:

ls -l /dev/kvm

If missing, enable virtualization in your BIOS/UEFI settings
Add your user to the kvm group:

sudo usermod -aG kvm $USER
newgrp kvm  # Or log out and back in

Verify permissions:

# Should show rw-rw---- with kvm group
ls -l /dev/kvm

Alternative: Use the QEMU backend if KVM is unavailable:

from smolvm import SmolVM
vm = SmolVM(backend="qemu")

Firecracker binary not found

Problem: The firecracker executable is not in PATH.Solution:

Run the setup command:

smolvm setup

Or install manually using the HostManager:

from smolvm.host import HostManager

host = HostManager()
host.install_firecracker()

Verify installation:

which firecracker
# Should show /usr/local/bin/firecracker or ~/.smolvm/bin/firecracker

Permission denied: Cannot create TAP device

Problem: User lacks permissions to create TAP networking devices.Solution:

Run the setup command:

smolvm setup

Or configure manually:

# Allow user to create TAP devices
sudo setcap cap_net_admin+ep $(which ip)

# Enable IP forwarding
sudo sysctl -w net.ipv4.ip_forward=1
echo 'net.ipv4.ip_forward=1' | sudo tee -a /etc/sysctl.conf

Verify nftables is installed:

sudo nft list ruleset

VM boots but SSH connection times out

Problem: VM starts successfully but SSH connection fails.Solution:

Check VM status:

from smolvm import SmolVM

vm = SmolVM.from_id("vm-xxxxx")
info = vm.info
print(f"Status: {info.status}")
if info.network:
    print(f"IP: {info.network.guest_ip}")
    print(f"SSH Port: {info.network.ssh_host_port}")
vm.close()

Test network connectivity:

# Ping guest IP (Firecracker backend)
ping 172.16.0.2

# Test SSH port forwarding
nc -zv localhost 2200

Check firewall rules:

sudo nft list ruleset | grep 2200

Examine VM logs:

cat ~/.local/state/smolvm/vm-xxxxx.log

Increase the startup wait time:

from smolvm import SmolVM

vm = SmolVM()
vm.start(boot_timeout=120)
print(vm.run("echo ready").stdout)
vm.close()

Guest agent or vsock command times out

Problem: A command, file transfer, shell, or snapshot preflight waits for the guest agent and then times out.Solution:

Check that the sandbox uses a recent SmolVM image:

smolvm sandbox info my-vm

On Linux QEMU hosts, verify the vsock device exists:

sudo modprobe vhost_vsock
test -e /dev/vhost-vsock

Try the SSH compatibility path for file or environment operations:

smolvm sandbox env list my-vm --comm-channel ssh
smolvm sandbox file upload my-vm ./report.txt /tmp/report.txt --comm-channel ssh

Inspect the sandbox log and guest-agent log output:

cat ~/.local/state/smolvm/my-vm.log

Recreate the sandbox with a current published image if the log says the guest agent is missing:

smolvm sandbox delete my-vm
smolvm sandbox create --name my-vm

Snapshot creation times out before any snapshot files appear

Problem: Snapshot creation fails before SmolVM writes a disk, memory, or state artifact.This usually means SmolVM could not finish the guest sync step. SmolVM asks the guest to save pending file changes before it pauses the sandbox.Solution:

Confirm the sandbox can run a simple command:

smolvm sandbox shell my-vm

If the fast shell fails, test SSH directly:

smolvm sandbox ssh my-vm

Retry with a current image or recreate the sandbox. Recent images include the guest-agent /sync endpoint used before snapshot capture.
Include the VM log when reporting the issue:

cat ~/.local/state/smolvm/my-vm.log

DatabaseError: database is locked

Problem: Multiple SmolVM processes trying to access the state database simultaneously.Solution:

The state database uses exclusive locks for writes. Ensure only one process modifies VMs at a time.
Check for stale processes:

ps aux | grep firecracker
ps aux | grep qemu-system

Clean up stale resources using the CLI:

smolvm sandbox delete --all --force

If persistent, manually remove the lock:

# WARNING: Only do this if no SmolVM processes are running
rm ~/.local/state/smolvm/smolvm.db-wal
rm ~/.local/state/smolvm/smolvm.db-shm

No IP addresses available in pool

Problem: All IPs in the 172.16.0.2-254 range are allocated.Solution:

List all VMs:

smolvm sandbox list

Clean up stopped or stale VMs:

smolvm sandbox delete --all --force

The IP pool supports 253 concurrent VMs (172.16.0.2 through 172.16.0.254). If you need more, consider implementing a custom IP allocator.

QEMU exited early while booting (macOS)

Problem: QEMU process terminates immediately after start.Solution:

Verify QEMU installation:

qemu-system-aarch64 --version
qemu-system-x86_64 --version

Check HVF acceleration support:

qemu-system-aarch64 -accel help
# Should list 'hvf' for Hypervisor.framework

Reinstall QEMU via Homebrew:

brew uninstall qemu
brew install qemu

Check VM logs for kernel panic:

cat ~/.local/state/smolvm/vm-xxxxx.log

Use auto-config mode to let SmolVM pick a compatible kernel and rootfs:

from smolvm import SmolVM

# Auto-config handles kernel/rootfs selection
with SmolVM() as vm:
    result = vm.run("echo works")
    print(result.stdout)

VM stuck in ERROR state

Problem: VM is marked as ERROR and cannot be restarted.Solution:

Check VM details:

from smolvm import SmolVM

vm = SmolVM.from_id("vm-xxxxx")
info = vm.info
print(f"Status: {info.status}")
print(f"PID: {info.pid}")
vm.close()

Examine logs:

cat ~/.local/state/smolvm/vm-xxxxx.log

Delete the failed VM and create a fresh one:

# Delete the failed VM
vm = SmolVM.from_id("vm-xxxxx")
vm.delete()
vm.close()

# Create a fresh VM
with SmolVM() as new_vm:
    result = new_vm.run("echo 'back in business'")
    print(result.stdout)

Run cleanup to remove all stale resources:

smolvm sandbox delete --all --force

RuntimeError: Unable to find writable data directory

Problem: SmolVM cannot create or write to any data directory.Solution:

Check directory permissions:

ls -ld ~/.local/state/smolvm
ls -ld /var/lib/smolvm

Create directory manually:

mkdir -p ~/.local/state/smolvm
chmod 755 ~/.local/state/smolvm

Set explicit data directory:

from pathlib import Path
from smolvm import SmolVM

data_dir = Path("/tmp/smolvm-data")
manager = SmolVM(data_dir=data_dir)

Use environment variable:

export SMOLVM_DATA_DIR=/tmp/smolvm-data

Debugging tips

Enable debug logging

import logging

logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

from smolvm import SmolVM

# Now you'll see detailed debug output
with SmolVM() as vm:
    vm.run("echo hello")

Inspect Firecracker socket

Manually query the Firecracker API:

from smolvm.api import FirecrackerClient
from pathlib import Path

socket_path = Path("/tmp/fc-vm-xxxxx.sock")
client = FirecrackerClient(socket_path)
info = client.get_instance_info()
print(info)
client.close()

Check network rules

# List all nftables rules
sudo nft list ruleset

# Check NAT rules for specific VM
sudo nft list table nat | grep smolvm

# List TAP devices
ip link show | grep tap

# Show routes to guest VMs
ip route show | grep 172.16.0

Monitor VM processes

# List all Firecracker processes
ps aux | grep firecracker

# List all QEMU processes  
ps aux | grep qemu-system

# Check resource usage
top -p $(pgrep -d',' firecracker)

Getting help

If you’re still experiencing issues:

Check the GitHub Issues for similar problems
Run smolvm doctor --json and include the output in your bug report
Include relevant logs from ~/.local/state/smolvm/*.log
Join the Celesto AI Discord community

Known limitations

IP Pool: Maximum 253 concurrent VMs (172.16.0.2-254)
SSH Port Pool: Maximum 800 concurrent VMs (ports 2200-2999)
Firecracker: Linux only and requires KVM
QEMU: Required for macOS and Windows guests; Linux direct-kernel guests can use the faster microvm machine model
libkrun: Experimental and does not support pause, resume, snapshots, or snapshot restore yet
Windows guests: Use SSH for the control channel

Get Started

Features

Guides

Architecture

Operations

Troubleshooting guide

Diagnostics

Common issues

Debugging tips

Enable debug logging

Inspect Firecracker socket

Check network rules

Monitor VM processes

Getting help

Known limitations

​Diagnostics

​Common issues

​Debugging tips

​Enable debug logging

​Inspect Firecracker socket

​Check network rules

​Monitor VM processes

​Getting help

​Known limitations

Diagnostics

Common issues

Debugging tips

Enable debug logging

Inspect Firecracker socket

Check network rules

Monitor VM processes

Getting help

Known limitations