> ## Documentation Index
> Fetch the complete documentation index at: https://docs.celesto.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting guide

> Troubleshoot common SmolVM issues — KVM permissions, networking failures, image download errors, SSH timeouts — using smolvm doctor and verified fixes.

This guide covers common issues you might encounter when using SmolVM and how to resolve them.

## Diagnostics

SmolVM includes a built-in diagnostic tool to check your system configuration:

```bash theme={null} theme={null}
# Auto-detect backend and check prerequisites
smolvm doctor

# Check specific backend
smolvm doctor --backend firecracker
smolvm doctor --backend qemu

# CI-friendly JSON output
smolvm doctor --json --strict
```

The `smolvm doctor` command validates:

* KVM availability (Linux/Firecracker)
* Firecracker binary installation
* QEMU installation and HVF support (macOS)
* Network configuration (nftables, iproute2)
* System permissions

## Common issues

<AccordionGroup>
  <Accordion title="Docker image build fails with daemon errors">
    **Problem**: Image builds fail because Docker is not installed, the daemon is not running, or your user lacks permission to access it.

    SmolVM automatically diagnoses the specific Docker issue and returns a targeted error message. The three most common scenarios are:

    **Docker not installed:**

    ```
    Docker is required to build images. Install Docker Desktop (macOS) or docker.io (Linux).
    ```

    **Daemon not running:**

    ```
    Docker is installed, but SmolVM could not reach the Docker daemon.
    Start Docker Desktop or the Docker service and try again.
    ```

    **Permission denied:**

    ```
    Docker is installed, but this user cannot access the Docker daemon socket.
    Make sure Docker Desktop is running or grant access to /var/run/docker.sock.
    ```

    **Solution**:

    1. Install Docker if missing:

    ```bash theme={null} theme={null}
    # macOS
    brew install --cask docker

    # Debian/Ubuntu
    sudo apt-get install docker.io
    ```

    2. Start the Docker daemon:

    ```bash theme={null} theme={null}
    # macOS
    open -a Docker

    # Linux
    sudo systemctl start docker
    ```

    3. Fix socket permissions (Linux):

    ```bash theme={null} theme={null}
    sudo usermod -aG docker $USER
    newgrp docker  # Or log out and back in
    ```

    You can also check Docker status programmatically before building:

    ```python theme={null} theme={null}
    from smolvm.build import ImageBuilder

    builder = ImageBuilder()
    if not builder.check_docker():
        error = builder.docker_requirement_error()
        print(error)  # Specific diagnosis and fix suggestion
    ```
  </Accordion>

  <Accordion title="KVM not available: /dev/kvm not found">
    **Problem**: The Firecracker backend requires KVM hardware virtualization.

    **Solution**:

    1. Verify KVM is available:

    ```bash theme={null} theme={null}
    ls -l /dev/kvm
    ```

    2. If missing, enable virtualization in your BIOS/UEFI settings

    3. Add your user to the `kvm` group:

    ```bash theme={null} theme={null}
    sudo usermod -aG kvm $USER
    newgrp kvm  # Or log out and back in
    ```

    4. Verify permissions:

    ```bash theme={null} theme={null}
    # Should show rw-rw---- with kvm group
    ls -l /dev/kvm
    ```

    **Alternative**: Use the QEMU backend if KVM is unavailable:

    ```python theme={null} theme={null}
    from smolvm import SmolVM
    vm = SmolVM(backend="qemu")
    ```
  </Accordion>

  <Accordion title="Firecracker binary not found">
    **Problem**: The `firecracker` executable is not in PATH.

    **Solution**:

    1. Run the setup command:

    ```bash theme={null} theme={null}
    smolvm setup
    ```

    2. Or install manually using the HostManager:

    ```python theme={null} theme={null}
    from smolvm.host import HostManager

    host = HostManager()
    host.install_firecracker()
    ```

    3. Verify installation:

    ```bash theme={null} theme={null}
    which firecracker
    # Should show /usr/local/bin/firecracker or ~/.smolvm/bin/firecracker
    ```
  </Accordion>

  <Accordion title="Permission denied: Cannot create TAP device">
    **Problem**: User lacks permissions to create TAP networking devices.

    **Solution**:

    1. Run the setup command:

    ```bash theme={null} theme={null}
    smolvm setup
    ```

    2. Or configure manually:

    ```bash theme={null} theme={null}
    # Allow user to create TAP devices
    sudo setcap cap_net_admin+ep $(which ip)

    # Enable IP forwarding
    sudo sysctl -w net.ipv4.ip_forward=1
    echo 'net.ipv4.ip_forward=1' | sudo tee -a /etc/sysctl.conf
    ```

    3. Verify nftables is installed:

    ```bash theme={null} theme={null}
    sudo nft list ruleset
    ```
  </Accordion>

  <Accordion title="VM boots but SSH connection times out">
    **Problem**: VM starts successfully but SSH connection fails.

    **Solution**:

    1. Check VM status:

    ```python theme={null} theme={null}
    from smolvm import SmolVM

    vm = SmolVM.from_id("vm-xxxxx")
    info = vm.info
    print(f"Status: {info.status}")
    if info.network:
        print(f"IP: {info.network.guest_ip}")
        print(f"SSH Port: {info.network.ssh_host_port}")
    vm.close()
    ```

    2. Test network connectivity:

    ```bash theme={null} theme={null}
    # Ping guest IP (Firecracker backend)
    ping 172.16.0.2

    # Test SSH port forwarding
    nc -zv localhost 2200
    ```

    3. Check firewall rules:

    ```bash theme={null} theme={null}
    sudo nft list ruleset | grep 2200
    ```

    4. Examine VM logs:

    ```bash theme={null} theme={null}
    cat ~/.local/state/smolvm/vm-xxxxx.log
    ```

    5. Increase the startup wait time:

    ```python theme={null} theme={null}
    from smolvm import SmolVM

    vm = SmolVM()
    vm.start(boot_timeout=120)
    print(vm.run("echo ready").stdout)
    vm.close()
    ```
  </Accordion>

  <Accordion title="Guest agent or vsock command times out">
    **Problem**: A command, file transfer, shell, or snapshot preflight waits for the guest agent and then times out.

    **Solution**:

    1. Check that the sandbox uses a recent SmolVM image:

    ```bash theme={null}
    smolvm sandbox info my-vm
    ```

    2. On Linux QEMU hosts, verify the vsock device exists:

    ```bash theme={null}
    sudo modprobe vhost_vsock
    test -e /dev/vhost-vsock
    ```

    3. Try the SSH compatibility path for file or environment operations:

    ```bash theme={null}
    smolvm sandbox env list my-vm --comm-channel ssh
    smolvm sandbox file upload my-vm ./report.txt /tmp/report.txt --comm-channel ssh
    ```

    4. Inspect the sandbox log and guest-agent log output:

    ```bash theme={null}
    cat ~/.local/state/smolvm/my-vm.log
    ```

    5. Recreate the sandbox with a current published image if the log says the guest agent is missing:

    ```bash theme={null}
    smolvm sandbox delete my-vm
    smolvm sandbox create --name my-vm
    ```
  </Accordion>

  <Accordion title="Snapshot creation times out before any snapshot files appear">
    **Problem**: Snapshot creation fails before SmolVM writes a disk, memory, or state artifact.

    This usually means SmolVM could not finish the guest sync step. SmolVM asks the guest to save pending file changes before it pauses the sandbox.

    **Solution**:

    1. Confirm the sandbox can run a simple command:

    ```bash theme={null}
    smolvm sandbox shell my-vm
    ```

    2. If the fast shell fails, test SSH directly:

    ```bash theme={null}
    smolvm sandbox ssh my-vm
    ```

    3. Retry with a current image or recreate the sandbox. Recent images include the guest-agent `/sync` endpoint used before snapshot capture.

    4. Include the VM log when reporting the issue:

    ```bash theme={null}
    cat ~/.local/state/smolvm/my-vm.log
    ```
  </Accordion>

  <Accordion title="DatabaseError: database is locked">
    **Problem**: Multiple SmolVM processes trying to access the state database simultaneously.

    **Solution**:

    1. The state database uses exclusive locks for writes. Ensure only one process modifies VMs at a time.

    2. Check for stale processes:

    ```bash theme={null} theme={null}
    ps aux | grep firecracker
    ps aux | grep qemu-system
    ```

    3. Clean up stale resources using the CLI:

    ```bash theme={null} theme={null}
    smolvm sandbox delete --all --force
    ```

    4. If persistent, manually remove the lock:

    ```bash theme={null} theme={null}
    # WARNING: Only do this if no SmolVM processes are running
    rm ~/.local/state/smolvm/smolvm.db-wal
    rm ~/.local/state/smolvm/smolvm.db-shm
    ```
  </Accordion>

  <Accordion title="No IP addresses available in pool">
    **Problem**: All IPs in the `172.16.0.2-254` range are allocated.

    **Solution**:

    1. List all VMs:

    ```bash theme={null} theme={null}
    smolvm sandbox list
    ```

    2. Clean up stopped or stale VMs:

    ```bash theme={null} theme={null}
    smolvm sandbox delete --all --force
    ```

    3. The IP pool supports 253 concurrent VMs (`172.16.0.2` through `172.16.0.254`). If you need more, consider implementing a custom IP allocator.
  </Accordion>

  <Accordion title="QEMU exited early while booting (macOS)">
    **Problem**: QEMU process terminates immediately after start.

    **Solution**:

    1. Verify QEMU installation:

    ```bash theme={null} theme={null}
    qemu-system-aarch64 --version
    qemu-system-x86_64 --version
    ```

    2. Check HVF acceleration support:

    ```bash theme={null} theme={null}
    qemu-system-aarch64 -accel help
    # Should list 'hvf' for Hypervisor.framework
    ```

    3. Reinstall QEMU via Homebrew:

    ```bash theme={null} theme={null}
    brew uninstall qemu
    brew install qemu
    ```

    4. Check VM logs for kernel panic:

    ```bash theme={null} theme={null}
    cat ~/.local/state/smolvm/vm-xxxxx.log
    ```

    5. Use auto-config mode to let SmolVM pick a compatible kernel and rootfs:

    ```python theme={null} theme={null}
    from smolvm import SmolVM

    # Auto-config handles kernel/rootfs selection
    with SmolVM() as vm:
        result = vm.run("echo works")
        print(result.stdout)
    ```
  </Accordion>

  <Accordion title="VM stuck in ERROR state">
    **Problem**: VM is marked as ERROR and cannot be restarted.

    **Solution**:

    1. Check VM details:

    ```python theme={null} theme={null}
    from smolvm import SmolVM

    vm = SmolVM.from_id("vm-xxxxx")
    info = vm.info
    print(f"Status: {info.status}")
    print(f"PID: {info.pid}")
    vm.close()
    ```

    2. Examine logs:

    ```bash theme={null} theme={null}
    cat ~/.local/state/smolvm/vm-xxxxx.log
    ```

    3. Delete the failed VM and create a fresh one:

    ```python theme={null} theme={null}
    # Delete the failed VM
    vm = SmolVM.from_id("vm-xxxxx")
    vm.delete()
    vm.close()

    # Create a fresh VM
    with SmolVM() as new_vm:
        result = new_vm.run("echo 'back in business'")
        print(result.stdout)
    ```

    4. Run cleanup to remove all stale resources:

    ```bash theme={null} theme={null}
    smolvm sandbox delete --all --force
    ```
  </Accordion>

  <Accordion title="RuntimeError: Unable to find writable data directory">
    **Problem**: SmolVM cannot create or write to any data directory.

    **Solution**:

    1. Check directory permissions:

    ```bash theme={null} theme={null}
    ls -ld ~/.local/state/smolvm
    ls -ld /var/lib/smolvm
    ```

    2. Create directory manually:

    ```bash theme={null} theme={null}
    mkdir -p ~/.local/state/smolvm
    chmod 755 ~/.local/state/smolvm
    ```

    3. Set explicit data directory:

    ```python theme={null} theme={null}
    from pathlib import Path
    from smolvm import SmolVM

    data_dir = Path("/tmp/smolvm-data")
    manager = SmolVM(data_dir=data_dir)
    ```

    4. Use environment variable:

    ```bash theme={null} theme={null}
    export SMOLVM_DATA_DIR=/tmp/smolvm-data
    ```
  </Accordion>
</AccordionGroup>

## Debugging tips

### Enable debug logging

```python theme={null} theme={null}
import logging

logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

from smolvm import SmolVM

# Now you'll see detailed debug output
with SmolVM() as vm:
    vm.run("echo hello")
```

### Inspect Firecracker socket

Manually query the Firecracker API:

```python theme={null} theme={null}
from smolvm.api import FirecrackerClient
from pathlib import Path

socket_path = Path("/tmp/fc-vm-xxxxx.sock")
client = FirecrackerClient(socket_path)
info = client.get_instance_info()
print(info)
client.close()
```

### Check network rules

```bash theme={null} theme={null}
# List all nftables rules
sudo nft list ruleset

# Check NAT rules for specific VM
sudo nft list table nat | grep smolvm

# List TAP devices
ip link show | grep tap

# Show routes to guest VMs
ip route show | grep 172.16.0
```

### Monitor VM processes

```bash theme={null} theme={null}
# List all Firecracker processes
ps aux | grep firecracker

# List all QEMU processes  
ps aux | grep qemu-system

# Check resource usage
top -p $(pgrep -d',' firecracker)
```

## Getting help

If you're still experiencing issues:

1. Check the [GitHub Issues](https://github.com/celestoai/smolvm/issues) for similar problems
2. Run `smolvm doctor --json` and include the output in your bug report
3. Include relevant logs from `~/.local/state/smolvm/*.log`
4. Join the [Celesto AI Discord](https://discord.gg/KNb5UkrAmm) community

## Known limitations

* **IP Pool**: Maximum 253 concurrent VMs (172.16.0.2-254)
* **SSH Port Pool**: Maximum 800 concurrent VMs (ports 2200-2999)
* **Firecracker**: Linux only and requires KVM
* **QEMU**: Required for macOS and Windows guests; Linux direct-kernel guests can use the faster `microvm` machine model
* **libkrun**: Experimental and does not support pause, resume, snapshots, or snapshot restore yet
* **Windows guests**: Use SSH for the control channel