Pharyvac Computer Vision

cvat-self-host-runbook

Properties1
tagscomputer-vision, validation, runbook, cvat, infrastructure

4 min read

Runbook — Self-Hosting CVAT for the Inter-Rater Study (P1)

Goal: stand up a private, local CVAT instance so ≥2 annotators can label the validation subset and we can compute inter-rater κ — without sending intraoperative video to any third party.

Decision context: CVAT chosen over Encord (see validation-plan P1). This runbook covers the self-hosted Community Edition (free, best privacy story). If annotators are remote and can’t share a local network, fall back to CVAT Online Team (~$23/user/mo) and skip this.

Skill level: comfortable with a terminal. No coding. ~1–2 hours first time.

1. Hardware

The CVAT server is light; disk is the only spec to size deliberately (surgical video is large). The annotators’ laptops matter more than the server because the UI is browser-side.

RoleSpecNotes
Server box4 CPU cores, 8–16 GB RAMMin is 2 cores / 4 GB; any modern desktop/Mac Mini/spare laptop clears this
Server disk256 GB–1 TB freeCVAT stores uploaded video + an extracted frame cache. Size to the video you host
GPUnoneOnly needed for AI auto-annotation, which we don’t use for manual κ work
Annotator laptops16 GB RAM, recent ChromeBounding-box-on-video is memory-hungry client-side

Disk-saving move: for P1, upload only the 3–5 cases / ~60 min being double-annotated, not all 16 cases. That drops the footprint to tens of GB — almost anything works as the server.

Where to run it: a machine on the lab/OR network that annotators can reach. Keep it local-only (not exposed to the public internet) — that’s the easy and IRB-clean configuration.

2. Prerequisites (install once)

  • Docker Engine 20.10+ and the Docker Compose v2 plugin.
    • Linux (Ubuntu): follow Docker’s official apt install (adds Docker’s GPG key + repo, installs docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin).
    • macOS / Windows: install Docker Desktop (bundles Compose).
  • Git (to clone the CVAT source).
  • Confirm both work: 
    docker --version
docker compose version
git --version

3. Install CVAT

# 1. Get the full source (not just the compose file)
git clone https://github.com/cvat-ai/cvat
cd cvat
 
# 2. (local-only default) CVAT_HOST stays localhost — fine if annotators
#    use this same machine. For other machines on the LAN, set the server's
#    LAN IP instead:
export CVAT_HOST=<server-LAN-IP>   # e.g. 192.168.1.50 ; skip for localhost-only
 
# 3. Start it
docker compose up -d
 
# 4. Create the admin account
docker exec -it cvat_server bash -ic 'python3 ~/manage.py createsuperuser'

Then open http://localhost:8080 (or http://<server-LAN-IP>:8080 from another machine on the network) in Chrome and log in.

Do not open port 8080 to the internet. Keep access on the local network only. If remote access is genuinely required, that’s the harder HTTPS/TLS/firewall path — get technical help or use CVAT Online instead.

4. Set up the annotation work

  1. Create a user account for each annotator (Admin panel → users), so contributions are attributable for the audit trail.
  2. Create a Project = “FESS Inter-Rater Validation”. Load the label set by importing specs/cvat-labels.json into the project’s Raw labels editor (one paste — no hand-typing). It encodes the annotation-protocol exactly: 9 instrument rectangle labels with an active checkbox + 4 event tags. The annotation-protocol is the standard κ is measured against.
  3. Create a Task per case/clip, upload the video. For an honest inter-rater study, have each annotator label independently — either duplicate tasks per annotator, or assign separate Jobs, so they don’t see each other’s boxes.
  4. Export annotations (COCO / CVAT XML) → feed into the κ / IoU / ICC computation (the data:statistical-analysis skill can do this).

5. Ongoing upkeep (light)

  • Backups — CVAT keeps everything in Docker volumes (cvat_db, cvat_data, cvat_keys, cvat_logs, cvat_inmem_db, cvat_cache_db). Back up cvat_db (annotations) + cvat_data (media) regularly. This is your ground truth — don’t lose it.
  • Upgrades — docker compose pull && docker compose up -d periodically; check the CVAT upgrade guide before major version jumps.
  • Stop / start — docker compose down (keeps data) / docker compose up -d.

6. Privacy / IRB notes

  • Local-only deployment means OR video never leaves infrastructure you control — aligns with the grant’s “privacy by design” framing.
  • Confirm the server box’s disk encryption + access controls meet your IRB / data-use terms before uploading patient video.

Decision: self-hosted vs CVAT Online

Self-host (this runbook)CVAT Online Team
Cost$0 + a machine~$23/user/mo annual
Setup1–2 hrs, terminalinstant
Privacybest (you control it)data on CVAT’s cloud
Best whenannotators share a LANannotators are remote

Rule of thumb: can all annotators work on one machine / one local network? → self-host. Scattered/remote? → CVAT Online.

Graph View

Backlinks