best-effort extent-same

bees

A fixed-size fingerprint book. When the book is full, bees still runs — it just forgets older chunks and dedupes less well.

bees walks the Btrfs pool in the background whenever ENABLE_BEES=1. It is not a nightly cron. Identical extents (near-duplicate GGUFs, HF blobs, Pinokio trees) start sharing disk.

Hash tableWhen
1GSmall experiments only
2GDefault for multi-terabyte AI pools
4GOnly if 2G stays nearly full after a full re-crawl; ~4 GiB sticky RAM, no swap

A full table is not a full disk

Unraid parity checks do not fill beeshash.dat. Occupancy near 100% means bees is forgetting old fingerprints, not that the NVMe pool is out of space.

Grow the table without drama

  1. Keep ALLOW_FORMAT=0 and FORCE_FORMAT=0.
  2. Stop bees only if you can; leave the container and NFS share up.
  3. Rename old beeshash.dat and beescrawl.dat.
  4. truncate -s 2G beeshash.dat — a fresh empty file. Do not enlarge a live 1G table in place.
  5. Set BEES_HASH_SIZE=2G in .env.
  6. Start bees again. Occupancy near zero, then climbing, is expected.

Look

docker exec fast-models pgrep -a bees
ls -lh /mnt/user/appdata/fast-models/bees/beeshash.dat
docker exec fast-models sh -c 'grep -E "cells occupied|Uptime" /var/lib/bees/beesstats.txt'

Optional deeper pass (IO-heavy; bees remains the everyday tool):

docker exec fast-models duperemove -r -d -h /ai-data/models
docker exec fast-models compsize /ai-data/models 2>/dev/null || true

Keep BEES_THREADS=1 on ~32 GiB hosts unless you clearly have spare capacity. Upstream: Zygo/bees. Markdown twin: docs/BEES_HASH_SIZING.md. Grafana import + sample screenshots: observability.html.