dustcouncil/bandsaunter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Package the speech recogniser and its model for apt

Browse source 
No speech recogniser is in Debian, so installing bandsaunter from a .deb
left transcription to a manual pip step on every machine. A repository of
one's own is not bound by archive policy, so build-repo.sh now packages
faster-whisper and the base.en model alongside the application:

  bandsaunter                the application (Architecture: all)
  bandsaunter-transcribe     faster-whisper, vendored (amd64)
  bandsaunter-model-base-en  the model, so nothing reaches the network

The wheels land in /usr/lib/bandsaunter/vendor rather than dist-packages,
and transcribe.py appends that directory to sys.path -- appends, so an
apt-managed numpy or PyYAML still wins and the vendor copy only fills the
gap. Duplicates of what Debian already ships are stripped from the tree.
resolve_model() turns a bare "base.en" into the packaged copy when one is
installed, and leaves it alone to be downloaded when none is.

The app package recommends the other two, so "apt install bandsaunter"
brings the lot and --no-install-recommends still gets just the scanner.
Its postinst explains how to add a recogniser only when there genuinely
is not one -- including the case where apt has already unpacked the
recogniser package but not yet configured it.

Verified with the source tree hidden and no home directory: the packaged
CLI runs, and a real recording transcribes offline from the vendored
engine and packaged model while numpy still resolves to the system one.
apt itself resolves the repository over HTTP and plans all three.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
The Dust Council 2026-08-21 22:12:24 -07:00
parent 16f3128690
commit 8d94a52942
5 changed files with 304 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

78
README.md
View file

@ -38,6 +38,52 @@ sudo apt install ./dist/bandsaunter_*.deb
apt pulls in every dependency itself, and the package blacklists the DVB-T
driver that would otherwise claim the receiver. Nothing else to do.
Speech transcription is the one part that cannot come from Debian, because no
speech recogniser is packaged there. Installed this way, everything else works
and the install prints a short note saying how to add one. To have that arrive
by apt as well, build the repository below instead.
### From your own apt repository
For installing on several machines, or on a fresh one, without hunting for the
recogniser afterwards. `build-repo.sh` builds three packages and an apt index:
| Package | Arch | Contents |
|---|---|---|
| `bandsaunter` | all | the application |
| `bandsaunter-transcribe` | amd64 | faster-whisper and its dependencies, in `/usr/lib/bandsaunter/vendor` |
| `bandsaunter-model-base-en` | all | the `base.en` model, in `/usr/share/bandsaunter/models` |
```bash
./packaging/build-repo.sh # writes dist/repo/
rsync -a dist/repo/ server:/var/www/html/bandsaunter/
```
Serve that directory over HTTP from anywhere on the LAN, then on each machine:
```bash
echo 'deb [trusted=yes] http://server/bandsaunter ./' \
| sudo tee /etc/apt/sources.list.d/bandsaunter.list
sudo apt update
sudo apt install bandsaunter
```
That single command brings the recogniser and its model too — they are
`Recommends`, which apt installs by default. `--no-install-recommends` gets
just the application. Nothing reaches the network afterwards: the model is on
disk, so the first transcription works offline.
`[trusted=yes]` skips signing, which is the sensible trade on a private LAN.
To sign it instead, run `gpg --clearsign` over `dist/repo/Release` to produce
`InRelease` and drop the `[trusted=yes]`.
The vendored packages are appended to `sys.path`, never prepended, so anything
apt provides — numpy, PyYAML — still wins; the vendor directory only fills the
gap Debian leaves. `BANDSAUNTER_VENDOR_DIR` and `BANDSAUNTER_MODEL_DIR`
override both locations.
Rebuilding for a new version is the same command; `apt upgrade` picks it up.
### From source
```bash
@ -79,13 +125,18 @@ pip install vosk # ~10 MB plus a 40 MB model, weaker on noise
bandsaunter transcribe --engines
```
That is why transcription is `Suggests:` rather than `Depends:` in the
package. Debian Policy forbids anything in the archive from requiring
software outside it, and a `postinst` that reaches out to PyPI would break
offline and reproducible installs — so a package simply cannot pull these in.
Transcription is therefore off by default and reports plainly when no
recogniser is present, rather than the install failing or the feature
appearing broken.
That is why the plain `.deb` cannot depend on one. Debian Policy forbids
anything in the archive from requiring software outside it, and a `postinst`
that reaches out to PyPI would break offline and reproducible installs — so a
package in the archive simply cannot pull these in. Transcription is therefore
off by default and reports plainly when no recogniser is present, rather than
the install failing or the feature appearing broken.
A repository of your own is not bound by that rule, which is what
[`build-repo.sh`](#from-your-own-apt-repository) exploits: it packages
faster-whisper and its model itself, into a private directory rather than into
`dist-packages`, and lets apt install them alongside. Nothing is downloaded at
install time, and nothing collides with an apt-managed module.
Mixing the two is nonetheless fine here. Modern Debian marks the system
Python as externally managed (PEP 668), so a pip install lands in your user
@ -100,11 +151,11 @@ from the `.deb` into `/usr/lib/python3/dist-packages` picks it up with no
further configuration — verified, not assumed. A virtual environment works
too, as long as bandsaunter runs inside it.
If you would rather keep everything under apt, the only route is packaging a
recogniser for Debian yourself. That is a real undertaking for whisper: it
would mean packaging ctranslate2, tokenizers, onnxruntime and their
dependencies, several of which are large C++ or Rust projects. It is why
none of them are there.
Getting these into Debian proper would be a different matter: it would mean
packaging ctranslate2, tokenizers, onnxruntime and their dependencies, several
of which are large C++ or Rust projects, each to archive standards. That is
why none of them are there, and why the local repository vendors the wheels
instead of trying to do it properly.
### Other distributions
@ -682,7 +733,8 @@ depends on a trained model, so there is no built-in fallback.
```bash
bandsaunter transcribe --engines # what is installed
pip install faster-whisper # the recommended one
sudo apt install bandsaunter-transcribe # from your own repository
pip install faster-whisper # or straight from PyPI
```
| Engine | Notes |

47
bandsaunter/transcribe.py
View file

@ -11,6 +11,7 @@ from __future__ import annotations
import os
import queue
import shutil
import sys
import subprocess
import tempfile
import threading
@ -24,12 +25,50 @@ import numpy as np
from ._quiet import suppress_stderr
__all__ = ["Transcript", "transcribe", "available_engine", "ENGINES",
"TranscriptionWorker", "describe_engines"]
"TranscriptionWorker", "describe_engines", "resolve_model",
"VENDOR_DIR", "MODEL_DIR"]
# Best first. Whisper handles the noise and clipping of radio audio far
# better than the smaller recognisers, which were trained on clean speech.
ENGINES = ("faster-whisper", "whisper", "whisper-cli", "vosk", "pocketsphinx")
# No recogniser is packaged for Debian, so a package that wants to supply one
# ships it here instead of in dist-packages. Appended to the path rather than
# prepended, so anything the system package manager provides still wins and a
# vendored copy only ever fills a gap.
VENDOR_DIR = Path(os.environ.get("BANDSAUNTER_VENDOR_DIR",
"/usr/lib/bandsaunter/vendor"))
# Models shipped alongside, so a machine never has to reach the network.
MODEL_DIR = Path(os.environ.get("BANDSAUNTER_MODEL_DIR",
"/usr/share/bandsaunter/models"))
def _add_vendor_path() -> None:
if VENDOR_DIR.is_dir():
path = str(VENDOR_DIR)
if path not in sys.path:
sys.path.append(path)
_add_vendor_path()
def resolve_model(name: str) -> str:
"""A packaged model directory if there is one, otherwise the name itself.
Lets ``base.en`` mean the copy installed on this machine when one exists,
and fall back to the recogniser downloading it when one does not.
"""
if not name:
return name
given = Path(name).expanduser()
if given.is_dir():
return str(given)
packaged = MODEL_DIR / name
if packaged.is_dir():
return str(packaged)
return name
# Vosk's bundled models are named by region.
_VOSK_LANGS = {"en": "en-us", "en-gb": "en-us", "": "en-us",
"pt": "pt", "zh": "cn", "cn": "cn"}
@ -108,7 +147,7 @@ _MODEL_LOCK = threading.Lock()
def _faster_whisper(audio, rate, model, language):
from faster_whisper import WhisperModel
key = ("faster-whisper", model or "base.en")
key = ("faster-whisper", resolve_model(model or "base.en"))
with _MODEL_LOCK:
engine = _MODELS.get(key)
if engine is None:
@ -129,7 +168,7 @@ def _faster_whisper(audio, rate, model, language):
def _openai_whisper(audio, rate, model, language):
import whisper
key = ("whisper", model or "base.en")
key = ("whisper", resolve_model(model or "base.en"))
with _MODEL_LOCK:
engine = _MODELS.get(key)
if engine is None:
@ -178,7 +217,7 @@ def _vosk(audio, rate, model, language):
# Vosk names its bundled models by region, so the plain codes the rest of
# the program uses have to be mapped or it reports "lang en does not exist".
lang = _VOSK_LANGS.get((language or "en").lower(), language or "en-us")
path = Path(model).expanduser() if model else None
path = Path(resolve_model(model)).expanduser() if model else None
use_path = bool(path and path.is_dir())
key = ("vosk", str(path) if use_path else f"lang:{lang}")
with _MODEL_LOCK:

32
packaging/build-deb.sh
View file

@ -30,10 +30,11 @@ sys.exit(main())
EOF
chmod 755 "$pkgdir/usr/bin/bandsaunter"
# Every one of these is in Debian, so apt resolves the lot. The speech
# recognisers are not packaged for Debian and can only come from pip, so they
# are suggested rather than depended on -- transcription is off by default and
# says so plainly when no recogniser is present.
# Every one of these is in Debian, so apt resolves the lot. No speech
# recogniser is packaged for Debian, so that part ships as its own package
# (built by build-repo.sh) and is recommended rather than depended on: apt
# pulls it in from the repository, and a bare "dpkg -i" of this file still
# installs, without transcription.
cat > "$pkgdir/DEBIAN/control" <<EOF
Package: bandsaunter
Version: ${version}-${revision}
@ -42,7 +43,7 @@ Priority: optional
Architecture: ${arch}
Depends: python3 (>= 3.10), python3-numpy, python3-scipy, python3-rich,
python3-yaml, librtlsdr0
Recommends: espeak-ng
Recommends: bandsaunter-transcribe, espeak-ng
Suggests: rtl-sdr
Maintainer: bandsaunter
Installed-Size: $(du -ks "$pkgdir" | cut -f1)
@ -66,6 +67,27 @@ if [ "$1" = configure ]; then
echo 'bandsaunter: blacklisted dvb_usb_rtl28xxu; unplug and replug the'
echo ' receiver, or run: sudo rmmod dvb_usb_rtl28xxu'
fi
# Said once, at install time, only when there is genuinely nothing to
# transcribe with -- otherwise the recogniser package has already
# arrived and there is nothing to explain.
if ! python3 -c 'import importlib.util as u, sys
sys.path.append("/usr/lib/bandsaunter/vendor")
try:
found = any(u.find_spec(m) for m in
("faster_whisper", "whisper", "vosk", "pocketsphinx"))
except Exception:
found = True # something is there, however unhappy; say nothing
sys.exit(0 if found else 1)' \
2>/dev/null; then
echo 'bandsaunter: speech transcription is unavailable -- no recogniser'
echo ' is installed. Scanning, recording and CW decoding all'
echo ' work without one. To add transcription, either:'
echo ' apt install bandsaunter-transcribe'
echo ' (from the repository this package came from), or'
echo ' pip install --break-system-packages faster-whisper'
echo ' Then: bandsaunter transcribe --engines'
fi
fi
exit 0
EOF

121
packaging/build-repo.sh Executable file
View file

@ -0,0 +1,121 @@
#!/bin/sh
# Build every package and an apt repository to serve them from.
#
# Produces three packages:
# bandsaunter the application itself
# bandsaunter-transcribe the speech recogniser, which Debian does not
# package, vendored into a private directory
# bandsaunter-model-<name> the recogniser's model, so a machine never
# has to reach the network
#
# The result is a flat apt repository: point a machine at it and
# "apt install bandsaunter" brings the lot, offline from then on.
set -eu
here=$(cd "$(dirname "$0")/.." && pwd)
out=${1:-$here/dist/repo}
model=${MODEL:-base.en}
hf_repo=${HF_REPO:-Systran/faster-whisper-$model}
revision=${DEB_REVISION:-1}
version=$(cd "$here" && python3 -c 'import bandsaunter; print(bandsaunter.debian_version())')
# Compiled wheels tie this to one architecture. amd64 only, by design.
arch=amd64
mkdir -p "$out"
stage=$(mktemp -d)
trap 'rm -rf "$stage"' EXIT
say() { printf '%s\n' "$*" >&2; }
# ---------------------------------------------------------------------------
say "building bandsaunter ${version}-${revision}"
"$here/packaging/build-deb.sh" "$out" >/dev/null
# ---------------------------------------------------------------------------
say "collecting the speech recogniser (this downloads from PyPI once)"
wheels="$stage/wheels"
mkdir -p "$wheels"
pip download --quiet --only-binary=:all: --dest "$wheels" faster-whisper >&2
pkg="$stage/bandsaunter-transcribe_${version}-${revision}_${arch}"
vendor="$pkg/usr/lib/bandsaunter/vendor"
mkdir -p "$vendor" "$pkg/DEBIAN"
for w in "$wheels"/*.whl; do
python3 -m zipfile -e "$w" "$vendor/"
done
# Debian already provides these and they win on sys.path anyway, so shipping
# them would be dead weight.
rm -rf "$vendor"/numpy "$vendor"/numpy-* "$vendor"/yaml "$vendor"/PyYAML-* \
"$vendor"/setuptools "$vendor"/setuptools-* "$vendor"/pkg_resources \
"$vendor"/_yaml "$vendor"/_distutils_hack "$vendor"/numpy.libs
# A .pth file only runs inside a real site directory; here it is dead weight.
rm -f "$vendor"/*.pth
find "$vendor" -name "__pycache__" -type d -exec rm -rf {} + 2>/dev/null || true
cat > "$pkg/DEBIAN/control" <<EOF
Package: bandsaunter-transcribe
Version: ${version}-${revision}
Section: hamradio
Priority: optional
Architecture: ${arch}
Depends: bandsaunter (= ${version}-${revision}), python3-numpy, python3-yaml
Recommends: bandsaunter-model-$(echo "$model" | tr '._' '--')
Maintainer: bandsaunter
Installed-Size: $(du -ks "$pkg" | cut -f1)
Description: speech recogniser for bandsaunter
Transcribes recorded voice transmissions to text.
.
No speech recogniser is packaged for Debian, so faster-whisper and its
dependencies are installed here into a private directory rather than into
dist-packages. That directory is searched after the system one, so anything
apt provides still takes precedence and these copies only fill the gap.
EOF
fakeroot dpkg-deb --build -Zxz "$pkg" "$out" >/dev/null
# ---------------------------------------------------------------------------
say "collecting the $model model"
modelpkg="$stage/bandsaunter-model-$(echo "$model" | tr '._' '--')_${version}-${revision}_all"
modeldir="$modelpkg/usr/share/bandsaunter/models/$model"
mkdir -p "$modeldir" "$modelpkg/DEBIAN"
PYTHONPATH="$stage/wheels-unused" python3 - "$hf_repo" "$modeldir" <<'PY' >&2
import shutil, sys
from pathlib import Path
from huggingface_hub import snapshot_download
src = Path(snapshot_download(sys.argv[1]))
dst = Path(sys.argv[2])
for item in src.iterdir():
if item.name.startswith("."):
continue
target = dst / item.name
# Resolve the cache's symlinks so the package holds real files.
shutil.copy2(item.resolve(), target)
PY
cat > "$modelpkg/DEBIAN/control" <<EOF
Package: bandsaunter-model-$(echo "$model" | tr '._' '--')
Version: ${version}-${revision}
Section: hamradio
Priority: optional
Architecture: all
Depends: bandsaunter-transcribe
Maintainer: bandsaunter
Installed-Size: $(du -ks "$modelpkg" | cut -f1)
Description: $model speech model for bandsaunter
The $model recognition model, installed locally so that transcription works
without reaching the network. Without this package the recogniser downloads
the model the first time it is used, on every machine.
EOF
fakeroot dpkg-deb --build -Zxz "$modelpkg" "$out" >/dev/null
# ---------------------------------------------------------------------------
say "indexing the repository"
( cd "$out" && dpkg-scanpackages --multiversion . /dev/null > Packages 2>/dev/null
gzip -9kfn Packages
apt-ftparchive -o APT::FTPArchive::Release::Suite=stable \
-o APT::FTPArchive::Release::Codename=bandsaunter \
release . > Release )
say ""
say "repository ready: $out"
ls -1sh "$out"/*.deb >&2

48
tests/test_transcribe.py
View file

@ -1,4 +1,5 @@
"""Speech to text: the engine plumbing, and how captures reach it."""
import sys
import time
from datetime import datetime
from pathlib import Path
@ -376,3 +377,50 @@ def test_vosk_accepts_the_plain_language_code():
result = tr.transcribe(say("one two three", 16000), 16000, engine="vosk",
language="en")
assert result is not None and not result.note, result.note
# --- packaged engine and models -------------------------------------------
def test_a_packaged_model_is_used_in_place_of_its_name(tmp_path, monkeypatch):
""""base.en" should mean the copy on this machine when one is installed."""
monkeypatch.setattr(tr, "MODEL_DIR", tmp_path)
(tmp_path / "base.en").mkdir()
assert tr.resolve_model("base.en") == str(tmp_path / "base.en")
def test_an_unpackaged_model_keeps_its_name_to_be_downloaded(tmp_path, monkeypatch):
monkeypatch.setattr(tr, "MODEL_DIR", tmp_path)
assert tr.resolve_model("small.en") == "small.en"
assert tr.resolve_model("") == ""
def test_an_explicit_model_directory_wins_over_the_packaged_one(tmp_path, monkeypatch):
packaged = tmp_path / "packaged"
(packaged / "base.en").mkdir(parents=True)
monkeypatch.setattr(tr, "MODEL_DIR", packaged)
mine = tmp_path / "base.en"
mine.mkdir()
assert tr.resolve_model(str(mine)) == str(mine)
def test_the_vendored_engine_is_searched_after_the_system_one(tmp_path, monkeypatch):
"""Anything apt provides must still win; the vendor copy fills a gap."""
monkeypatch.setattr(sys, "path", list(sys.path))
monkeypatch.setattr(tr, "VENDOR_DIR", tmp_path)
tr._add_vendor_path()
assert sys.path[-1] == str(tmp_path)
def test_the_vendor_path_is_added_once(tmp_path, monkeypatch):
monkeypatch.setattr(sys, "path", list(sys.path))
monkeypatch.setattr(tr, "VENDOR_DIR", tmp_path)
tr._add_vendor_path()
tr._add_vendor_path()
assert sys.path.count(str(tmp_path)) == 1
def test_a_missing_vendor_directory_is_not_added(tmp_path, monkeypatch):
monkeypatch.setattr(sys, "path", list(sys.path))
monkeypatch.setattr(tr, "VENDOR_DIR", tmp_path / "absent")
tr._add_vendor_path()
assert str(tmp_path / "absent") not in sys.path