diff --git a/README.md b/README.md index 26f5e8f..8fecabf 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,52 @@ sudo apt install ./dist/bandsaunter_*.deb apt pulls in every dependency itself, and the package blacklists the DVB-T driver that would otherwise claim the receiver. Nothing else to do. +Speech transcription is the one part that cannot come from Debian, because no +speech recogniser is packaged there. Installed this way, everything else works +and the install prints a short note saying how to add one. To have that arrive +by apt as well, build the repository below instead. + +### From your own apt repository + +For installing on several machines, or on a fresh one, without hunting for the +recogniser afterwards. `build-repo.sh` builds three packages and an apt index: + +| Package | Arch | Contents | +|---|---|---| +| `bandsaunter` | all | the application | +| `bandsaunter-transcribe` | amd64 | faster-whisper and its dependencies, in `/usr/lib/bandsaunter/vendor` | +| `bandsaunter-model-base-en` | all | the `base.en` model, in `/usr/share/bandsaunter/models` | + +```bash +./packaging/build-repo.sh # writes dist/repo/ +rsync -a dist/repo/ server:/var/www/html/bandsaunter/ +``` + +Serve that directory over HTTP from anywhere on the LAN, then on each machine: + +```bash +echo 'deb [trusted=yes] http://server/bandsaunter ./' \ + | sudo tee /etc/apt/sources.list.d/bandsaunter.list +sudo apt update +sudo apt install bandsaunter +``` + +That single command brings the recogniser and its model too — they are +`Recommends`, which apt installs by default. `--no-install-recommends` gets +just the application. Nothing reaches the network afterwards: the model is on +disk, so the first transcription works offline. + +`[trusted=yes]` skips signing, which is the sensible trade on a private LAN. +To sign it instead, run `gpg --clearsign` over `dist/repo/Release` to produce +`InRelease` and drop the `[trusted=yes]`. + +The vendored packages are appended to `sys.path`, never prepended, so anything +apt provides — numpy, PyYAML — still wins; the vendor directory only fills the +gap Debian leaves. `BANDSAUNTER_VENDOR_DIR` and `BANDSAUNTER_MODEL_DIR` +override both locations. + +Rebuilding for a new version is the same command; `apt upgrade` picks it up. + ### From source ```bash @@ -79,13 +125,18 @@ pip install vosk # ~10 MB plus a 40 MB model, weaker on noise bandsaunter transcribe --engines ``` -That is why transcription is `Suggests:` rather than `Depends:` in the -package. Debian Policy forbids anything in the archive from requiring -software outside it, and a `postinst` that reaches out to PyPI would break -offline and reproducible installs — so a package simply cannot pull these in. -Transcription is therefore off by default and reports plainly when no -recogniser is present, rather than the install failing or the feature -appearing broken. +That is why the plain `.deb` cannot depend on one. Debian Policy forbids +anything in the archive from requiring software outside it, and a `postinst` +that reaches out to PyPI would break offline and reproducible installs — so a +package in the archive simply cannot pull these in. Transcription is therefore +off by default and reports plainly when no recogniser is present, rather than +the install failing or the feature appearing broken. + +A repository of your own is not bound by that rule, which is what +[`build-repo.sh`](#from-your-own-apt-repository) exploits: it packages +faster-whisper and its model itself, into a private directory rather than into +`dist-packages`, and lets apt install them alongside. Nothing is downloaded at +install time, and nothing collides with an apt-managed module. Mixing the two is nonetheless fine here. Modern Debian marks the system Python as externally managed (PEP 668), so a pip install lands in your user @@ -100,11 +151,11 @@ from the `.deb` into `/usr/lib/python3/dist-packages` picks it up with no further configuration — verified, not assumed. A virtual environment works too, as long as bandsaunter runs inside it. -If you would rather keep everything under apt, the only route is packaging a -recogniser for Debian yourself. That is a real undertaking for whisper: it -would mean packaging ctranslate2, tokenizers, onnxruntime and their -dependencies, several of which are large C++ or Rust projects. It is why -none of them are there. +Getting these into Debian proper would be a different matter: it would mean +packaging ctranslate2, tokenizers, onnxruntime and their dependencies, several +of which are large C++ or Rust projects, each to archive standards. That is +why none of them are there, and why the local repository vendors the wheels +instead of trying to do it properly. ### Other distributions @@ -682,7 +733,8 @@ depends on a trained model, so there is no built-in fallback. ```bash bandsaunter transcribe --engines # what is installed -pip install faster-whisper # the recommended one +sudo apt install bandsaunter-transcribe # from your own repository +pip install faster-whisper # or straight from PyPI ``` | Engine | Notes | diff --git a/bandsaunter/transcribe.py b/bandsaunter/transcribe.py index 6e1b06a..edd2ab6 100644 --- a/bandsaunter/transcribe.py +++ b/bandsaunter/transcribe.py @@ -11,6 +11,7 @@ from __future__ import annotations import os import queue import shutil +import sys import subprocess import tempfile import threading @@ -24,12 +25,50 @@ import numpy as np from ._quiet import suppress_stderr __all__ = ["Transcript", "transcribe", "available_engine", "ENGINES", - "TranscriptionWorker", "describe_engines"] + "TranscriptionWorker", "describe_engines", "resolve_model", + "VENDOR_DIR", "MODEL_DIR"] # Best first. Whisper handles the noise and clipping of radio audio far # better than the smaller recognisers, which were trained on clean speech. ENGINES = ("faster-whisper", "whisper", "whisper-cli", "vosk", "pocketsphinx") +# No recogniser is packaged for Debian, so a package that wants to supply one +# ships it here instead of in dist-packages. Appended to the path rather than +# prepended, so anything the system package manager provides still wins and a +# vendored copy only ever fills a gap. +VENDOR_DIR = Path(os.environ.get("BANDSAUNTER_VENDOR_DIR", + "/usr/lib/bandsaunter/vendor")) +# Models shipped alongside, so a machine never has to reach the network. +MODEL_DIR = Path(os.environ.get("BANDSAUNTER_MODEL_DIR", + "/usr/share/bandsaunter/models")) + + +def _add_vendor_path() -> None: + if VENDOR_DIR.is_dir(): + path = str(VENDOR_DIR) + if path not in sys.path: + sys.path.append(path) + + +_add_vendor_path() + + +def resolve_model(name: str) -> str: + """A packaged model directory if there is one, otherwise the name itself. + + Lets ``base.en`` mean the copy installed on this machine when one exists, + and fall back to the recogniser downloading it when one does not. + """ + if not name: + return name + given = Path(name).expanduser() + if given.is_dir(): + return str(given) + packaged = MODEL_DIR / name + if packaged.is_dir(): + return str(packaged) + return name + # Vosk's bundled models are named by region. _VOSK_LANGS = {"en": "en-us", "en-gb": "en-us", "": "en-us", "pt": "pt", "zh": "cn", "cn": "cn"} @@ -108,7 +147,7 @@ _MODEL_LOCK = threading.Lock() def _faster_whisper(audio, rate, model, language): from faster_whisper import WhisperModel - key = ("faster-whisper", model or "base.en") + key = ("faster-whisper", resolve_model(model or "base.en")) with _MODEL_LOCK: engine = _MODELS.get(key) if engine is None: @@ -129,7 +168,7 @@ def _faster_whisper(audio, rate, model, language): def _openai_whisper(audio, rate, model, language): import whisper - key = ("whisper", model or "base.en") + key = ("whisper", resolve_model(model or "base.en")) with _MODEL_LOCK: engine = _MODELS.get(key) if engine is None: @@ -178,7 +217,7 @@ def _vosk(audio, rate, model, language): # Vosk names its bundled models by region, so the plain codes the rest of # the program uses have to be mapped or it reports "lang en does not exist". lang = _VOSK_LANGS.get((language or "en").lower(), language or "en-us") - path = Path(model).expanduser() if model else None + path = Path(resolve_model(model)).expanduser() if model else None use_path = bool(path and path.is_dir()) key = ("vosk", str(path) if use_path else f"lang:{lang}") with _MODEL_LOCK: diff --git a/packaging/build-deb.sh b/packaging/build-deb.sh index 5debe0e..5b03ecd 100755 --- a/packaging/build-deb.sh +++ b/packaging/build-deb.sh @@ -30,10 +30,11 @@ sys.exit(main()) EOF chmod 755 "$pkgdir/usr/bin/bandsaunter" -# Every one of these is in Debian, so apt resolves the lot. The speech -# recognisers are not packaged for Debian and can only come from pip, so they -# are suggested rather than depended on -- transcription is off by default and -# says so plainly when no recogniser is present. +# Every one of these is in Debian, so apt resolves the lot. No speech +# recogniser is packaged for Debian, so that part ships as its own package +# (built by build-repo.sh) and is recommended rather than depended on: apt +# pulls it in from the repository, and a bare "dpkg -i" of this file still +# installs, without transcription. cat > "$pkgdir/DEBIAN/control" <= 3.10), python3-numpy, python3-scipy, python3-rich, python3-yaml, librtlsdr0 -Recommends: espeak-ng +Recommends: bandsaunter-transcribe, espeak-ng Suggests: rtl-sdr Maintainer: bandsaunter Installed-Size: $(du -ks "$pkgdir" | cut -f1) @@ -66,6 +67,27 @@ if [ "$1" = configure ]; then echo 'bandsaunter: blacklisted dvb_usb_rtl28xxu; unplug and replug the' echo ' receiver, or run: sudo rmmod dvb_usb_rtl28xxu' fi + + # Said once, at install time, only when there is genuinely nothing to + # transcribe with -- otherwise the recogniser package has already + # arrived and there is nothing to explain. + if ! python3 -c 'import importlib.util as u, sys +sys.path.append("/usr/lib/bandsaunter/vendor") +try: + found = any(u.find_spec(m) for m in + ("faster_whisper", "whisper", "vosk", "pocketsphinx")) +except Exception: + found = True # something is there, however unhappy; say nothing +sys.exit(0 if found else 1)' \ + 2>/dev/null; then + echo 'bandsaunter: speech transcription is unavailable -- no recogniser' + echo ' is installed. Scanning, recording and CW decoding all' + echo ' work without one. To add transcription, either:' + echo ' apt install bandsaunter-transcribe' + echo ' (from the repository this package came from), or' + echo ' pip install --break-system-packages faster-whisper' + echo ' Then: bandsaunter transcribe --engines' + fi fi exit 0 EOF diff --git a/packaging/build-repo.sh b/packaging/build-repo.sh new file mode 100755 index 0000000..cd4b4f8 --- /dev/null +++ b/packaging/build-repo.sh @@ -0,0 +1,121 @@ +#!/bin/sh +# Build every package and an apt repository to serve them from. +# +# Produces three packages: +# bandsaunter the application itself +# bandsaunter-transcribe the speech recogniser, which Debian does not +# package, vendored into a private directory +# bandsaunter-model- the recogniser's model, so a machine never +# has to reach the network +# +# The result is a flat apt repository: point a machine at it and +# "apt install bandsaunter" brings the lot, offline from then on. +set -eu + +here=$(cd "$(dirname "$0")/.." && pwd) +out=${1:-$here/dist/repo} +model=${MODEL:-base.en} +hf_repo=${HF_REPO:-Systran/faster-whisper-$model} +revision=${DEB_REVISION:-1} +version=$(cd "$here" && python3 -c 'import bandsaunter; print(bandsaunter.debian_version())') + +# Compiled wheels tie this to one architecture. amd64 only, by design. +arch=amd64 + +mkdir -p "$out" +stage=$(mktemp -d) +trap 'rm -rf "$stage"' EXIT + +say() { printf '%s\n' "$*" >&2; } + +# --------------------------------------------------------------------------- +say "building bandsaunter ${version}-${revision}" +"$here/packaging/build-deb.sh" "$out" >/dev/null + +# --------------------------------------------------------------------------- +say "collecting the speech recogniser (this downloads from PyPI once)" +wheels="$stage/wheels" +mkdir -p "$wheels" +pip download --quiet --only-binary=:all: --dest "$wheels" faster-whisper >&2 + +pkg="$stage/bandsaunter-transcribe_${version}-${revision}_${arch}" +vendor="$pkg/usr/lib/bandsaunter/vendor" +mkdir -p "$vendor" "$pkg/DEBIAN" +for w in "$wheels"/*.whl; do + python3 -m zipfile -e "$w" "$vendor/" +done +# Debian already provides these and they win on sys.path anyway, so shipping +# them would be dead weight. +rm -rf "$vendor"/numpy "$vendor"/numpy-* "$vendor"/yaml "$vendor"/PyYAML-* \ + "$vendor"/setuptools "$vendor"/setuptools-* "$vendor"/pkg_resources \ + "$vendor"/_yaml "$vendor"/_distutils_hack "$vendor"/numpy.libs +# A .pth file only runs inside a real site directory; here it is dead weight. +rm -f "$vendor"/*.pth +find "$vendor" -name "__pycache__" -type d -exec rm -rf {} + 2>/dev/null || true + +cat > "$pkg/DEBIAN/control" </dev/null + +# --------------------------------------------------------------------------- +say "collecting the $model model" +modelpkg="$stage/bandsaunter-model-$(echo "$model" | tr '._' '--')_${version}-${revision}_all" +modeldir="$modelpkg/usr/share/bandsaunter/models/$model" +mkdir -p "$modeldir" "$modelpkg/DEBIAN" +PYTHONPATH="$stage/wheels-unused" python3 - "$hf_repo" "$modeldir" <<'PY' >&2 +import shutil, sys +from pathlib import Path +from huggingface_hub import snapshot_download +src = Path(snapshot_download(sys.argv[1])) +dst = Path(sys.argv[2]) +for item in src.iterdir(): + if item.name.startswith("."): + continue + target = dst / item.name + # Resolve the cache's symlinks so the package holds real files. + shutil.copy2(item.resolve(), target) +PY + +cat > "$modelpkg/DEBIAN/control" </dev/null + +# --------------------------------------------------------------------------- +say "indexing the repository" +( cd "$out" && dpkg-scanpackages --multiversion . /dev/null > Packages 2>/dev/null + gzip -9kfn Packages + apt-ftparchive -o APT::FTPArchive::Release::Suite=stable \ + -o APT::FTPArchive::Release::Codename=bandsaunter \ + release . > Release ) + +say "" +say "repository ready: $out" +ls -1sh "$out"/*.deb >&2 diff --git a/tests/test_transcribe.py b/tests/test_transcribe.py index 8edbb2f..a7b7a10 100644 --- a/tests/test_transcribe.py +++ b/tests/test_transcribe.py @@ -1,4 +1,5 @@ """Speech to text: the engine plumbing, and how captures reach it.""" +import sys import time from datetime import datetime from pathlib import Path @@ -376,3 +377,50 @@ def test_vosk_accepts_the_plain_language_code(): result = tr.transcribe(say("one two three", 16000), 16000, engine="vosk", language="en") assert result is not None and not result.note, result.note + + +# --- packaged engine and models ------------------------------------------- + +def test_a_packaged_model_is_used_in_place_of_its_name(tmp_path, monkeypatch): + """"base.en" should mean the copy on this machine when one is installed.""" + monkeypatch.setattr(tr, "MODEL_DIR", tmp_path) + (tmp_path / "base.en").mkdir() + assert tr.resolve_model("base.en") == str(tmp_path / "base.en") + + +def test_an_unpackaged_model_keeps_its_name_to_be_downloaded(tmp_path, monkeypatch): + monkeypatch.setattr(tr, "MODEL_DIR", tmp_path) + assert tr.resolve_model("small.en") == "small.en" + assert tr.resolve_model("") == "" + + +def test_an_explicit_model_directory_wins_over_the_packaged_one(tmp_path, monkeypatch): + packaged = tmp_path / "packaged" + (packaged / "base.en").mkdir(parents=True) + monkeypatch.setattr(tr, "MODEL_DIR", packaged) + mine = tmp_path / "base.en" + mine.mkdir() + assert tr.resolve_model(str(mine)) == str(mine) + + +def test_the_vendored_engine_is_searched_after_the_system_one(tmp_path, monkeypatch): + """Anything apt provides must still win; the vendor copy fills a gap.""" + monkeypatch.setattr(sys, "path", list(sys.path)) + monkeypatch.setattr(tr, "VENDOR_DIR", tmp_path) + tr._add_vendor_path() + assert sys.path[-1] == str(tmp_path) + + +def test_the_vendor_path_is_added_once(tmp_path, monkeypatch): + monkeypatch.setattr(sys, "path", list(sys.path)) + monkeypatch.setattr(tr, "VENDOR_DIR", tmp_path) + tr._add_vendor_path() + tr._add_vendor_path() + assert sys.path.count(str(tmp_path)) == 1 + + +def test_a_missing_vendor_directory_is_not_added(tmp_path, monkeypatch): + monkeypatch.setattr(sys, "path", list(sys.path)) + monkeypatch.setattr(tr, "VENDOR_DIR", tmp_path / "absent") + tr._add_vendor_path() + assert str(tmp_path / "absent") not in sys.path