commit db3e0c79b9517103820672774d1beb97fe827855 Author: The Dust Council Date: Fri Aug 21 20:50:20 2026 -0700 Initial commit: bandsaunter, an RTL-SDR signal scanner Sweeps any set of frequency ranges, records what it finds, and works out what kind of signal it was. - Frequency ranges entered by hand or picked from a 135-entry US band plan, including whole-band and all-CW sweeps that resolve the demodulator per segment. - Detection calibrated against the peak-hold detector's own noise statistics, so the threshold means real margin over static rather than over the floor. - A content gate: captures are kept only if they carry voice, decodable CW, or an identified digital keying scheme. Speech is recognised by a pitch track that drifts, which static cannot imitate. - Identification of NFM/WFM/AM/SSB, CW with Morse decoded to text, P25, DMR, NXDN, D-STAR, POCSAG, FLEX, ACARS, AIS, APRS, n-FSK and n-PSK. - Gapless streaming capture, with the signal path fast enough to keep up in real time, so recordings play back at the right speed. - Optional one-file-per-frequency recording with spoken timestamps, and speech-to-text transcription. - Menus and command line generated from one settings table, so neither can offer something the other cannot; settings persist in ~/.config. 367 tests, run against synthetic signals, a built-in receiver simulator, and real hardware. Co-Authored-By: Claude Opus 5 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..e5a6c1a --- /dev/null +++ b/.gitignore @@ -0,0 +1,18 @@ +__pycache__/ +*.py[cod] +*.egg-info/ +.pytest_cache/ +build/ +dist/ +recordings/ +*.wav +*.cf32 +*.cs16 +*.sigmf-meta + +# transcripts land beside recordings +*_transcription.txt + +# local environments +.venv/ +venv/ diff --git a/README.md b/README.md new file mode 100644 index 0000000..ca9bbc6 --- /dev/null +++ b/README.md @@ -0,0 +1,723 @@ +# bandsaunter + +A signal scanner, recorder and identifier for RTL-SDR receivers. + +Give it any number of frequency ranges — typed in by hand or picked from a +built-in US band plan — and it sweeps them, stops on anything above the noise +floor, records it, and works out what kind of signal it was. CW/Morse is +decoded to text. + +``` +╭──────────────────────────────── receiver ────────────────────────────────╮ +│ Rafael Micro R820T/R820T2 2.048 MS/s gain auto +0 ppm │ +╰──────────────────────────────────────────────────────────────────────────╯ +╭───────────────────────────────── sweep ──────────────────────────────────╮ +│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ step 4/6 146 MHz - 146.666667 MHz │ +│ ▆ █ ▄▄▄ peak -18.0 dBFS │ +│ recording cycle 2 hits 3 dropped 0 detections 7 up 0:04 │ +╰──────────────────────────────────────────────────────────────────────────╯ +╭──────────────────────────────────────────────────────────────────────────╮ +│ REC 146.52 MHz [nfm] ███████████░░░░░░░░░ 11.4/30s SIGNAL SNR 27 dB │ +╰──────────────────────────────────────────────────────────────────────────╯ +╭───────────────────────────── recorded signals ───────────────────────────╮ +│ 19:38:43 460.025 MHz 8.2s 27.0 P25 Phase 1 C4FM digital voice │ +│ 19:38:43 144.1 MHz 8.2s 27.0 CW / Morse at 18 WPM "VVV DE…" │ +│ 19:38:43 146.52 MHz 8.2s 27.0 Narrowband FM voice (CTCSS 100) │ +╰──────────────────────────────────────────────────────────────────────────╯ +``` + +## Install + +```bash +sudo apt install rtl-sdr librtlsdr0 espeak-ng # Debian/Ubuntu +pip install -e . +``` + +### Dependencies + +| | Package | Needed for | +|---|---|---| +| **required** | `librtlsdr0` (`rtl-sdr`) | talking to the dongle at all | +| **required** | `numpy`, `scipy` | all signal processing | +| **required** | `rich` | the menus and the live display | +| **required** | `PyYAML` | the settings file and profiles | +| *recommended* | `espeak-ng` | clearer spoken timestamps | +| *optional* | `faster-whisper` (pip) | transcribing speech to text | + +`espeak-ng` is a **recommendation, not a requirement**: without it the spoken +timestamps are produced by a built-in formant synthesiser, so the feature +works on a machine with nothing else installed. With it they are clearer and +render about three times faster. Any of `espeak`, `pico2wave`, `flite` or +macOS `say` is used if found instead. + +For a Debian package that maps to `Depends: librtlsdr0, python3-numpy, +python3-scipy, python3-rich, python3-yaml`, `Recommends: espeak-ng, +rtl-sdr` and `Suggests: python3-vosk` — transcription is off by default and +the scan reports it plainly when no recogniser is installed. `rtl-sdr` itself only supplies command-line tools bandsaunter does not +call — the library is what matters. + +`bandsaunter scan --combine --dry-run` reports which engine will be used. + +If the dongle is not accessible, the DVB-T kernel driver has usually claimed +it: + +```bash +echo 'blacklist dvb_usb_rtl28xxu' | sudo tee /etc/modprobe.d/blacklist-rtl.conf +sudo rmmod dvb_usb_rtl28xxu +``` + +Check with `bandsaunter devices --test`. + +## Quick start + +```bash +bandsaunter # the menus: set up and scan +bandsaunter scan -b 2m -b marine-vhf # band-plan presets +bandsaunter scan -r 144M-148M -r 420M-450M # your own ranges +bandsaunter scan -b 2m --simulate # try it without hardware +``` + +## Two ways to drive it + +Everything is available both ways. Run `bandsaunter` with no arguments for the +menus, or pass flags for scripting — the two are generated from the same +table of settings, so neither can offer something the other cannot. + +``` + 1 Frequency ranges 3 configured + 2 Band plan 107 US presets + 3 Settings record no limit, hang 6s, squelch +12 dB, keep voice, cw + 4 Saved settings and profiles + h Help + s Start scanning + q Quit +``` + +Settings are grouped, show their current value against the built-in default, +and carry their own help: + +``` + # setting value what it does + 1 Record for * no limit longest one signal may hold the receiver + 2 Wait for quiet * 6 s quiet time before the sweep resumes + 3 Absolute limit 900 s ceiling on one capture, even when 'Record for' is 0 +* differs from the built-in default + +Number to change it, ?N for help on one, d to reset the group, blank to go back. +``` + +`?2` explains a setting in full, including the command-line flag that does the +same thing. Typing a search term instead of a number finds settings by any +word in their name or description — `voice score` finds the speech threshold. + +Values may be typed with their units: `5 s`, `2.048 MHz`, `12 dB`, `48k`, or +`no limit` for the settings that accept 0. + +## Settings that persist + +Settings are saved to `~/.config/bandsaunter/config.yaml` and picked up by every +later run. Save them from the menus (**4 → s**) or from the command line: + +```bash +bandsaunter config # open the settings menu +bandsaunter config hang_seconds=6 record_seconds=0 # set and save directly +bandsaunter config --show # every setting, with defaults +bandsaunter config --describe hang_seconds # explain one in full +bandsaunter config --path # where the file lives +bandsaunter config --reset # back to defaults +``` + +Three layers apply in order, each overriding the last: + +1. the saved settings file +2. a named profile, if `--profile` is given +3. any flags on the command line + +So a saved squelch of 12 dB stays in force while `--hang 1.5` overrides just +the hang for one run. `--no-config` ignores the file entirely; `--save` stores +the resulting settings as the new default. + +Named profiles live beside it in the same directory: + +```bash +bandsaunter scan -b 2m -b 70cm --record 0 --hang 6 --save-profile local +bandsaunter scan -p local +bandsaunter profiles +``` + +## Entering frequencies + +**By hand** — repeat `-r` as many times as you like; there is no limit on the +number of start/end pairs. + +```bash +bandsaunter scan -r 144M-148M -r 462.5M-467.8M -r 929M-932M +``` + +A range is `start-end`, with optional `/step` and `@mode`: + +| Form | Meaning | +|---|---| +| `144M-148M` | explicit start and end | +| `144-148M` | the unit carries over to the left end | +| `146.52M` | a single frequency | +| `144M-148M/25k` | with a channel step | +| `144M-148M/25k@nfm` | and a forced demodulator | + +Units may be written `144M`, `144 MHz`, `144000k`, or plain Hz. A bare number +below 10000 is read as MHz, so `-r 162.4-162.55` does what you expect. + +**From the US band plan** — 107 presets across 18 categories: + +```bash +bandsaunter bands --categories # list categories +bandsaunter bands --category Aviation # everything in one category +bandsaunter bands pager # search +bandsaunter scan -b gmrs -b railroad -b noaa-weather +``` + +Each preset carries its own channel spacing, demodulator and bandwidth, so +`-b marine-vhf` scans 25 kHz channels in NFM while `-b fm-broadcast` uses +200 kHz WFM, without being told. + +Each amateur band also has a **complete** entry that covers the whole band and +picks the demodulator per segment, because a band is not one mode: + +```bash +bandsaunter scan -b 2m-complete +``` + +scans 144-148 MHz as CW below 144.1, SSB to 144.3 and FM above it, so a CW +beacon at the bottom is decoded to text while a repeater at the top is +demodulated as FM — in one sweep. There is one for every band from 160 m to +33 cm: `160m-complete`, `80m-complete`, ... `70cm-complete`, `33cm-complete`. + +Where an amateur band overlaps another service the amateur reading wins inside +a complete-band sweep — 433 MHz is treated as 70 cm rather than as the ISM +band it shares — while scanning `-r 433.9M` on its own still treats it as ISM. + +A few presets stand for a *set* of others, so scattered segments can be picked +in one go: + +```bash +bandsaunter scan -b all-cw --record 0 --hang 6 +``` + +`all-cw` covers every CW allocation in the plan — 160, 80, 40, 30, 20, 17, 15, +12, 10, 6 and 2 metres — as eleven separate ranges rather than one span from +1.8 to 144 MHz. That is 1.25 MHz of spectrum in total, so a full pass takes +under a second and CW gets decoded to text as it turns up. Direct sampling +switches itself on for the HF segments and off again from 12 m upward; the HF +part needs an HF antenna to be worth anything. + +The menus do both: browse the band plan by category, or type in start/end +pairs one after another. Ranges can be listed, removed, toggled on and off, +and have their demodulator changed from the ranges menu. + +## The two dwell settings + +These are the settings that decide how the scanner behaves when it finds +something: + +| Setting | Flag | What it does | +|---|---|---| +| Record for X seconds before continuing | `--record 30` | The longest a single signal may hold the receiver. `0` means stay as long as it keeps transmitting. | +| Wait for X seconds of no signal before continuing | `--hang 3` | How long the channel must stay quiet before the sweep resumes. **Gaps shorter than this are recorded straight through.** | + +```bash +bandsaunter scan -b 2m --record 30 --hang 3 +``` + +Whichever comes first wins, and that is worth being clear about: **`--record` +overrides `--hang`.** A transmission still in progress at the record limit is +cut off there, however long the hang time is. If a recording keeps ending at +exactly 30 seconds, that is the default record limit doing it, not the hang — +set `--record 0`. The scan reports it when this happens. + +Both are measured in **samples**, not wall-clock time, so a 30 second setting +produces a 30.0 second recording. + +Supporting settings: + +- `--min-record 0.5` — discard anything shorter, so brief noise spikes leave + nothing behind on disk. +- `--revisit 8` — ignore a frequency for this long after recording it, so a + busy repeater does not monopolise the sweep. +- `--max-record 900` — absolute ceiling on one capture, applied even when + `--record` is 0. +- `--threshold 8` — squelch, in dB above the measured noise floor. + +### Capturing both sides of a conversation + +`--hang` is what holds a recording open across the natural pauses in two-way +traffic. Set it longer than the gap between overs and the whole exchange lands +in one file: + +```bash +bandsaunter scan -b 2m --record 0 --hang 6 +``` + +`--record 0` is the important half. Without it the per-signal cap cuts the +exchange off mid-sentence no matter what the hang time is — the default 30 +seconds is a common surprise. `--max-record` (default 900 s) still bounds an +unlimited capture so nothing runs away. + +To make it the permanent default: + +```bash +bandsaunter config record_seconds=0 hang_seconds=6 +``` + +"Quiet" means *no real signal*, not merely a closed squelch. Silence, static +and interference all count towards the timer, so a burst of noise during a +pause does not reset it and park the receiver on a finished conversation. +Recognising that a signal carries nothing takes a couple of seconds of +evidence, so expect the tail to run a little past `--hang` in that case. + +Once a capture has produced real content it is never abandoned as noise, since +a quiet spell between overs would otherwise throw the conversation away. + +## Only real signals get recorded + +A power threshold cannot tell a transmission from a hump of interference, so +every capture is checked for *content* before it is kept. Recording happens +only for: + +| Category | What it means | +|---|---| +| `voice` | speech structure in the demodulated audio: a pitch track in the 70-400 Hz range that drifts the way intonation does, pauses between phrases, syllable-rate envelope modulation, and formants that move | +| `cw` | a keyed carrier whose timing resolves as Morse | +| `digital` | an identified keying scheme: discrete FSK levels, an M-PSK phase line, or on-off keying -- corroborated by a symbol rate | + +Everything else -- static, hum, switch-mode power supply harmonics, clock +spurs, bare carriers -- is discarded, and the files it wrote are deleted. + +The check runs *while* the capture is still going, so interference is dropped +after a second or two instead of holding the receiver for the whole record +time. + +```bash +bandsaunter scan -b 2m # voice, CW and digital (default) +bandsaunter scan -b 2m --accept voice # voice only +bandsaunter scan -b ism-433 --accept digital # data bursts only +bandsaunter scan -b 2m --keep-carriers # also keep unmodulated carriers +bandsaunter scan -b 2m --keep-everything # no content check at all +``` + +Tuning knobs: `--min-voice-score` (0-1, default 0.45) sets how speech-like +audio must be; `--min-signal-score` sets the confidence needed to keep +anything; `--verify-max` caps how long a contentless capture is given to prove +itself. + +### What makes this hold up against interference + +Static is good at imitating most of the things that look like structure, so +each test is built so that noise cannot pass it: + +- **A pitch track that moves gates the voice score**, rather than contributing + a share of it. Dynamics, syllable-rate modulation and energy in the voice + band are all things hiss does too; weighted alongside voicing they were + enough to carry noise over the line on their own. Speech is the only thing + here that produces a pitch period that drifts, so nothing is called voice + without one. A steady tone or mains hum has a perfectly stable "pitch" and + is rejected for exactly that reason. Measuring drift needs several voiced + frames, so the requirement eases for a short over that cannot supply them; + the steady-tone guard still applies. +- **A symbol rate has to hold still across the capture.** The estimator always + returns its best peak, so on noise it reports one rate for the first half of + a capture and a different one for the second. Real data keeps one. +- **Keying has to land on a grid.** On/off contrast alone is not evidence: a + signal fading across the squelch produces plenty of it, with run lengths + that fit no symbol period at all. +- **A phase line has to be created by the exponentiation.** An unmodulated + carrier -- including the gaps between phrases on an FM channel -- already + has a line at every power, and would otherwise look like textbook PSK. +- **The demodulator is chosen from the signal**, not from the band plan, by + probing briefly before recording. An AM signal inside a + band listed as FM would otherwise be recorded through the wrong detector, + giving audio that is useless to listen to and impossible to judge. Three + ratios settle it — how much the envelope varies, how far the tone swings, + and how much power sits in a carrier — because those hold steady over a + fraction of a second whatever is being said. Running the full classifier on + so short a probe was tried and is not reliable: speech makes any modulation + look bursty over half a second, and AM came back as on-off keying while FM + came back as AM. The probe is played into the recording rather than + discarded, so a short over does not lose its opening. +- **Content is judged only on the audio that was actually recorded**, and only + on the stretches the squelch called signal. Demodulating two ways and + keeping whichever scored higher is cherry-picking, and on noise one of the + two always flatters it. + +The gate needs roughly two seconds of audio to judge speech reliably, so pair +it with `--record 5` or more rather than very short capture limits. + +### Keeping up with the radio + +An RTL-SDR only delivers samples while the host is actively reading. Anything +that arrives while the program is busy demodulating is discarded by the +driver, and a recording then holds *less* than really went by -- which plays +back too fast. + +Two things keep that from happening: + +- **Captures stream asynchronously.** A ring of USB transfers stays queued in + a background thread, so the dongle is never waiting for the host. Sweeping + still uses plain reads, because each dwell is an independent snapshot and a + gap between them costs nothing. +- **The signal path is fast enough to keep up.** Decimation computes only the + samples that survive, rather than filtering at the input rate and throwing + away seven of every eight outputs; the quarter-rate local oscillator is the + four-step cycle 1, -j, -1, +j and needs no trigonometry; and pitch tracking + runs through the FFT instead of a direct autocorrelation per frame. Together + those took the capture loop from 65% of the real-time budget to under 10%. + +If the host does fall behind anyway, the scan reports how many samples were +lost rather than silently producing a fast recording. + +### Why the threshold is what it is + +The sweep uses peak-hold, which keeps the largest value each FFT bin reached +during the dwell. That finds bursty traffic that averaging would bury -- but +it also means noise alone rides several dB above the measured floor. On this +hardware, empty spectrum reaches 5-9 dB above a percentile floor with nothing +transmitting. + +So `--threshold` is a margin over *noise*, not over the floor: the offset that +noise alone clears is computed from the detector (segment count and bin count) +and added automatically. A threshold of 8 means 8 dB of real headroom. It is +deliberately not measured from the spectrum -- a spread estimated from the +data reads five times higher across the packed broadcast FM band than on empty +spectrum, which would suppress exactly the stations you are looking for. + +## Signal identification + +Every recording is classified from its own IQ. The classifier measures +occupied bandwidth, envelope statistics, discriminator levels, phase +behaviour, spectral flatness and symbol rate, then combines those with the +frequency to name the signal: + +| Family | Recognised as | +|---|---| +| Analogue voice | Narrowband FM (with CTCSS tone or DCS), wideband FM (stereo pilot detected), AM, SSB (USB/LSB) | +| CW | Keyed carrier, **decoded to text** with the speed in WPM | +| Digital voice | P25 C4FM, DMR (TDMA burst structure), NXDN, D-STAR | +| Data | POCSAG and FLEX paging, ACARS, AIS, APRS/AFSK1200, 2-FSK and 4-FSK, BPSK/QPSK/8-PSK | +| Other | Unmodulated carriers, OOK/ISM devices, ADS-B and UAT, DME/TACAN pulses, wideband OFDM/cellular | + +Each result carries a confidence and the reasoning behind it: + +``` +146.520038 MHz 3.0s SNR 27.6 dB Narrowband FM voice (CTCSS 100.0 Hz) (88%) + 4.9 kHz wide, 0.9 kHz rms deviation +``` + +Low SNR reduces confidence rather than producing a confident wrong answer. + +### CW / Morse + +Keyed carriers are decoded to text. The speed is measured from the signal, so +nothing has to be configured, and anything from about 8 to 40 WPM reads +reliably: + +``` +144.1 MHz 12.0s SNR 50.8 dB CW / Morse at 18 WPM CW "VVV DE W1AW FN31" +``` + +The decoder runs its own CW detector over the captured IQ, so Morse is found +even when the recording itself was made in FM or SSB. + +## Output + +Everything lands in one directory, named +`frequency--yyyy-mm-dd_hour_minute_second-modulation.wav`: + +``` +0014.058000MHz--2026-08-21_20_35_41-cw.wav +0098.299255MHz--2026-08-21_20_24_01-wfm.wav demodulated audio +0098.299255MHz--2026-08-21_20_24_01-wfm.json identification, features, timing +0098.299255MHz--2026-08-21_20_24_01-wfm.cf32 raw IQ (with --iq) +0098.299255MHz--2026-08-21_20_24_01-wfm.sigmf-meta SigMF sidecar (with --iq) +0098.299255MHz--2026-08-21_20_24_01-wfm_transcription.txt (with --transcribe) +0098.361991MHz--2026-08-21_20_24_18-wfm.wav +0146.520000MHz--2026-08-21_20_31_02-nfm.wav +1090.000000MHz--2026-08-21_20_38_12-raw.wav +scan_log.jsonl one line per hit +scan_log.csv the same, as a spreadsheet +``` + +Frequency leads and is padded to four digits, so a plain directory listing +sorts by frequency across the whole tuning range — unpadded, 1090 MHz would +sort before 146 MHz. Each channel's captures group together with the +timestamp ordering them. Every artefact of one capture shares a stem, and the +modulation suffix is what the signal was *identified* as, so the file is +renamed once the capture has been analysed. + +### Where files go + +The directory is asked for the first time bandsaunter is run and remembered +afterwards: + +``` +Recordings, transcripts and the scan log are all written to one directory. +Where would you like them? + + recordings directory (~/bandsaunter): +``` + +It is an ordinary setting, so it can be changed at any time: + +```bash +bandsaunter config output_dir=~/somewhere-else +bandsaunter scan -b 2m -o /tmp/just-this-once +``` + +The question is only asked when there is someone to answer it: a scan run from +a script or with output redirected uses the default rather than blocking, and +`--no-config` skips it entirely. + +### One file per frequency + +With `--combine`, each frequency gets a single file that every later reception +is appended to, so a whole watch on a channel plays back as one recording: + +```bash +bandsaunter scan -b 2m --combine --record 0 --hang 6 +``` + +``` +recordings/ + 146.520000MHz.wav every transmission heard on 146.520, in order + 147.100000MHz.wav + 146.520692MHz--2026-08-21_13_21_20-nfm.json what each capture was + scan_log.csv +``` + +Each transmission is preceded by **its date and time read aloud**, so the file +says when everything was heard without needing the log open. Receptions within +`--combine-tolerance` (6.25 kHz by default) of each other count as the same +channel, which covers the few hundred hertz a detection wanders by. + +The file is rewritten to stay valid after every append, so it can be opened +and played while the scan is still running, and a scan stopped part-way still +leaves a working recording. A later run continues the same file rather than +starting a new one. + +By default the per-transmission WAVs are removed once they have been added, so +each recording exists in one place; `--keep-individual` keeps both. The `.json` +describing each capture is written either way. + +`--no-announce` leaves the timestamps out, and `--announce-frequency` reads the +frequency out as well. + +### Where the speech comes from + +The announcements use an installed text-to-speech program if there is one +(`espeak-ng`, `espeak`, `pico2wave`, `flite`, `say`) and fall back to a +built-in formant synthesiser otherwise, so the feature works with nothing else +installed. `--announce-engine builtin` forces the built-in one; the scan plan +reports which is in use. + +The two are given different wording. An installed engine gets ordinary text, +where punctuation is what produces the phrasing — and the obvious spellings +are traps: espeak-ng reads `14:38:05` as "fourteen thirty, eight zero five", +and an ISO date as "two thousand and twenty six dash zero eight dash twenty +one". It is given `August 21, twenty twenty six, at 14 38 and 05 seconds` +instead. The built-in synthesiser gets the word list it has pronunciations +for. Both are level-matched, so switching between them does not change how +loud the announcements sit against the recordings. + +It only has to say numbers, month names and a handful of words, which makes +direct synthesis practical: a glottal source through a cascade of three +formant resonators, with the formant tracks interpolated between phonemes. +The test suite checks by LPC analysis that all ten vowels come out within +130 Hz of their intended first formant and 250 Hz of their second — summing +the resonators in parallel rather than cascading them loses the first formant +entirely and makes every vowel sound the same. + +### Transcribing speech to text + +With `--transcribe`, anything the content check identified as **voice** is +passed to a speech recogniser and the words are written beside the recording: + +```bash +bandsaunter scan -b 2m --transcribe +``` + +``` +146.520000MHz--2026-08-21_12_18_38-nfm.wav +146.520000MHz--2026-08-21_12_18_38-nfm_transcription.txt +146.520000MHz--2026-08-21_12_18_38-nfm.json +``` + +Only voice is transcribed — running a recogniser over Morse or a data burst +costs seconds and produces nothing. CW is decoded separately and appears in +the metadata as text already. + +**A capture with nothing recognisable in it produces no file.** Music, a +carrier with an open mic, a fragment too short to make out: nothing is +written, rather than a directory of placeholders. The transcript is also +copied into the capture's `.json`, which names it only once it exists — so the +metadata never points at a file that was never created. The scan reports the +tally at the end: + +``` + 2 transcript(s) written, 1 with no recognisable speech +``` + +Recognition takes seconds per capture, far longer than a capture itself, so it +runs on its own thread and the scan never waits for it; anything still queued +is finished when the scan stops. When recordings are being combined by +frequency there is one transcript per frequency too, each line stamped with +the time: + +``` +[2026-08-21 12:18:38] this is what the first transmission said +[2026-08-21 12:24:02] and this is the second +``` + +### Getting a recogniser + +Unlike the spoken announcements, this needs an installed engine — recognition +depends on a trained model, so there is no built-in fallback. + +```bash +bandsaunter transcribe --engines # what is installed +pip install faster-whisper # the recommended one +``` + +| Engine | Notes | +|---|---| +| `faster-whisper` | best on radio audio; ~120 MB of dependencies, model downloads on first use | +| `whisper` | the original; heavier | +| `whisper-cli` | whisper.cpp, no Python dependencies | +| `vosk` | ~10 MB plus a 40 MB model, fully offline, but weaker on noisy audio | +| `pocketsphinx` | tiny; poor on radio audio | + +The difference is easy to measure. Both engines on the same 26-second +off-air recording: + +``` +faster-whisper 3.4s "August 21, 2026, at 13.42 and 28 seconds, + 96.108 megahertz, 13.42 and 47 seconds, ..." +vosk 11.2s "august twenty one twenty twenty six at thirteen + forty two i'm twenty eight seconds ... forty + family factories ..." +``` + +Whisper is both more accurate and three times faster, and it punctuates. +Vosk's advantage is size and that it needs nothing after its model is +downloaded once. + +`--transcribe-model` selects the size (`tiny.en`, `base.en`, `small.en`, +`medium.en`) and `--transcribe-language` fixes the language — worth setting, +since on a short noisy clip automatic detection often guesses wrong and +returns nonsense in another language. + +Existing recordings can be transcribed after the fact: + +```bash +bandsaunter transcribe recordings/ # every WAV in a directory +bandsaunter transcribe one.wav --stdout +``` + +Re-examine anything later: + +```bash +bandsaunter analyze recordings/2026-08-19/.../iq.cf32 # identify +bandsaunter analyze recordings/2026-08-19/.../audio.wav # decode CW +``` + +## Live controls + +The display sizes itself to the terminal, giving up the spectrum row, then the +hit list, then the key hints as space runs short. A frame taller than the +terminal cannot be redrawn in place, so an oversized one would leave a copy of +itself behind on every refresh. + +For the same reason the driver's own messages are suppressed while a scan +runs: librtlsdr writes them straight to file descriptor 2 from C — including +`Allocating 15 zero-copy buffers` on *every* capture — and they draw over the +display and break its cursor tracking. `bandsaunter devices` still shows them, +since that is the command to run when something is wrong, and +`BANDSAUNTER_DRIVER_MESSAGES=1` restores them everywhere. + + +| Key | Action | +|---|---| +| `q` | stop | +| `p` | pause / resume | +| `s` | skip this signal, resume sweeping | +| `l` | lock out this frequency for the rest of the run | +| `+` / `-` | adjust the squelch threshold | + +## Built-in help + +Press `h` in the menus for topics covering setup, how the sweep works, why +nothing (or too much) is being recorded, capturing conversations, where files +go, HF reception and the keys available during a scan. Typing a setting name +there explains that setting instead. + +From the command line, `bandsaunter config --describe ` does the same, +and `bandsaunter scan --help` lists every flag grouped the same way as the menus. + +## HF + +Frequencies below 24 MHz need direct sampling, which most RTL-SDR dongles +support on the Q branch. It is selected automatically: + +```bash +bandsaunter scan -b 40m-cw --record 60 # 40 m CW, decoded to text +bandsaunter scan -b am-broadcast +``` + +You will need an HF antenna; the tuner is bypassed in this mode, so there is +no front-end filtering or gain. + +## How the sweep works + +- The band is covered in steps of `sample_rate x usable_fraction / 2`. The + local oscillator is parked *below* the span each step covers, so the + RTL2832's DC spike never lands inside the frequencies being searched. +- The noise floor is measured per FFT bin as a sliding low percentile, which + follows the receiver's passband shape and steps over signals. There is no + warm-up period, and a station that transmits constantly does not learn + itself into the floor. +- The sweep uses peak-hold rather than averaging across each dwell, so bursty + traffic — CW, packet, a short over — is not averaged into the noise. +- On a hit, the receiver retunes with a quarter-rate LO offset (moving the DC + spike off the signal), probes once at ~60 Hz resolution to measure the real + occupied bandwidth, and picks the demodulator from that plus the band plan. + +## Without hardware + +`--simulate` swaps in a synthetic receiver carrying one of each interesting +signal type, which is also what the test suite runs against: + +```bash +bandsaunter scan -r 144M-148M --simulate +``` + +## Testing + +```bash +python -m pytest +``` + +Covers DSP invariants, frequency parsing, the classifier against synthetic +signals at several SNRs and random seeds, Morse decoding from 8 to 40 WPM, the +voice detector against synthetic speech and against noise, tones and hum, and +full scan runs through the simulator checking that `--record` and `--hang` are +obeyed, that static and bare carriers are never written to disk, and that +audio, IQ and metadata are correct. + +The simulator's voice transmitters carry synthesised speech -- glottal pulses +through moving formants, compressed the way a real transmitter compresses, +then gated into syllables and phrases -- because sine tones would not exercise +the speech detector at all. + +## Legal note + +Receiving is not the same as being allowed to use or divulge what you hear. In +the US, the ECPA prohibits intercepting cellular and other private +communications, and rebroadcasting or acting on what you receive is separately +restricted. Check your local rules. diff --git a/bandsaunter/__init__.py b/bandsaunter/__init__.py new file mode 100755 index 0000000..92a998f --- /dev/null +++ b/bandsaunter/__init__.py @@ -0,0 +1,8 @@ +"""bandsaunter -- a signal scanner for RTL-SDR receivers. + +Sweeps arbitrary frequency ranges (or US band-plan presets), records what it +finds, and identifies the modulation -- including decoding CW/Morse to text. +""" + +__version__ = "1.0.0" +__all__ = ["__version__"] diff --git a/bandsaunter/__main__.py b/bandsaunter/__main__.py new file mode 100755 index 0000000..dbdd066 --- /dev/null +++ b/bandsaunter/__main__.py @@ -0,0 +1,6 @@ +import sys + +from .cli import main + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bandsaunter/_quiet.py b/bandsaunter/_quiet.py new file mode 100644 index 0000000..cd034a5 --- /dev/null +++ b/bandsaunter/_quiet.py @@ -0,0 +1,48 @@ +"""Silencing output that comes from underneath Python. + +Libraries write progress bars and status lines straight to file descriptor 2, +where redirecting :data:`sys.stderr` cannot reach them. During a scan those +land in the middle of the live display and break its cursor tracking, so the +header ends up drawn several times over. +""" + +from __future__ import annotations + +import contextlib +import os +import sys + +__all__ = ["suppress_stderr"] + + +@contextlib.contextmanager +def suppress_stderr(enabled: bool = True): + """Send file descriptor 2 to nowhere for the duration of a call. + + Kept to the narrowest possible window so a Python traceback is never + swallowed by it. + """ + if not enabled: + yield + return + saved = devnull = None + try: + sys.stderr.flush() + saved = os.dup(2) + devnull = os.open(os.devnull, os.O_WRONLY) + os.dup2(devnull, 2) + except OSError: + if saved is not None: + os.close(saved) + if devnull is not None: + os.close(devnull) + yield + return + try: + yield + finally: + try: + os.dup2(saved, 2) + finally: + os.close(saved) + os.close(devnull) diff --git a/bandsaunter/announce.py b/bandsaunter/announce.py new file mode 100644 index 0000000..d1cb5d9 --- /dev/null +++ b/bandsaunter/announce.py @@ -0,0 +1,436 @@ +"""Spoken announcements, for labelling recordings with the time. + +Uses an installed text-to-speech program when there is one. When there is +not -- which is the common case on a headless box -- it falls back to a +built-in formant synthesiser. The vocabulary an announcement needs is small +and fixed (numbers, month names, a few words), so synthesising it directly is +practical and keeps the feature working with nothing else installed. +""" + +from __future__ import annotations + +import math +import shutil +import subprocess +import tempfile +import wave +from dataclasses import dataclass +from datetime import datetime +from pathlib import Path + +import numpy as np +from scipy import signal as sps + +__all__ = ["say", "speak_timestamp", "timestamp_phrase", "timestamp_text", + "number_words", "available_engine", "ENGINES"] + + +# --------------------------------------------------------------------------- +# Phonemes. Formant targets in Hz, roughly the Peterson-Barney vowel centres +# with consonant loci from the standard synthesis-by-rule tables. +# --------------------------------------------------------------------------- + +@dataclass(frozen=True) +class Phone: + kind: str # vowel, nasal, fric, stop, approx, silence + f1: float = 500.0 + f2: float = 1500.0 + f3: float = 2500.0 + duration: float = 0.09 # seconds at normal speed + voiced: bool = True + noise: float = 0.0 # fricative noise level + amp: float = 1.0 + # Diphthongs and glides move towards a second target. + to: tuple[float, float, float] | None = None + + +P = Phone +PHONEMES: dict[str, Phone] = { + # -- vowels ---------------------------------------------------------- + "iy": P("vowel", 300, 2300, 3000, 0.11), + "ih": P("vowel", 400, 2000, 2550, 0.075), + "eh": P("vowel", 550, 1850, 2500, 0.09), + "ae": P("vowel", 660, 1720, 2410, 0.11), + "aa": P("vowel", 730, 1090, 2440, 0.11), + "ao": P("vowel", 570, 840, 2410, 0.10), + "uh": P("vowel", 440, 1020, 2240, 0.075), + "uw": P("vowel", 320, 900, 2200, 0.11), + "ah": P("vowel", 640, 1190, 2390, 0.075), + "er": P("vowel", 490, 1350, 1690, 0.10), + # -- diphthongs ------------------------------------------------------- + "ay": P("vowel", 730, 1090, 2440, 0.16, to=(400, 2000, 2550)), + "ey": P("vowel", 550, 1850, 2500, 0.15, to=(300, 2300, 3000)), + "ow": P("vowel", 570, 840, 2410, 0.15, to=(320, 900, 2200)), + "aw": P("vowel", 730, 1090, 2440, 0.16, to=(440, 1020, 2240)), + "oy": P("vowel", 570, 840, 2410, 0.16, to=(400, 2000, 2550)), + # -- nasals ------------------------------------------------------------ + "m": P("nasal", 300, 1100, 2200, 0.07, amp=0.55), + "n": P("nasal", 300, 1600, 2600, 0.07, amp=0.55), + "ng": P("nasal", 300, 2000, 2600, 0.07, amp=0.55), + # -- approximants ------------------------------------------------------ + "l": P("approx", 380, 1100, 2600, 0.07, amp=0.75), + "r": P("approx", 400, 1100, 1600, 0.07, amp=0.75), + "w": P("approx", 320, 800, 2200, 0.06, amp=0.7, to=(500, 1500, 2500)), + "y": P("approx", 300, 2300, 3000, 0.06, amp=0.7, to=(500, 1500, 2500)), + # -- fricatives -------------------------------------------------------- + "s": P("fric", 500, 4500, 6500, 0.11, voiced=False, noise=1.0, amp=0.5), + "z": P("fric", 400, 4200, 6000, 0.09, voiced=True, noise=0.7, amp=0.45), + "f": P("fric", 500, 1800, 4500, 0.10, voiced=False, noise=0.7, amp=0.32), + "v": P("fric", 400, 1600, 3000, 0.07, voiced=True, noise=0.4, amp=0.35), + "th": P("fric", 500, 1700, 4000, 0.09, voiced=False, noise=0.6, amp=0.28), + "dh": P("fric", 400, 1500, 2800, 0.06, voiced=True, noise=0.35, amp=0.35), + "sh": P("fric", 500, 2400, 4000, 0.12, voiced=False, noise=1.0, amp=0.5), + "h": P("fric", 500, 1500, 2500, 0.06, voiced=False, noise=0.5, amp=0.25), + # -- stops (closure, then burst, then the transition into the vowel) --- + "p": P("stop", 400, 1100, 2200, 0.085, voiced=False, noise=0.8, amp=0.4), + "t": P("stop", 400, 1800, 2600, 0.085, voiced=False, noise=0.9, amp=0.45), + "k": P("stop", 400, 1900, 2400, 0.085, voiced=False, noise=0.85, amp=0.42), + "b": P("stop", 350, 1100, 2200, 0.07, voiced=True, noise=0.35, amp=0.4), + "d": P("stop", 350, 1800, 2600, 0.07, voiced=True, noise=0.4, amp=0.4), + "g": P("stop", 350, 1900, 2400, 0.07, voiced=True, noise=0.4, amp=0.4), + # -- affricates --------------------------------------------------------- + "ch": P("stop", 500, 2400, 4000, 0.13, voiced=False, noise=1.0, amp=0.45), + "jh": P("stop", 450, 2200, 3200, 0.11, voiced=True, noise=0.6, amp=0.45), + # -- silence ------------------------------------------------------------- + "_": P("silence", duration=0.09, voiced=False, amp=0.0), + "__": P("silence", duration=0.22, voiced=False, amp=0.0), +} + + +# --------------------------------------------------------------------------- +# Vocabulary +# --------------------------------------------------------------------------- + +WORDS: dict[str, str] = { + "zero": "z iy r ow", "one": "w ah n", "two": "t uw", "three": "th r iy", + "four": "f ao r", "five": "f ay v", "six": "s ih k s", + "seven": "s eh v ah n", "eight": "ey t", "nine": "n ay n", + "ten": "t eh n", "eleven": "ih l eh v ah n", "twelve": "t w eh l v", + "thirteen": "th er t iy n", "fourteen": "f ao r t iy n", + "fifteen": "f ih f t iy n", "sixteen": "s ih k s t iy n", + "seventeen": "s eh v ah n t iy n", "eighteen": "ey t iy n", + "nineteen": "n ay n t iy n", "twenty": "t w eh n t iy", + "thirty": "th er t iy", "forty": "f ao r t iy", "fifty": "f ih f t iy", + "sixty": "s ih k s t iy", "seventy": "s eh v ah n t iy", + "eighty": "ey t iy", "ninety": "n ay n t iy", + "hundred": "h ah n d r ah d", "thousand": "th aw z ah n d", + "oh": "ow", "point": "p oy n t", + "january": "jh ae n y uw eh r iy", "february": "f eh b y uw eh r iy", + "march": "m aa r ch", "april": "ey p r ih l", "may": "m ey", + "june": "jh uw n", "july": "jh uw l ay", "august": "ao g ah s t", + "september": "s eh p t eh m b er", "october": "aa k t ow b er", + "november": "n ow v eh m b er", "december": "d ih s eh m b er", + "a.m.": "ey eh m", "p.m.": "p iy eh m", + "megahertz": "m eh g ah h er t s", "kilohertz": "k ih l ow h er t s", + "hours": "aw er z", "at": "ae t", "on": "ao n", +} + +_ONES = ("zero", "one", "two", "three", "four", "five", "six", "seven", + "eight", "nine") +_TEENS = ("ten", "eleven", "twelve", "thirteen", "fourteen", "fifteen", + "sixteen", "seventeen", "eighteen", "nineteen") +_TENS = ("", "", "twenty", "thirty", "forty", "fifty", "sixty", "seventy", + "eighty", "ninety") +MONTHS = ("january", "february", "march", "april", "may", "june", "july", + "august", "september", "october", "november", "december") + + +def number_words(n: int) -> list[str]: + """Spell a whole number from 0 to 9999 the way it is normally said.""" + n = int(n) + if n < 0: + return ["minus"] + number_words(-n) + if n < 10: + return [_ONES[n]] + if n < 20: + return [_TEENS[n - 10]] + if n < 100: + tens, ones = divmod(n, 10) + return [_TENS[tens]] + ([_ONES[ones]] if ones else []) + if n < 1000: + hundreds, rest = divmod(n, 100) + out = [_ONES[hundreds], "hundred"] + return out + (number_words(rest) if rest else []) + thousands, rest = divmod(n, 1000) + out = number_words(thousands) + ["thousand"] + return out + (number_words(rest) if rest else []) + + +def _year_words(year: int) -> list[str]: + """Years are said in pairs: 2026 is "twenty twenty six", not "two thousand".""" + if 2000 <= year <= 2099: + rest = year - 2000 + if rest == 0: + return ["two", "thousand"] + if rest < 10: + return ["two", "thousand", _ONES[rest]] + return ["twenty"] + number_words(rest) + return number_words(year) + + +def _two_digit_clock(value: int) -> list[str]: + """Clock fields: 05 is "oh five", 30 is "thirty".""" + if value == 0: + return ["oh", "oh"] + if value < 10: + return ["oh", _ONES[value]] + return number_words(value) + + +def timestamp_phrase(when: datetime, frequency: float | None = None, + with_date: bool = True, with_seconds: bool = True) -> str: + """The words to speak for one timestamp.""" + words: list[str] = [] + if with_date: + words += [MONTHS[when.month - 1]] + number_words(when.day) + words += ["_"] + _year_words(when.year) + ["_"] + words += number_words(when.hour) if when.hour >= 10 else \ + _two_digit_clock(when.hour) + words += _two_digit_clock(when.minute) + if with_seconds: + words += ["_"] + _two_digit_clock(when.second) + if frequency: + words += ["__"] + _frequency_words(frequency) + return " ".join(words) + + +def timestamp_text(when: datetime, frequency: float | None = None, + with_date: bool = True, with_seconds: bool = True) -> str: + """The same timestamp as ordinary text, for an installed engine. + + Written the way those engines read best rather than as the word list the + built-in synthesiser needs. Punctuation is what gives them their phrasing, + and the obvious spellings are traps: espeak reads "14:38:05" as "fourteen + thirty, eight zero five", and an ISO date as "two thousand and twenty six + dash zero eight dash twenty one". + """ + parts: list[str] = [] + if with_date: + parts.append(f"{MONTHS[when.month - 1].capitalize()} {when.day}") + parts.append(" ".join(_year_words(when.year))) + clock = f"{when.hour:02d} {when.minute:02d}" + if with_seconds: + clock += f" and {when.second:02d} seconds" + parts.append(f"at {clock}" if with_date else clock) + if frequency: + parts.append(f"{frequency / 1e6:.3f} megahertz") + return ", ".join(parts) + + +def _frequency_words(hz: float) -> list[str]: + mhz = hz / 1e6 + whole = int(mhz) + frac = round((mhz - whole) * 1000) # kHz, three digits + out = number_words(whole) + ["point"] + for digit in f"{frac:03d}": + out.append(_ONES[int(digit)]) + return out + ["megahertz"] + + +# --------------------------------------------------------------------------- +# Synthesis +# --------------------------------------------------------------------------- + +def _resonator(freq: np.ndarray, bandwidth: float, fs: float, + x: np.ndarray, block: int = 64) -> np.ndarray: + """Two-pole resonator whose centre frequency moves as the phrase does. + + Processed in short blocks with the coefficients held constant across each + one. Articulation moves far more slowly than 4 ms, so this is + indistinguishable from recomputing every sample, and it runs through a + compiled filter instead of a Python loop. + """ + r = math.exp(-math.pi * bandwidth / fs) + out = np.zeros(x.size) + zi = np.zeros(2) + for start in range(0, x.size, block): + stop = min(start + block, x.size) + centre = float(np.mean(freq[start:stop])) + theta = 2.0 * math.pi * centre / fs + b1 = 2.0 * r * math.cos(theta) + b2 = -(r * r) + gain = 1.0 - b1 - b2 + chunk, zi = sps.lfilter([gain], [1.0, -b1, -b2], x[start:stop], zi=zi) + out[start:stop] = chunk + return out + + +def _phones_of(text: str) -> list[tuple[str, Phone]]: + """Turn a phrase into phones, looking each word up in the vocabulary.""" + out: list[tuple[str, Phone]] = [] + for word in text.split(): + if word in ("_", "__"): + out.append((word, PHONEMES[word])) + continue + spelling = WORDS.get(word.lower()) + if spelling is None: + continue + for name in spelling.split(): + phone = PHONEMES.get(name) + if phone is not None: + out.append((name, phone)) + out.append(("_", PHONEMES["_"])) + return out + + +def synthesize(text: str, fs: float = 16000.0, pitch: float = 115.0, + rate: float = 1.0) -> np.ndarray: + """Render a phrase with the built-in formant synthesiser.""" + phones = _phones_of(text) + if not phones: + return np.zeros(0, dtype=np.float32) + + # Lay out the formant tracks and the source description sample by sample, + # so that transitions between phones are continuous rather than stepped. + tracks_f, tracks_a, voiced, noise = [], [], [], [] + for name, ph in phones: + n = max(1, int(ph.duration / max(rate, 0.1) * fs)) + start = np.array([ph.f1, ph.f2, ph.f3]) + end = np.array(ph.to) if ph.to else start + ramp = np.linspace(0.0, 1.0, n)[:, None] + tracks_f.append(start[None, :] * (1 - ramp) + end[None, :] * ramp) + + env = np.full(n, ph.amp) + if ph.kind == "stop": + # Closure, then a burst: silence for the first half, then a short + # noisy release. Without the silence a stop is just a fricative. + hold = int(n * 0.55) + env[:hold] = 0.0 + env[hold:] = np.linspace(ph.amp, ph.amp * 0.35, n - hold) + elif ph.kind == "silence": + env[:] = 0.0 + else: + edge = max(1, int(0.012 * fs)) + if n > 2 * edge: + env[:edge] *= np.linspace(0, 1, edge) + env[-edge:] *= np.linspace(1, 0, edge) + tracks_a.append(env) + voiced.append(np.full(n, 1.0 if ph.voiced else 0.0)) + noise.append(np.full(n, ph.noise)) + + f_track = np.vstack(tracks_f) + amp = np.concatenate(tracks_a) + voiced = np.concatenate(voiced) + noise = np.concatenate(noise) + total = amp.size + + # Smooth the tracks: articulation is continuous, and abrupt formant jumps + # are heard as clicks. + smooth = max(3, int(0.02 * fs)) + kernel = np.ones(smooth) / smooth + for i in range(3): + f_track[:, i] = np.convolve(f_track[:, i], kernel, mode="same") + voiced = np.convolve(voiced, kernel, mode="same") + + # Source: a glottal pulse train with a falling pitch, plus noise. + t = np.arange(total) / fs + f0 = pitch * (1.0 - 0.18 * t / max(t[-1], 1e-6)) * \ + (1.0 + 0.03 * np.sin(2 * np.pi * 3.1 * t)) + phase = np.cumsum(2.0 * np.pi * f0 / fs) + pulses = np.zeros(total) + edges = np.flatnonzero(np.diff(np.floor(phase / (2 * np.pi))) > 0) + pulses[edges] = 1.0 + pulses -= pulses.mean() + rng = np.random.default_rng(12345) + hiss = rng.standard_normal(total) + + # A cascade, not three resonators added together. Summing them in + # parallel with fixed weights loses the first formant: each resonator is + # normalised at DC, which flatters a high formant far more than a low one, + # so F1 ends up buried. In cascade the product of the three responses is + # the vocal-tract envelope, and the relative levels come out on their own. + voiced_src = sps.lfilter([1.0], [1.0, -0.94], voiced * pulses) + voiced_src /= max(float(np.abs(voiced_src).max()), 1e-9) + src = voiced_src + noise * hiss * 0.30 + + out = _resonator(f_track[:, 0], 80.0, fs, src) + out = _resonator(f_track[:, 1], 110.0, fs, out) + out = _resonator(f_track[:, 2], 160.0, fs, out) + out *= amp + peak = float(np.abs(out).max()) + if peak > 0: + out = out / peak * 0.7 + return out.astype(np.float32) + + +# --------------------------------------------------------------------------- +# External engines +# --------------------------------------------------------------------------- + +ENGINES = ("espeak-ng", "espeak", "pico2wave", "flite", "say") + + +def available_engine() -> str | None: + """The first installed text-to-speech program, if any.""" + for name in ENGINES: + if shutil.which(name): + return name + return None + + +def _external(text: str, fs: float, engine: str) -> np.ndarray | None: + """Render with an installed engine, returning None if it will not play.""" + words = " ".join(w for w in text.split() if w not in ("_", "__")) + with tempfile.TemporaryDirectory() as tmp: + out = Path(tmp) / "say.wav" + if engine in ("espeak-ng", "espeak"): + cmd = [engine, "-w", str(out), "-s", "150", words] + elif engine == "pico2wave": + cmd = [engine, "-w", str(out), words] + elif engine == "flite": + cmd = [engine, "-t", words, "-o", str(out)] + elif engine == "say": + cmd = [engine, "-o", str(out), "--data-format=LEI16@16000", words] + else: + return None + try: + subprocess.run(cmd, check=True, capture_output=True, timeout=20) + with wave.open(str(out)) as w: + raw = w.readframes(w.getnframes()) + rate = w.getframerate() + channels = w.getnchannels() + except (OSError, subprocess.SubprocessError, wave.Error): + return None + audio = np.frombuffer(raw, dtype=" 1: + audio = audio.reshape(-1, channels).mean(axis=1) + if rate != fs and audio.size: + from scipy.signal import resample_poly + g = math.gcd(int(rate), int(fs)) + audio = resample_poly(audio, int(fs) // g, int(rate) // g) + # Match the built-in synthesiser's level, so switching engines does not + # change how loud the announcements are against the recordings. + peak = float(np.abs(audio).max()) if audio.size else 0.0 + if peak > 0: + audio = audio / peak * 0.7 + return audio.astype(np.float32) + + +def say(text: str, fs: float = 16000.0, engine: str = "auto") -> np.ndarray: + """Speak a phrase, preferring an installed engine over the built-in one.""" + if engine != "builtin": + chosen = available_engine() if engine == "auto" else engine + if chosen: + audio = _external(text, fs, chosen) + if audio is not None and audio.size: + return audio + return synthesize(text, fs) + + +def speak_timestamp(when: datetime, fs: float = 16000.0, + frequency: float | None = None, engine: str = "auto", + with_date: bool = True) -> np.ndarray: + """Audio saying the date and time, for splicing before a recording. + + The wording differs by engine: an installed one is given ordinary text + with punctuation, while the built-in synthesiser is given the word list it + has pronunciations for. + """ + chosen = None if engine == "builtin" else ( + available_engine() if engine == "auto" else engine) + if chosen: + audio = _external(timestamp_text(when, frequency, with_date=with_date), + fs, chosen) + if audio is not None and audio.size: + return audio + return synthesize(timestamp_phrase(when, frequency, with_date=with_date), fs) diff --git a/bandsaunter/bandplan.py b/bandsaunter/bandplan.py new file mode 100755 index 0000000..92573e7 --- /dev/null +++ b/bandsaunter/bandplan.py @@ -0,0 +1,537 @@ +"""US band plan presets. + +Each preset is a tunable span plus enough metadata for the scanner to pick a +sensible demodulator, channel spacing and bandwidth without being told. +Frequencies are in Hz. Coverage follows the FCC allocations and the common +NA channel plans (ITU Region 2). +""" + +from __future__ import annotations + +from dataclasses import dataclass, field + +__all__ = ["BandPreset", "PRESETS", "CATEGORIES", "by_key", "search", + "in_category", "presets_covering", "expand_preset"] + + +@dataclass(frozen=True) +class BandPreset: + key: str + name: str + category: str + start: float + stop: float + step: float = 12_500.0 # channel spacing / scan resolution + mode: str = "nfm" # demodulator hint: nfm wfm am usb lsb cw raw + bandwidth: float = 12_500.0 # nominal signal bandwidth + note: str = "" + tags: tuple[str, ...] = field(default_factory=tuple) + # A preset may instead stand for a set of others, so that scattered + # segments -- every CW allocation, say -- can be picked in one go. + members: tuple[str, ...] = field(default_factory=tuple) + + @property + def is_group(self) -> bool: + return bool(self.members) + + def expand(self) -> list["BandPreset"]: + """The presets this one actually scans: itself, or its members.""" + if not self.members: + return [self] + out = [] + for key in self.members: + member = _BY_KEY.get(key) + if member is not None: + out.extend(member.expand()) + return out + + @property + def span(self) -> float: + return self.stop - self.start + + @property + def needs_direct_sampling(self) -> bool: + return self.stop < 24_000_000 + + def describe(self) -> str: + return f"{self.name} {fmt_hz(self.start)}-{fmt_hz(self.stop)} [{self.mode}]" + + +def fmt_hz(hz: float) -> str: + """Render a frequency the way a radio operator would write it.""" + hz = float(hz) + if hz >= 1e9: + return f"{hz/1e9:.6f}".rstrip("0").rstrip(".") + " GHz" + if hz >= 1e6: + s = f"{hz/1e6:.6f}".rstrip("0").rstrip(".") + return f"{s} MHz" + if hz >= 1e3: + s = f"{hz/1e3:.4f}".rstrip("0").rstrip(".") + return f"{s} kHz" + return f"{hz:.0f} Hz" + + +_P = BandPreset + +PRESETS: tuple[BandPreset, ...] = ( + + # ------------------------------------------------------------------ + # HF -- requires direct sampling (Q branch) on an RTL2832 dongle. + # ------------------------------------------------------------------ + _P("am-broadcast", "AM Broadcast Band", "HF / Shortwave", + 530_000, 1_710_000, 10_000, "am", 10_000, + "US MW broadcast, 10 kHz channel spacing", ("broadcast", "hf")), + _P("160m", "160 m Amateur (Top Band)", "Amateur Radio", + 1_800_000, 2_000_000, 500, "lsb", 2_800, + "CW/digital low, phone above 1.843", ("ham", "hf")), + _P("160m-cw", "160 m CW", "Amateur Radio", + 1_800_000, 1_840_000, 200, "cw", 500, + "CW and digital segment", ("ham", "hf", "cw")), + _P("120m-swbc", "120 m Shortwave Broadcast", "HF / Shortwave", + 2_300_000, 2_495_000, 5_000, "am", 9_000, "Tropical band", ("swl", "hf")), + _P("marine-hf-2mhz", "Marine HF 2 MHz", "Marine", + 2_000_000, 2_850_000, 1_000, "usb", 2_800, + "Includes 2182 kHz distress", ("marine", "hf")), + _P("wwv", "WWV / WWVH / CHU Time Signals", "Utility / Time", + 2_490_000, 20_010_000, 1_000, "am", 6_000, + "2.5/5/10/15/20 MHz WWV, 3.33/7.85/14.67 CHU", ("utility", "hf")), + _P("90m-swbc", "90 m Shortwave Broadcast", "HF / Shortwave", + 3_200_000, 3_400_000, 5_000, "am", 9_000, "Tropical band", ("swl", "hf")), + _P("80m", "80/75 m Amateur", "Amateur Radio", + 3_500_000, 4_000_000, 500, "lsb", 2_800, + "CW 3.500-3.600, phone 3.800-4.000", ("ham", "hf")), + _P("80m-cw", "80 m CW / Digital", "Amateur Radio", + 3_500_000, 3_600_000, 200, "cw", 500, + "CW segment, FT8 at 3.573", ("ham", "hf", "cw")), + _P("75m-swbc", "75 m Shortwave Broadcast", "HF / Shortwave", + 3_900_000, 4_000_000, 5_000, "am", 9_000, "", ("swl", "hf")), + _P("60m", "60 m Amateur (channelised)", "Amateur Radio", + 5_330_500, 5_405_000, 100, "usb", 2_800, + "Five fixed US channels", ("ham", "hf")), + _P("49m-swbc", "49 m Shortwave Broadcast", "HF / Shortwave", + 5_900_000, 6_200_000, 5_000, "am", 9_000, "", ("swl", "hf")), + _P("aero-hf", "HF Aeronautical (oceanic)", "Aviation", + 5_450_000, 6_700_000, 1_000, "usb", 2_800, + "MWARA / VOLMET / SELCAL", ("aviation", "hf")), + _P("40m", "40 m Amateur", "Amateur Radio", + 7_000_000, 7_300_000, 500, "lsb", 2_800, + "CW 7.000-7.125, phone 7.125-7.300", ("ham", "hf")), + _P("40m-cw", "40 m CW / Digital", "Amateur Radio", + 7_000_000, 7_125_000, 200, "cw", 500, + "CW segment, FT8 at 7.074", ("ham", "hf", "cw")), + _P("41m-swbc", "41 m Shortwave Broadcast", "HF / Shortwave", + 7_200_000, 7_600_000, 5_000, "am", 9_000, "", ("swl", "hf")), + _P("31m-swbc", "31 m Shortwave Broadcast", "HF / Shortwave", + 9_400_000, 9_900_000, 5_000, "am", 9_000, "", ("swl", "hf")), + _P("30m", "30 m Amateur (CW/digital only)", "Amateur Radio", + 10_100_000, 10_150_000, 200, "cw", 500, + "No phone; FT8 at 10.136", ("ham", "hf", "cw")), + _P("25m-swbc", "25 m Shortwave Broadcast", "HF / Shortwave", + 11_600_000, 12_100_000, 5_000, "am", 9_000, "", ("swl", "hf")), + _P("22m-swbc", "22 m Shortwave Broadcast", "HF / Shortwave", + 13_570_000, 13_870_000, 5_000, "am", 9_000, "", ("swl", "hf")), + _P("20m", "20 m Amateur", "Amateur Radio", + 14_000_000, 14_350_000, 500, "usb", 2_800, + "CW 14.000-14.150, phone above", ("ham", "hf")), + _P("20m-cw", "20 m CW / Digital", "Amateur Radio", + 14_000_000, 14_150_000, 200, "cw", 500, + "CW segment, FT8 at 14.074", ("ham", "hf", "cw")), + _P("19m-swbc", "19 m Shortwave Broadcast", "HF / Shortwave", + 15_100_000, 15_830_000, 5_000, "am", 9_000, "", ("swl", "hf")), + _P("17m", "17 m Amateur", "Amateur Radio", + 18_068_000, 18_168_000, 500, "usb", 2_800, "", ("ham", "hf")), + _P("17m-cw", "17 m CW / Digital", "Amateur Radio", + 18_068_000, 18_110_000, 200, "cw", 500, + "CW segment, FT8 at 18.100", ("ham", "hf", "cw")), + _P("16m-swbc", "16 m Shortwave Broadcast", "HF / Shortwave", + 17_480_000, 17_900_000, 5_000, "am", 9_000, "", ("swl", "hf")), + _P("15m", "15 m Amateur", "Amateur Radio", + 21_000_000, 21_450_000, 500, "usb", 2_800, "", ("ham", "hf")), + _P("15m-cw", "15 m CW / Digital", "Amateur Radio", + 21_000_000, 21_200_000, 200, "cw", 500, + "CW segment, FT8 at 21.074", ("ham", "hf", "cw")), + _P("13m-swbc", "13 m Shortwave Broadcast", "HF / Shortwave", + 21_450_000, 21_850_000, 5_000, "am", 9_000, "", ("swl", "hf")), + _P("12m", "12 m Amateur", "Amateur Radio", + 24_890_000, 24_990_000, 500, "usb", 2_800, "", ("ham", "hf")), + _P("12m-cw", "12 m CW / Digital", "Amateur Radio", + 24_890_000, 24_930_000, 200, "cw", 500, + "CW segment, FT8 at 24.915", ("ham", "hf", "cw")), + _P("cb", "Citizens Band (CB) 11 m", "Land Mobile", + 26_965_000, 27_405_000, 10_000, "am", 8_000, + "40 channels; ch 19 = 27.185, SSB above ch 35", ("cb",)), + _P("cb-ssb", "CB SSB / Freeband", "Land Mobile", + 27_235_000, 27_995_000, 5_000, "usb", 2_800, "", ("cb",)), + _P("10m", "10 m Amateur", "Amateur Radio", + 28_000_000, 29_700_000, 1_000, "usb", 2_800, + "CW/data low, FM repeaters 29.5-29.7", ("ham", "hf")), + _P("10m-cw", "10 m CW / Digital", "Amateur Radio", + 28_000_000, 28_300_000, 200, "cw", 500, + "CW segment, FT8 at 28.074, beacons 28.190-28.300", ("ham", "hf", "cw")), + _P("10m-fm", "10 m FM Simplex / Repeaters", "Amateur Radio", + 29_500_000, 29_700_000, 20_000, "nfm", 16_000, "", ("ham",)), + + # ------------------------------------------------------------------ + # VHF low band + # ------------------------------------------------------------------ + _P("vhf-low-public-safety", "VHF Low Band Public Safety", "Public Safety", + 30_000_000, 50_000_000, 5_000, "nfm", 12_500, + "State police, forestry, highway crews", ("publicsafety",)), + _P("mil-lowband", "Military / SINCGARS Low Band", "Military", + 30_000_000, 88_000_000, 25_000, "nfm", 25_000, + "Tactical FM, mostly frequency hopping", ("military",)), + _P("6m", "6 m Amateur", "Amateur Radio", + 50_000_000, 54_000_000, 20_000, "nfm", 16_000, + "CW/SSB 50.0-50.3, FM 52-54", ("ham",)), + _P("6m-ssb", "6 m Weak Signal (CW/SSB)", "Amateur Radio", + 50_000_000, 50_300_000, 500, "usb", 2_800, + "50.125 SSB calling, FT8 at 50.313", ("ham", "cw")), + _P("6m-cw", "6 m CW", "Amateur Radio", + 50_000_000, 50_100_000, 200, "cw", 500, + "CW segment, beacons 50.060-50.080", ("ham", "cw")), + _P("rc-radio-control", "R/C Radio Control 72 MHz", "Telemetry / Control", + 72_000_000, 76_000_000, 20_000, "nfm", 10_000, + "Model aircraft channels 11-60", ("control",)), + _P("tv-low-vhf", "VHF TV Channels 2-6", "Broadcast", + 54_000_000, 88_000_000, 6_000_000, "raw", 6_000_000, + "ATSC 8VSB, mostly vacated", ("broadcast",)), + + # ------------------------------------------------------------------ + # VHF high band + # ------------------------------------------------------------------ + _P("fm-broadcast", "FM Broadcast Band", "Broadcast", + 88_100_000, 107_900_000, 200_000, "wfm", 180_000, + "US odd-tenth channel centres", ("broadcast",)), + _P("aircraft-vhf", "VHF Airband (voice)", "Aviation", + 118_000_000, 137_000_000, 25_000, "am", 12_500, + "Tower/approach/centre, 8.33 kHz not used in US", ("aviation",)), + _P("aircraft-vhf-tower", "VHF Airband - Tower & Ground", "Aviation", + 118_000_000, 122_000_000, 25_000, "am", 12_500, "", ("aviation",)), + _P("aircraft-vhf-unicom", "VHF Airband - UNICOM / CTAF", "Aviation", + 122_000_000, 123_600_000, 25_000, "am", 12_500, + "122.700-122.975 and 123.000 CTAF", ("aviation",)), + _P("acars", "ACARS Datalink", "Aviation", + 129_125_000, 136_975_000, 25_000, "am", 12_500, + "Primary US ACARS on 131.550", ("aviation", "data")), + _P("satellite-vhf", "VHF Satellite Downlinks", "Satellite", + 136_000_000, 138_000_000, 12_500, "nfm", 40_000, + "NOAA APT 137.100/137.9125, Meteor, cubesats", ("satellite",)), + _P("noaa-apt", "NOAA APT Weather Satellites", "Satellite", + 137_000_000, 138_000_000, 25_000, "wfm", 40_000, + "NOAA-15 137.620, NOAA-18 137.9125, NOAA-19 137.100", ("satellite",)), + _P("federal-vhf", "Federal Government VHF", "Government", + 138_000_000, 144_000_000, 12_500, "nfm", 12_500, + "Military land mobile and federal agencies", ("government",)), + _P("2m", "2 m Amateur", "Amateur Radio", + 144_000_000, 148_000_000, 15_000, "nfm", 16_000, + "CW/SSB 144.0-144.3, FM simplex/repeaters above", ("ham",)), + _P("2m-ssb", "2 m Weak Signal (CW/SSB)", "Amateur Radio", + 144_000_000, 144_300_000, 500, "usb", 2_800, + "144.200 SSB calling, FT8 at 144.174", ("ham", "cw")), + _P("2m-cw", "2 m CW", "Amateur Radio", + 144_000_000, 144_100_000, 200, "cw", 500, + "CW segment, EME and beacons at the bottom", ("ham", "cw")), + _P("2m-simplex", "2 m FM Simplex", "Amateur Radio", + 146_400_000, 147_600_000, 15_000, "nfm", 16_000, + "146.520 national simplex calling", ("ham",)), + _P("2m-repeaters", "2 m Repeater Outputs", "Amateur Radio", + 145_100_000, 148_000_000, 15_000, "nfm", 16_000, "", ("ham",)), + _P("aprs", "APRS / Packet 2 m", "Amateur Radio", + 144_380_000, 144_400_000, 12_500, "nfm", 16_000, + "144.390 AFSK1200 in North America", ("ham", "data")), + _P("federal-vhf-2", "Federal Government VHF (upper)", "Government", + 148_000_000, 150_800_000, 12_500, "nfm", 12_500, "", ("government",)), + _P("vhf-business", "VHF Business / Industrial", "Land Mobile", + 150_800_000, 156_000_000, 7_500, "nfm", 12_500, + "Itinerant, taxi, utilities, tow", ("business",)), + _P("murs", "MURS", "Land Mobile", + 151_820_000, 154_600_000, 7_500, "nfm", 12_500, + "5 licence-free channels: 151.820/.880/.940, 154.570/.600", ("business",)), + _P("railroad", "Railroad (AAR channels)", "Land Mobile", + 159_810_000, 161_565_000, 7_500, "nfm", 12_500, + "97 AAR road/yard channels", ("railroad",)), + _P("marine-vhf", "Marine VHF", "Marine", + 156_000_000, 162_025_000, 25_000, "nfm", 16_000, + "Ch 16 = 156.800 distress, Ch 13/22A working", ("marine",)), + _P("ais", "AIS Ship Transponders", "Marine", + 161_975_000, 162_025_000, 25_000, "raw", 25_000, + "AIS-A 161.975, AIS-B 162.025, 9600 GMSK", ("marine", "data")), + _P("noaa-weather", "NOAA Weather Radio", "Broadcast", + 162_400_000, 162_550_000, 25_000, "nfm", 16_000, + "7 channels, 162.400-162.550", ("broadcast",)), + _P("federal-vhf-3", "Federal VHF (162-174)", "Government", + 162_000_000, 174_000_000, 12_500, "nfm", 12_500, + "FBI, DHS, Forest Service, NPS", ("government",)), + _P("vhf-highband-ps", "VHF High Band Public Safety", "Public Safety", + 154_000_000, 160_000_000, 7_500, "nfm", 12_500, + "Fire, EMS, police, local government", ("publicsafety",)), + _P("wireless-mics-vhf", "Wireless Microphones VHF", "Audio / Production", + 169_000_000, 172_000_000, 25_000, "nfm", 25_000, + "Travelling band and assistive listening", ("production",)), + _P("tv-high-vhf", "VHF TV Channels 7-13", "Broadcast", + 174_000_000, 216_000_000, 6_000_000, "raw", 6_000_000, + "ATSC 8VSB", ("broadcast",)), + _P("dab-us", "VHF 216-225 (misc)", "Broadcast", + 216_000_000, 225_000_000, 25_000, "nfm", 25_000, + "Telemetry, AMTS maritime", ("misc",)), + + # ------------------------------------------------------------------ + # UHF + # ------------------------------------------------------------------ + _P("mil-uhf-air", "Military UHF Aircraft", "Aviation", + 225_000_000, 400_000_000, 25_000, "am", 12_500, + "243.000 guard, air refuelling, ranges", ("military", "aviation")), + _P("milsat-uhf", "UHF SATCOM Downlinks", "Satellite", + 240_000_000, 270_000_000, 5_000, "nfm", 25_000, + "FLTSATCOM / UFO downlinks", ("satellite", "military")), + _P("1.25m", "1.25 m (220 MHz) Amateur", "Amateur Radio", + 222_000_000, 225_000_000, 20_000, "nfm", 16_000, "", ("ham",)), + _P("1.25m-weak", "1.25 m Weak Signal (CW/SSB)", "Amateur Radio", + 222_000_000, 222_150_000, 500, "usb", 2_800, + "222.100 SSB calling", ("ham", "cw")), + _P("federal-uhf", "Federal Government UHF", "Government", + 406_100_000, 420_000_000, 12_500, "nfm", 12_500, + "Federal land mobile", ("government",)), + _P("radiosonde", "Radiosondes / Weather Balloons", "Telemetry / Control", + 400_000_000, 406_000_000, 10_000, "nfm", 15_000, + "RS41, DFM, iMet -- 403 MHz is the US cluster", ("telemetry",)), + _P("70cm", "70 cm Amateur", "Amateur Radio", + 420_000_000, 450_000_000, 12_500, "nfm", 16_000, + "Repeaters 440-450, ATV low", ("ham",)), + _P("70cm-cw", "70 cm CW / EME", "Amateur Radio", + 432_000_000, 432_100_000, 200, "cw", 500, + "CW and moonbounce at the bottom of the band", ("ham", "cw")), + _P("70cm-weak", "70 cm Weak Signal (SSB)", "Amateur Radio", + 432_100_000, 432_400_000, 500, "usb", 2_800, + "432.100 SSB calling", ("ham",)), + _P("70cm-fm", "70 cm FM Repeaters", "Amateur Radio", + 440_000_000, 450_000_000, 12_500, "nfm", 16_000, + "Repeater outputs and simplex", ("ham",)), + _P("70cm-simplex", "70 cm FM Simplex", "Amateur Radio", + 445_000_000, 447_000_000, 12_500, "nfm", 16_000, + "446.000 national simplex calling", ("ham",)), + _P("uhf-business", "UHF Business / Industrial", "Land Mobile", + 450_000_000, 470_000_000, 6_250, "nfm", 12_500, + "Includes 464/469 itinerant 'colour dot' channels", ("business",)), + _P("uhf-public-safety", "UHF Public Safety", "Public Safety", + 453_000_000, 460_000_000, 6_250, "nfm", 12_500, "", ("publicsafety",)), + _P("gmrs", "GMRS / FRS", "Land Mobile", + 462_550_000, 467_725_000, 12_500, "nfm", 12_500, + "22 shared channels + 8 GMRS repeater pairs", ("consumer",)), + _P("frs-simplex", "FRS Simplex Channels", "Land Mobile", + 462_562_500, 467_712_500, 12_500, "nfm", 12_500, + "Licence-free handhelds", ("consumer",)), + _P("dot-itinerant", "Itinerant 'Colour Dot' Channels", "Land Mobile", + 464_500_000, 469_562_500, 12_500, "nfm", 12_500, + "Red/Blue/Green/Purple dot business itinerant", ("business",)), + _P("uhf-tv", "UHF TV Channels 14-36", "Broadcast", + 470_000_000, 608_000_000, 6_000_000, "raw", 6_000_000, + "ATSC 1.0/3.0", ("broadcast",)), + _P("t-band", "T-Band Public Safety (470-512)", "Public Safety", + 470_000_000, 512_000_000, 12_500, "nfm", 12_500, + "Only in 11 major metro areas", ("publicsafety",)), + _P("wireless-mics-uhf", "Wireless Microphones UHF", "Audio / Production", + 512_000_000, 608_000_000, 25_000, "nfm", 200_000, + "Shure/Sennheiser IEM and mics in the TV band", ("production",)), + _P("600-duplex-gap", "600 MHz Duplex Gap / White Space", "Telecom", + 614_000_000, 698_000_000, 100_000, "raw", 200_000, + "T-Mobile 600 LTE, wireless mics in the gap", ("telecom",)), + _P("700-public-safety", "700 MHz Public Safety", "Public Safety", + 763_000_000, 806_000_000, 12_500, "nfm", 12_500, + "P25 Phase I/II narrowband + FirstNet LTE", ("publicsafety", "trunked")), + _P("800-public-safety", "800 MHz Public Safety / SMR", "Public Safety", + 806_000_000, 824_000_000, 12_500, "nfm", 12_500, + "NPSPAC, Motorola trunked systems", ("publicsafety", "trunked")), + _P("800-trunked", "800 MHz Trunked Downlinks", "Public Safety", + 851_000_000, 869_000_000, 12_500, "nfm", 12_500, + "Repeater outputs for 806-824 inputs", ("trunked",)), + _P("900-smr", "900 MHz SMR / Business", "Land Mobile", + 896_000_000, 940_000_000, 12_500, "nfm", 12_500, + "Includes 935-940 trunked outputs", ("business", "trunked")), + _P("cell-850", "Cellular 850 MHz", "Telecom", + 824_000_000, 894_000_000, 200_000, "raw", 1_400_000, + "GSM/LTE band 5", ("telecom",)), + _P("cell-1900", "PCS 1900 MHz", "Telecom", + 1_850_000_000, 1_990_000_000, 200_000, "raw", 1_400_000, + "LTE band 2/25 -- above most RTL tuners", ("telecom",)), + _P("33cm", "33 cm (902-928) Amateur", "Amateur Radio", + 902_000_000, 928_000_000, 25_000, "nfm", 16_000, "", ("ham",)), + _P("33cm-weak", "33 cm Weak Signal (CW/SSB)", "Amateur Radio", + 902_100_000, 903_100_000, 500, "usb", 2_800, + "903.100 SSB calling", ("ham", "cw")), + + # ------------------------------------------------------------------ + # ISM / SRD / consumer + # ------------------------------------------------------------------ + _P("ism-315", "ISM 315 MHz (SRD)", "ISM / Devices", + 314_900_000, 315_100_000, 10_000, "raw", 50_000, + "TPMS, key fobs, garage doors, OOK/FSK", ("ism",)), + _P("ism-390", "ISM 390 MHz (SRD)", "ISM / Devices", + 389_900_000, 390_100_000, 10_000, "raw", 50_000, + "GM/Ford TPMS and remotes", ("ism",)), + _P("ism-433", "ISM 433 MHz (SRD)", "ISM / Devices", + 433_050_000, 434_790_000, 10_000, "raw", 50_000, + "Weather stations, sensors, remotes", ("ism",)), + _P("ism-915", "ISM 902-928 MHz", "ISM / Devices", + 902_000_000, 928_000_000, 100_000, "raw", 200_000, + "Smart meters, LoRa, FHSS, Z-Wave 908.4", ("ism",)), + _P("zwave", "Z-Wave (US)", "ISM / Devices", + 908_000_000, 916_500_000, 100_000, "raw", 100_000, + "908.4 / 916.0 MHz", ("ism",)), + _P("tpms", "TPMS Tyre Sensors", "ISM / Devices", + 314_900_000, 315_100_000, 10_000, "raw", 50_000, + "Also check 433.92 MHz", ("ism",)), + _P("cordless-phones", "Cordless Phones (legacy)", "Consumer", + 43_000_000, 50_000_000, 20_000, "nfm", 12_500, + "43-50 MHz analogue handsets", ("consumer",)), + _P("baby-monitors", "Baby Monitors / Analogue Video", "Consumer", + 49_000_000, 50_000_000, 20_000, "nfm", 25_000, "", ("consumer",)), + + # ------------------------------------------------------------------ + # Data / paging / aviation surveillance + # ------------------------------------------------------------------ + _P("pagers-vhf", "VHF Paging (POCSAG/FLEX)", "Paging", + 152_000_000, 159_000_000, 12_500, "nfm", 12_500, + "152.0-152.24, 157.45, 158.1", ("paging", "data")), + _P("pagers-uhf", "UHF / 900 MHz Paging", "Paging", + 929_000_000, 932_000_000, 25_000, "nfm", 25_000, + "FLEX and POCSAG carriers", ("paging", "data")), + _P("adsb", "ADS-B (1090 MHz)", "Aviation", + 1_089_000_000, 1_091_000_000, 1_000_000, "raw", 2_000_000, + "Mode S extended squitter, PPM at 1 Mbit/s", ("aviation", "data")), + _P("uat-978", "UAT / ADS-B 978 MHz", "Aviation", + 977_000_000, 979_000_000, 1_000_000, "raw", 2_000_000, + "US general aviation ADS-B and FIS-B", ("aviation", "data")), + _P("dme-tacan", "DME / TACAN", "Aviation", + 960_000_000, 1_215_000_000, 1_000_000, "raw", 1_000_000, + "Pulse pairs, aircraft navigation", ("aviation",)), + _P("inmarsat", "Inmarsat L-band Downlink", "Satellite", + 1_525_000_000, 1_559_000_000, 25_000, "raw", 50_000, + "STD-C / AERO -- needs an LNA and a patch antenna", ("satellite",)), + _P("iridium", "Iridium Downlink", "Satellite", + 1_616_000_000, 1_626_500_000, 100_000, "raw", 500_000, + "Bursty QPSK, top of the RTL tuning range", ("satellite",)), + _P("gps-l1", "GPS L1", "Satellite", + 1_575_000_000, 1_576_000_000, 1_000_000, "raw", 2_000_000, + "Below the noise floor without correlation", ("satellite",)), + _P("weather-fax", "HF Weather Fax / RTTY", "Utility / Time", + 3_800_000, 17_200_000, 1_000, "usb", 2_800, + "NOAA HF FAX schedules", ("utility", "hf")), + + # ------------------------------------------------------------------ + # Broad sweeps + # ------------------------------------------------------------------ + _P("everything-vhf", "Full VHF Sweep", "Wide Sweeps", + 30_000_000, 300_000_000, 12_500, "nfm", 12_500, + "Slow but thorough", ("sweep",)), + _P("everything-uhf", "Full UHF Sweep", "Wide Sweeps", + 300_000_000, 1_000_000_000, 12_500, "nfm", 12_500, + "Very slow", ("sweep",)), + _P("everything", "Full Tuner Range Sweep", "Wide Sweeps", + 24_000_000, 1_700_000_000, 25_000, "nfm", 12_500, + "The whole R820T range; expect long cycle times", ("sweep",)), + _P("everything-hf", "Full HF Sweep (direct sampling)", "Wide Sweeps", + 500_000, 28_000_000, 1_000, "usb", 3_000, + "Requires direct sampling and an HF antenna", ("sweep", "hf")), + _P("common-scanner", "Common Scanner Bands", "Wide Sweeps", + 144_000_000, 174_000_000, 12_500, "nfm", 12_500, + "The classic 'action band'", ("sweep",)), + # Whole amateur bands. Mode "auto" and bandwidth 0 mean the demodulator + # and filter width are looked up per frequency from the segment presets + # above, so one range covers a band whose bottom is CW, middle SSB and + # top FM without any of it being demodulated the wrong way. + _P("160m-complete", "160 m Complete Band", "Amateur Radio", + 1_800_000, 2_000_000, 500, "auto", 0, + "CW, digital and LSB phone", ("ham", "hf", "complete")), + _P("80m-complete", "80/75 m Complete Band", "Amateur Radio", + 3_500_000, 4_000_000, 500, "auto", 0, + "CW below 3.600, LSB phone above", ("ham", "hf", "complete")), + _P("60m-complete", "60 m Complete Band", "Amateur Radio", + 5_330_500, 5_405_000, 100, "auto", 0, + "Five fixed USB channels", ("ham", "hf", "complete")), + _P("40m-complete", "40 m Complete Band", "Amateur Radio", + 7_000_000, 7_300_000, 500, "auto", 0, + "CW below 7.125, LSB phone above", ("ham", "hf", "complete")), + _P("30m-complete", "30 m Complete Band", "Amateur Radio", + 10_100_000, 10_150_000, 200, "auto", 0, + "CW and digital only, no phone", ("ham", "hf", "complete")), + _P("20m-complete", "20 m Complete Band", "Amateur Radio", + 14_000_000, 14_350_000, 500, "auto", 0, + "CW below 14.150, USB phone above", ("ham", "hf", "complete")), + _P("17m-complete", "17 m Complete Band", "Amateur Radio", + 18_068_000, 18_168_000, 500, "auto", 0, + "CW below 18.110, USB phone above", ("ham", "hf", "complete")), + _P("15m-complete", "15 m Complete Band", "Amateur Radio", + 21_000_000, 21_450_000, 500, "auto", 0, + "CW below 21.200, USB phone above", ("ham", "hf", "complete")), + _P("12m-complete", "12 m Complete Band", "Amateur Radio", + 24_890_000, 24_990_000, 500, "auto", 0, + "CW below 24.930, USB phone above", ("ham", "hf", "complete")), + _P("10m-complete", "10 m Complete Band", "Amateur Radio", + 28_000_000, 29_700_000, 1_000, "auto", 0, + "CW, USB phone, then FM repeaters at the top", ("ham", "hf", "complete")), + _P("6m-complete", "6 m Complete Band", "Amateur Radio", + 50_000_000, 54_000_000, 5_000, "auto", 0, + "CW, SSB, then FM above 50.3", ("ham", "complete")), + _P("2m-complete", "2 m Complete Band", "Amateur Radio", + 144_000_000, 148_000_000, 5_000, "auto", 0, + "CW, SSB, then FM above 144.3", ("ham", "complete")), + _P("1.25m-complete", "1.25 m Complete Band", "Amateur Radio", + 222_000_000, 225_000_000, 5_000, "auto", 0, + "Weak signal at the bottom, FM above", ("ham", "complete")), + _P("70cm-complete", "70 cm Complete Band", "Amateur Radio", + 420_000_000, 450_000_000, 6_250, "auto", 0, + "ATV and weak signal low, FM repeaters above 440", + ("ham", "complete")), + _P("33cm-complete", "33 cm Complete Band", "Amateur Radio", + 902_000_000, 928_000_000, 12_500, "auto", 0, + "Shared with ISM devices throughout", ("ham", "complete")), + + _P("all-cw", "All CW Segments", "Wide Sweeps", + 1_800_000, 432_100_000, 200, "cw", 500, + "Every CW allocation from 160 m to 70 cm in one sweep; the HF part " + "needs direct sampling and an HF antenna", + ("sweep", "cw", "ham"), + members=("160m-cw", "80m-cw", "40m-cw", "30m", "20m-cw", "17m-cw", + "15m-cw", "12m-cw", "10m-cw", "6m-cw", "2m-cw", "70cm-cw")), +) + + +CATEGORIES: tuple[str, ...] = tuple( + dict.fromkeys(p.category for p in PRESETS) +) + +_BY_KEY = {p.key: p for p in PRESETS} + + +def by_key(key: str) -> BandPreset | None: + return _BY_KEY.get(key.strip().lower()) + + +def in_category(category: str) -> list[BandPreset]: + cl = category.strip().lower() + return [p for p in PRESETS if p.category.lower() == cl] + + +def search(term: str) -> list[BandPreset]: + """Fuzzy-ish preset lookup over key, name, note and tags.""" + t = term.strip().lower() + if not t: + return [] + hits = [] + for p in PRESETS: + hay = " ".join((p.key, p.name, p.note, p.category, " ".join(p.tags))).lower() + if t in hay: + hits.append(p) + return hits + + +def presets_covering(hz: float) -> list[BandPreset]: + """Every preset whose span contains ``hz`` -- used to label detections.""" + return [p for p in PRESETS if not p.is_group and p.start <= hz <= p.stop] + + +def expand_preset(key: str) -> list[BandPreset]: + """The presets a key stands for, following groups to their members.""" + preset = by_key(key) + return preset.expand() if preset is not None else [] diff --git a/bandsaunter/classify.py b/bandsaunter/classify.py new file mode 100755 index 0000000..abfa337 --- /dev/null +++ b/bandsaunter/classify.py @@ -0,0 +1,980 @@ +"""Modulation and signal-type classification. + +The classifier works on a block of complex baseband that has already been +centred on the signal and decimated to a rate a few times its bandwidth. It +extracts a feature vector, scores it against a rule set for the common +modulation families, then refines the answer with a table of known systems +keyed on frequency, bandwidth and symbol rate. + +Nothing here decodes traffic; it names what the signal *is*. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass, field + +import numpy as np +from scipy import signal as sps + +from .dsp import (db, instantaneous_frequency, + occupied_bandwidth, spectral_flatness, welch_psd) + +__all__ = ["classify", "Classification", "SignalFeatures", "extract_features", + "CTCSS_TONES", "detect_ctcss"] + + +def _pow2_floor(n: int, cap: int = 1 << 16) -> int: + """Largest power of two that is <= n (and <= cap).""" + n = int(min(n, cap)) + return 1 << int(math.floor(math.log2(max(2, n)))) + + +# EIA/TIA-603 standard CTCSS tones, Hz. +CTCSS_TONES = ( + 67.0, 69.3, 71.9, 74.4, 77.0, 79.7, 82.5, 85.4, 88.5, 91.5, + 94.8, 97.4, 100.0, 103.5, 107.2, 110.9, 114.8, 118.8, 123.0, 127.3, + 131.8, 136.5, 141.3, 146.2, 151.4, 156.7, 159.8, 162.2, 165.5, 167.9, + 171.3, 173.8, 177.3, 179.9, 183.5, 186.2, 189.9, 192.8, 196.6, 199.5, + 203.5, 206.5, 210.7, 218.1, 225.7, 229.1, 233.6, 241.8, 250.3, 254.1, +) + +# Symbol rates worth naming when the cyclostationary estimator lands near one. +_KNOWN_BAUD = ( + (300, "300 baud"), (512, "POCSAG-512"), (1200, "1200 baud"), + (1600, "FLEX-1600"), (2400, "2400 baud"), (3200, "FLEX-3200"), + (4800, "4800 baud"), (6400, "FLEX-6400"), (9600, "9600 baud"), + (18000, "TETRA"), (19200, "19.2 kbaud"), (36000, "36 kbaud"), +) + + +@dataclass +class SignalFeatures: + """Everything the rules and the report are allowed to look at.""" + + sample_rate: float + n_samples: int + duration: float + + # spectrum + bandwidth: float = 0.0 # span holding 99% of the power + bw_noise: float = 0.0 # span standing above the noise floor + am_depth: float = 0.0 # envelope modulation in the audio band + bw3: float = 0.0 + bw20: float = 0.0 + centre_offset: float = 0.0 + flatness: float = 0.0 + papr_spectral: float = 0.0 # dB, peak bin over median bin + carrier_ratio: float = 0.0 # fraction of power in the peak bin + symmetry: float = 0.0 # -1 all lower sideband, +1 all upper + snr_db: float = 0.0 + + # envelope + env_cv: float = 0.0 # std/mean of |x| + env_kurtosis: float = 0.0 + ook_contrast_db: float = 0.0 + ook_duty: float = 0.0 + keying_regularity: float = 0.0 # do on/off runs fit a symbol grid? + is_bursty: bool = False + burst_rate_hz: float = 0.0 + duty_cycle: float = 1.0 + + # frequency / phase + fdev_rms: float = 0.0 + fdev_peak: float = 0.0 + fdev_ratio: float = 0.0 # fdev_rms / bandwidth: separates AM from SSB/FM + ifreq_kurtosis: float = 0.0 # peaky (PSK) vs multimodal (FSK) + level_dwell: float = 0.0 # fraction of time the tone sits still + freq_modes: int = 0 + mode_spacing: float = 0.0 + psk_order: int = 0 + psk_strength: float = 0.0 + + # timing + baud: float = 0.0 + baud_strength: float = 0.0 + baud_stability: float = 0.0 # does the symbol rate hold across the capture? + + # audio-domain + ctcss_hz: float = 0.0 + has_subaudible_data: bool = False + stereo_pilot: bool = False + afsk_1200: bool = False + audio_peak_hz: float = 0.0 + + extras: dict = field(default_factory=dict) + + +@dataclass +class Classification: + label: str + family: str + confidence: float + reasons: list[str] = field(default_factory=list) + alternatives: list[tuple[str, float]] = field(default_factory=list) + suggested_mode: str = "nfm" + features: SignalFeatures | None = None + + def summary(self) -> str: + pct = int(round(self.confidence * 100)) + return f"{self.label} ({pct}%)" + + +# --------------------------------------------------------------------------- +# Feature extraction +# --------------------------------------------------------------------------- + +def _otsu(values: np.ndarray, bins: int = 128) -> float: + """Otsu threshold -- splits an on/off envelope into its two populations.""" + hist, edges = np.histogram(values, bins=bins) + hist = hist.astype(np.float64) + total = hist.sum() + if total == 0: + return float(np.median(values)) + centres = 0.5 * (edges[1:] + edges[:-1]) + w0 = np.cumsum(hist) + w1 = total - w0 + mu0 = np.cumsum(hist * centres) / np.maximum(w0, 1e-12) + grand = (hist * centres).sum() + mu1 = (grand - np.cumsum(hist * centres)) / np.maximum(w1, 1e-12) + var_between = w0 * w1 * (mu0 - mu1) ** 2 + var_between[~np.isfinite(var_between)] = 0.0 + return float(centres[int(np.argmax(var_between))]) + + +def _cyclic_line(feature: np.ndarray, fs: float, + lo_hz: float = 40.0, hi_hz: float | None = None): + """Find the strongest periodic line in a nonnegative feature signal. + + Symbol transitions sit on a symbol-rate grid, so the transition-magnitude + signal carries a spectral line at the baud rate. Returns + ``(frequency_hz, prominence_db)``. + """ + n = feature.size + if n < 256: + return 0.0, 0.0 + hi_hz = hi_hz or fs / 2.5 + x = feature.astype(np.float64) + x = x - x.mean() + if not np.any(x): + return 0.0, 0.0 + nfft = _pow2_floor(n) + x = x[:nfft] * np.hanning(nfft) + spec = np.abs(np.fft.rfft(x, nfft)) + freqs = np.fft.rfftfreq(nfft, 1.0 / fs) + + band = (freqs >= lo_hz) & (freqs <= hi_hz) + if not np.any(band): + return 0.0, 0.0 + sub = spec[band] + subf = freqs[band] + k = int(np.argmax(sub)) + peak = sub[k] + med = np.median(sub) + 1e-12 + return float(subf[k]), float(20.0 * math.log10(peak / med)) + + +def _count_modes(values: np.ndarray, weights: np.ndarray | None = None, + bins: int = 96): + """Histogram-based mode counting for FSK level detection.""" + if values.size < 64: + return 0, 0.0, np.zeros(0) + lo, hi = np.percentile(values, [1.0, 99.0]) + if hi <= lo: + return 0, 0.0, np.zeros(0) + hist, edges = np.histogram(values, bins=bins, range=(lo, hi), weights=weights) + hist = hist.astype(np.float64) + if hist.sum() == 0: + return 0, 0.0, np.zeros(0) + # Light smoothing so shot noise does not create spurious modes. + kern = np.array([1.0, 3.0, 6.0, 8.0, 6.0, 3.0, 1.0]) + kern /= kern.sum() + sm = np.convolve(hist, kern, mode="same") + centres = 0.5 * (edges[1:] + edges[:-1]) + + peaks, props = sps.find_peaks(sm, height=0.22 * sm.max(), + distance=max(3, bins // 16), + prominence=0.15 * sm.max()) + if peaks.size == 0: + return 0, 0.0, np.zeros(0) + order = np.argsort(props["peak_heights"])[::-1][:8] + sel = np.sort(peaks[order]) + locs = centres[sel] + if locs.size < 2: + return int(locs.size), 0.0, locs + + diffs = np.diff(locs) + spacing = float(np.median(diffs)) + # FSK levels are evenly spaced. If the peaks fit a uniform grid, report + # the grid size instead of the raw peak count -- one spurious shoulder + # should not turn 4-FSK into 5-FSK. + if spacing > 0: + grid = (locs - locs[0]) / spacing + if np.max(np.abs(grid - np.round(grid))) < 0.28: + n_levels = int(round(grid[-1])) + 1 + if 2 <= n_levels <= 8: + return n_levels, spacing, locs + return int(locs.size), spacing, locs + + +def _psk_order(x: np.ndarray, fs: float): + """Detect M-PSK by looking for the spectral line produced by x**M.""" + if x.size < 1024: + return 0, 0.0 + xn = x / (np.abs(x) + 1e-9) # constant-modulus, phase only + best = (0, 0.0) + for m in (2, 4, 8): + y = xn ** m + nfft = _pow2_floor(y.size, 1 << 15) + spec = np.abs(np.fft.fftshift(np.fft.fft(y[:nfft] * np.hanning(nfft), nfft))) + peak = spec.max() + med = np.median(spec) + 1e-12 + strength = 20.0 * math.log10(peak / med) + if strength > best[1]: + best = (m, strength) + return best + + +def detect_ctcss(audio: np.ndarray, fs: float): + """Return ``(tone_hz, is_dcs_like)`` from a demodulated FM audio block.""" + if audio.size < int(fs * 0.25): + return 0.0, False + n = _pow2_floor(audio.size) + x = audio[:n].astype(np.float64) + x = x - x.mean() + spec = np.abs(np.fft.rfft(x * np.hanning(n), n)) + freqs = np.fft.rfftfreq(n, 1.0 / fs) + + sub = (freqs >= 60.0) & (freqs <= 260.0) + voice = (freqs >= 300.0) & (freqs <= 3000.0) + if not np.any(sub): + return 0.0, False + sub_spec = spec[sub] + sub_f = freqs[sub] + k = int(np.argmax(sub_spec)) + peak_f = float(sub_f[k]) + peak_v = float(sub_spec[k]) + floor = float(np.median(spec[voice])) + 1e-12 if np.any(voice) else 1e-12 + + if peak_v / floor < 6.0: + return 0.0, False + # A CTCSS tone is a single sharp line; DCS is a 134.4 bps square wave and + # spreads its energy across the whole subaudible region. + band_energy = float(np.sum(sub_spec ** 2)) + tone_energy = float(np.sum(sub_spec[max(0, k - 2):k + 3] ** 2)) + if tone_energy / max(band_energy, 1e-12) < 0.35: + return 0.0, True + + nearest = min(CTCSS_TONES, key=lambda t: abs(t - peak_f)) + if abs(nearest - peak_f) <= max(1.5, 0.02 * nearest): + return float(nearest), False + return 0.0, False + + +def _center_and_filter(x: np.ndarray, sample_rate: float, offset_hz: float, + bw_hz: float, wide_bw_hz: float = 0.0) -> np.ndarray: + """Shift the signal to DC and low-pass it to its own occupied bandwidth. + + Time-domain features (envelope, discriminator, phase) are meaningless when + they are dominated by noise from the rest of the IF, so every measurement + after the spectrum step runs on this filtered copy. + """ + # Be generous: a carrier-dominated signal (AM) has a small 99%-power + # bandwidth but sidebands well outside it, so take the wider of the two + # measures and leave headroom on top. + keep = max(bw_hz, wide_bw_hz) * 1.8 + keep = max(keep, sample_rate / 200.0) + if keep >= sample_rate * 0.9 and abs(offset_hz) < sample_rate / 50.0: + return x # already occupies most of the band; filtering buys nothing + + if abs(offset_hz) > sample_rate / 1000.0: + n = np.arange(x.size, dtype=np.float64) + x = (x * np.exp(-2j * math.pi * offset_hz * n / sample_rate)).astype(np.complex64) + + # firwin's cutoff is in units of Nyquist; we want +-keep/2 around DC. + norm = (keep / 2.0) / (sample_rate / 2.0) + if norm >= 0.95: + return x + ntaps = 127 + taps = sps.firwin(ntaps, norm).astype(np.float64) + y = sps.lfilter(taps, [1.0], x).astype(np.complex64) + # Drop the filter's start-up ramp: it looks exactly like a signal fading + # in, which would otherwise register as on-off keying. + return y[ntaps:] if y.size > 4 * ntaps else y + + +def _robust_floor(psd_db: np.ndarray) -> tuple[float, float]: + """Noise floor and spread of one spectrum, by sigma clipping. + + Repeatedly drops the bins that stand out until only the noise population + is left. A sliding percentile is the right tool while sweeping, where + signals are narrow slivers of a wide span, but not here: by this point + the capture is centred on one signal that may fill most of the analysis + band, and a windowed percentile would sit on the signal itself. + """ + v = np.asarray(psd_db, dtype=np.float64) + if v.size < 8: + return float(np.median(v)), 1.0 + mask = np.ones(v.size, dtype=bool) + floor = float(np.median(v)) + sigma = 1.0 + for _ in range(6): + sel = v[mask] + if sel.size < max(8, v.size // 10): + break + floor = float(np.median(sel)) + sigma = float(1.4826 * np.median(np.abs(sel - floor))) or 1.0 + new_mask = v < floor + max(3.0 * sigma, 3.0) + if new_mask.sum() < max(8, v.size // 10): + break + if np.array_equal(new_mask, mask): + break + mask = new_mask + return floor, max(sigma, 0.3) + + +def _occupied_span(psd_db: np.ndarray, bin_hz: float, + margin_db: float = 0.0) -> tuple[float, float]: + """Occupied bandwidth as an analyst reads it off a spectrum display. + + Returns ``(bandwidth_hz, centre_offset_hz)`` for the contiguous run of + bins standing above the noise floor around the strongest peak. + + A 99%-of-power measure cannot be used here: AM puts almost all of its + power in the carrier, so 99% of the power lives in a single bin and the + channel would be reported as tens of hertz wide. + """ + n = psd_db.size + if n < 8: + return bin_hz, 0.0 + floor, sigma = _robust_floor(psd_db) + threshold = floor + (margin_db or max(6.0, 4.0 * sigma)) + above = psd_db > threshold + if not np.any(above): + return bin_hz, 0.0 + + idx = np.flatnonzero(above) + # 2nd-to-98th percentile of the *positions* that stand above the noise. + # Counting positions rather than weighting them by power matters: a + # carrier holds so much more power than its sidebands that a weighted + # measure collapses onto the carrier bin and reports a few hertz. + lo = int(np.percentile(idx, 2)) + hi = int(np.percentile(idx, 98)) + if hi < lo: + lo, hi = hi, lo + w = np.maximum(psd_db[idx] - floor, 1e-9) + centroid = float(np.dot(idx.astype(np.float64), w) / w.sum()) + offset = (centroid - (n - 1) / 2.0) * bin_hz + return max(bin_hz, float((hi - lo + 1) * bin_hz)), offset + + +def _run_spans(on: np.ndarray): + """Yield ``(state, start, length)`` for each constant run of a mask.""" + if on.size == 0: + return + change = np.flatnonzero(np.diff(on.astype(np.int8))) + bounds = np.concatenate(([0], change + 1, [on.size])) + for i in range(bounds.size - 1): + yield bool(on[bounds[i]]), int(bounds[i]), int(bounds[i + 1] - bounds[i]) + + +def _longest_active_run(on: np.ndarray) -> tuple[int, int]: + """Start and length of the longest contiguous key-down / active stretch.""" + best = (0, 0) + for state, start, length in _run_spans(on): + if state and length > best[1]: + best = (start, length) + return best + + +def extract_features(x: np.ndarray, sample_rate: float, + snr_db: float = 0.0) -> SignalFeatures: + """Compute the full feature vector for one captured block.""" + x = np.asarray(x, dtype=np.complex64) + n = x.size + f = SignalFeatures(sample_rate=float(sample_rate), n_samples=n, + duration=n / float(sample_rate), snr_db=float(snr_db)) + if n < 512: + return f + + # ---- keying structure, measured across the whole capture ---------- + env_full = np.abs(x).astype(np.float64) + smooth_n = max(4, int(sample_rate / 4000.0)) + env_s = np.convolve(env_full, np.ones(smooth_n) / smooth_n, mode="same") + thr = _otsu(env_s) + on = env_s > thr + f.ook_duty = float(on.mean()) + f.duty_cycle = f.ook_duty + if np.any(on) and np.any(~on): + hi = float(np.mean(env_s[on])) + lo = float(np.mean(env_s[~on])) + 1e-12 + f.ook_contrast_db = float(20.0 * math.log10(hi / lo)) + + # Do the on and off runs land on a common grid? Keyed data and Morse + # both quantise to a symbol or dot length; a signal drifting across the + # squelch threshold produces runs of every length, which is what tells + # real keying apart from a fading carrier. + spans = [ln for _, _, ln in _run_spans(on)] + if len(spans) >= 6: + arr = np.array(spans[1:-1] if len(spans) > 8 else spans, dtype=np.float64) + unit = float(np.percentile(arr, 20)) + # The unit has to be resolvable. When the envelope merely jitters + # across the threshold the runs are one or two samples long, and every + # length is then trivially an "integer multiple" of one sample -- a + # test that noise passes perfectly. + if unit >= 8.0: + ratios = arr / unit + usable = ratios[ratios <= 12.0] + if usable.size >= 4: + f.keying_regularity = float( + np.mean(np.abs(usable - np.round(usable)) < 0.2)) + + # Burst / TDMA structure: how often does the envelope gate on and off? + if 0.02 < f.ook_duty < 0.98: + edges = np.diff(on.astype(np.int8)) + rises = np.where(edges > 0)[0] + if rises.size >= 3: + periods = np.diff(rises) / sample_rate + med = float(np.median(periods)) + if med > 0: + f.burst_rate_hz = 1.0 / med + f.is_bursty = bool(np.std(periods) / med < 0.5) + + # ---- pick the stretch to characterise ------------------------------ + # Speech has pauses, and on SSB the carrier disappears with them. Judging + # modulation across the silence would describe the silence: an AM voice + # channel reads as a bare carrier, SSB voice reads as on-off keying. So + # measure the longest continuously-active stretch instead -- provided it + # is long enough to be a transmission rather than a data symbol. + seg = x + if 0.05 < f.ook_duty < 0.92: + s_start, s_len = _longest_active_run(on) + if s_len >= max(int(0.3 * sample_rate), 4096) and s_len < int(0.92 * n): + seg = x[s_start:s_start + s_len] + f.extras["analysed_seconds"] = round(s_len / sample_rate, 3) + f.extras["analysed_fraction"] = round(s_len / n, 3) + + # ---- spectrum ------------------------------------------------------ + nfft = min(4096, 1 << int(math.floor(math.log2(seg.size)))) + freqs, psd = welch_psd(seg, nfft) + bin_hz = sample_rate / nfft + psd_db = db(psd) + + # Primary bandwidth is the span holding 99% of the power. The span + # standing above the noise is kept alongside it because the two disagree + # in a useful way: AM puts nearly all its power in the carrier, so a large + # gap between them is itself evidence of a carrier-dominated signal. + f.bandwidth, f.centre_offset = occupied_bandwidth(psd, bin_hz, 0.99) + f.bw_noise, _ = _occupied_span(psd_db, bin_hz) + total = psd.sum() + peak_lin = psd.max() + f.carrier_ratio = float(peak_lin / total) if total > 0 else 0.0 + f.papr_spectral = float(psd_db.max() - np.median(psd_db)) + f.flatness = spectral_flatness(psd) + + peak_db = psd_db.max() + for lvl, attr in ((3.0, "bw3"), (20.0, "bw20")): + above = psd_db >= (peak_db - lvl) + if np.any(above): + idx = np.where(above)[0] + setattr(f, attr, float((idx[-1] - idx[0] + 1) * bin_hz)) + + # Sideband asymmetry measured about the strongest bin and only across the + # occupied band -- comparing the two halves of the whole IF just measures + # where the noise sits. + pk = int(np.argmax(psd)) + span = max(2, int(f.bandwidth / bin_hz)) + lo_i, hi_i = max(0, pk - span), min(nfft, pk + span + 1) + lower = float(psd[lo_i:pk].sum()) + upper = float(psd[pk + 1:hi_i].sum()) + if lower + upper > 0: + f.symmetry = (upper - lower) / (upper + lower) + + # Every time-domain measurement below runs on the signal alone. + # Size the analysis filter from the widest honest estimate. Using the + # 99%-power figure alone would band-limit an AM channel to its carrier and + # every later measurement would describe a dead carrier. + keep_bw = max(f.bandwidth, f.bw_noise, f.bw20) + xf = _center_and_filter(seg, sample_rate, f.centre_offset, keep_bw, f.bw20) + f.extras["_filtered"] = xf + seg = xf + + # ---- envelope ----------------------------------------------------- + env = np.abs(seg).astype(np.float64) + mean_env = float(env.mean()) + if mean_env > 0: + f.env_cv = float(env.std() / mean_env) + centred = env - mean_env + var = float(centred.var()) + if var > 0: + f.env_kurtosis = float(np.mean(centred ** 4) / (var ** 2) - 3.0) + + # Does the envelope carry audio? A bare carrier's envelope is flat, an + # FM carrier's is flat by construction, and an AM or SSB voice channel's + # envelope *is* the speech. This is what tells a modulated AM channel + # apart from a dead carrier, which the 99%-power bandwidth cannot do + # because the carrier holds nearly all the power either way. + if mean_env > 0 and env.size > 1024: + ac = env - mean_env + dec = max(1, int(sample_rate / 16_000)) + a = ac[::dec] + fs_a = sample_rate / dec + n_a = 1 << int(math.floor(math.log2(max(256, min(a.size, 1 << 15))))) + if a.size >= n_a: + spec = np.abs(np.fft.rfft(a[:n_a] * np.hanning(n_a), n_a)) ** 2 + fr = np.fft.rfftfreq(n_a, 1.0 / fs_a) + band = (fr >= 100.0) & (fr <= min(4000.0, fs_a / 2.2)) + if np.any(band): + f.am_depth = float(np.sqrt(np.sum(spec[band])) / + (mean_env * n_a / 2.0)) + + # ---- instantaneous frequency -------------------------------------- + ifreq = instantaneous_frequency(seg, sample_rate) + if ifreq.size: + w = env[1:] + strong = w > (0.5 * mean_env) if mean_env > 0 else np.ones_like(w, bool) + sel = ifreq[strong] if strong.sum() > 64 else ifreq + f.fdev_rms = float(np.std(sel)) + f.fdev_peak = float(np.percentile(np.abs(sel - np.mean(sel)), 99.0)) + f.fdev_ratio = f.fdev_rms / max(f.bandwidth, 1.0) + centred = sel - np.mean(sel) + var = float(centred.var()) + if var > 0: + # Negative => the discriminator sits on discrete levels (FSK); + # strongly positive => it is flat with impulses (PSK phase jumps). + f.ifreq_kurtosis = float(np.mean(centred ** 4) / (var ** 2) - 3.0) + + # How much of the time does the tone hold still? FSK parks on a level + # for a whole symbol and jumps between them, so its slope is zero + # almost everywhere. Tone-modulated or voice FM sweeps continuously + # and is never flat for long. Without this, the two peaks that + # sinusoidal FM puts at the ends of its swing (a CTCSS tone during a + # speech pause, say) read as a pair of FSK levels. + slope = np.diff(sel) + if slope.size > 64: + scale = float(np.std(slope)) + if scale > 0: + f.level_dwell = float(np.mean(np.abs(slope) < 0.25 * scale)) + modes, spacing, _ = _count_modes(sel, weights=None) + f.freq_modes = modes + f.mode_spacing = spacing + + # ---- phase -------------------------------------------------------- + order, strength = _psk_order(seg, sample_rate) + f.psk_order, f.psk_strength = order, strength + + # ---- symbol rate --------------------------------------------------- + # For FSK the transition magnitude of the discriminator carries the line; + # for linear modulations the squared envelope does. + cand = [] + if ifreq.size > 512: + cand.append((_cyclic_line(np.abs(np.diff(ifreq)), sample_rate), + np.abs(np.diff(ifreq)))) + cand.append((_cyclic_line(np.abs(np.diff(env)), sample_rate), + np.abs(np.diff(env)))) + cand.append((_cyclic_line(env ** 2, sample_rate), env ** 2)) + (baud, strength), winner = max(cand, key=lambda c: c[0][1]) + if strength > 8.0: + f.baud, f.baud_strength = baud, strength + # A real symbol rate is a property of the transmission and holds for + # its whole length. The estimator always returns its best peak, so + # on noise it returns a different answer for each half of the same + # capture -- which is exactly how to tell the two apart. + half = winner.size // 2 + if half > 1024: + b1, _ = _cyclic_line(winner[:half], sample_rate) + b2, _ = _cyclic_line(winner[half:], sample_rate) + if b1 > 0 and b2 > 0: + f.baud_stability = float( + 1.0 - abs(b1 - b2) / max(b1, b2)) + + return f + + +def _analyse_fm_audio(x: np.ndarray, sample_rate: float, f: SignalFeatures) -> None: + """Fill in the audio-domain features that need an FM demodulation.""" + ifreq = instantaneous_frequency(x, sample_rate) + if ifreq.size < 1024: + return + + # 19 kHz stereo pilot -> broadcast FM. + if sample_rate > 60_000: + n = _pow2_floor(ifreq.size) + spec = np.abs(np.fft.rfft((ifreq[:n] - ifreq[:n].mean()) * np.hanning(n), n)) + freqs = np.fft.rfftfreq(n, 1.0 / sample_rate) + near = (freqs > 18_800) & (freqs < 19_200) + ref = (freqs > 22_000) & (freqs < 30_000) + if np.any(near) and np.any(ref): + f.stereo_pilot = bool(spec[near].max() > 8.0 * (np.median(spec[ref]) + 1e-12)) + + # Decimate the discriminator output to a voice rate for tone work. + dec = max(1, int(sample_rate // 16_000)) + audio = sps.decimate(ifreq, dec, ftype="fir", zero_phase=False) if dec > 1 else ifreq + fs_a = sample_rate / dec + + tone, dcs = detect_ctcss(audio, fs_a) + f.ctcss_hz = tone + f.has_subaudible_data = dcs + + n = _pow2_floor(audio.size, 1 << 15) + if n >= 1024: + a = audio[:n].astype(np.float64) + a -= a.mean() + spec = np.abs(np.fft.rfft(a * np.hanning(n), n)) + freqs = np.fft.rfftfreq(n, 1.0 / fs_a) + band = (freqs > 250.0) & (freqs < 3500.0) + if np.any(band): + sb, sf = spec[band], freqs[band] + f.audio_peak_hz = float(sf[int(np.argmax(sb))]) + # AFSK1200 (APRS, Bell 202) sits on 1200 Hz and 2200 Hz marks. + def energy(target, width=90.0): + m = (sf > target - width) & (sf < target + width) + return float(sb[m].max()) if np.any(m) else 0.0 + base = float(np.median(sb)) + 1e-12 + e12, e22 = energy(1200.0), energy(2200.0) + f.afsk_1200 = bool(e12 > 5 * base and e22 > 5 * base) + + +# --------------------------------------------------------------------------- +# Rule engine +# --------------------------------------------------------------------------- + +def _score_rules(f: SignalFeatures) -> list[tuple[str, str, float, str, str]]: + """Return ``(label, family, score, reason, suggested_mode)`` candidates.""" + out = [] + + def add(label, family, score, reason, mode="nfm"): + if score > 0: + out.append((label, family, float(score), reason, mode)) + + const_env = f.env_cv < 0.35 + narrow = f.bandwidth < 30_000 + very_narrow = f.bandwidth < 1_500 + # A dead carrier occupies essentially no bandwidth, whatever the noise does + # to its measured phase jitter. + tone_bw = max(60.0, f.sample_rate / 400.0) + + # -- unmodulated carrier / tone ------------------------------------- + if f.bandwidth < tone_bw and f.env_cv < 0.20 \ + and (f.ook_contrast_db < 6 or f.ook_duty > 0.93): + add("Unmodulated carrier", "carrier", + 0.72 + min(0.2, f.papr_spectral / 200.0), + f"single steady line only {f.bandwidth:.0f} Hz wide, no keying or " + f"modulation sidebands", "cw") + + # -- CW / on-off keying --------------------------------------------- + if f.ook_contrast_db > 10 and very_narrow and 0.05 < f.ook_duty < 0.85: + base = 0.55 + min(0.3, f.ook_contrast_db / 60.0) + if f.baud and f.baud < 60: + base += 0.1 + add("CW / Morse (on-off keyed carrier)", "cw", base, + f"keyed carrier, {f.ook_contrast_db:.0f} dB on/off contrast, " + f"{f.bandwidth:.0f} Hz wide", "cw") + elif f.ook_contrast_db > 12 and f.bandwidth < 60_000 and 0.02 < f.ook_duty < 0.9: + add("OOK / ASK data burst", "ook", + 0.5 + min(0.25, f.ook_contrast_db / 80.0), + f"on-off keying, {f.ook_contrast_db:.0f} dB contrast, " + f"{f.baud:.0f} baud" if f.baud else "on-off keying", "raw") + + # -- FSK families ---------------------------------------------------- + # Discrete levels, and the tone actually rests on them. + level_like = f.ifreq_kurtosis < 1.5 and f.level_dwell > 0.32 + if const_env and f.freq_modes >= 2 and f.fdev_rms > 200 and level_like: + levels = f.freq_modes + if levels in (2, 3): + name, conf = "2-FSK (binary FSK)", 0.6 + elif levels == 4: + name, conf = "4-FSK / C4FM", 0.68 + elif levels in (5, 6, 7, 8): + name, conf = f"{levels}-level FSK", 0.5 + else: + name, conf = "Multi-level FSK", 0.4 + if f.baud_strength > 12: + conf += 0.12 + reason = (f"constant envelope, {levels} discriminator levels " + f"{f.mode_spacing:.0f} Hz apart") + if f.baud: + reason += f", ~{f.baud:.0f} baud" + add(name, "fsk", conf, reason, "nfm") + + # -- analogue FM ----------------------------------------------------- + # A quiet FM channel carrying only a CTCSS tone has very little deviation, + # so this floor has to sit low. + if const_env and f.fdev_rms > 120 and not level_like: + if f.bandwidth > 100_000: + conf = 0.72 + (0.15 if f.stereo_pilot else 0.0) + label = "Wideband FM (broadcast)" + if f.stereo_pilot: + label = "Wideband FM broadcast (stereo, 19 kHz pilot)" + add(label, "wfm", conf, + f"{f.bandwidth/1e3:.0f} kHz wide, {f.fdev_rms/1e3:.1f} kHz rms deviation", + "wfm") + elif narrow: + conf = 0.6 + if f.ctcss_hz: + conf += 0.2 + if 300 < f.audio_peak_hz < 3200: + conf += 0.08 + label = "Narrowband FM voice" + if f.ctcss_hz: + label += f" (CTCSS {f.ctcss_hz:.1f} Hz)" + elif f.has_subaudible_data: + label += " (DCS subaudible data)" + add(label, "nfm", conf, + f"{f.bandwidth/1e3:.1f} kHz wide, {f.fdev_rms/1e3:.1f} kHz rms deviation", + "nfm") + + # -- AM --------------------------------------------------------------- + # AM keeps its carrier, so the phase hardly moves and the sidebands are + # mirror images. That is exactly what separates it from SSB. + # Real AM often runs at modest modulation depth, so the envelope only has + # to vary more than receiver noise alone would make it vary. + snr_lin = 10.0 ** (max(f.snr_db, 0.0) / 10.0) + env_noise = 1.0 / math.sqrt(max(2.0, snr_lin)) + am_modulated = f.env_cv > max(0.045, 2.0 * env_noise) + if (am_modulated and narrow and f.bandwidth > 4 * tone_bw + and f.carrier_ratio > 0.08 + and f.fdev_ratio < 0.18 and abs(f.symmetry) < 0.60): + conf = 0.55 + min(0.2, 2.0 * f.env_cv) - 0.2 * abs(f.symmetry) + add("AM (amplitude modulation)", "am", conf, + f"symmetric sidebands around a surviving carrier, " + f"{f.env_cv:.2f} envelope variation, {f.bandwidth/1e3:.1f} kHz wide", "am") + + # -- SSB --------------------------------------------------------------- + if (300 < f.bandwidth < 6_500 and f.env_cv > 0.35 and f.fdev_ratio > 0.15 + and f.ook_contrast_db < 14): + # Which sideband cannot be recovered once we have centred on the + # signal, so fall back on the HF convention (LSB below 10 MHz). + conf = 0.52 + min(0.2, f.fdev_ratio) + min(0.15, abs(f.symmetry) * 0.2) + add("SSB voice (suppressed carrier)", "ssb", conf, + f"no carrier line, phase swings across the full {f.bandwidth:.0f} Hz " + f"of audio bandwidth", "usb") + + # -- PSK --------------------------------------------------------------- + impulsive_phase = f.ifreq_kurtosis > 3.0 + fsk_like = f.freq_modes >= 2 and f.mode_spacing > 0 and level_like + # Raising to the Mth power must *create* the spectral line. An + # unmodulated carrier -- including the silent gaps between phrases on an + # FM channel -- already has a line at every power, and would otherwise + # look like textbook PSK. + psk_line_created = f.psk_strength > f.papr_spectral + 8.0 + if const_env and f.psk_strength > 20 and f.psk_order and impulsive_phase \ + and psk_line_created and f.carrier_ratio < 0.10 \ + and not fsk_like and f.bandwidth > tone_bw: + m = f.psk_order + name = {2: "BPSK", 4: "QPSK / pi-4 DQPSK", 8: "8-PSK"}.get(m, f"{m}-PSK") + conf = 0.5 + min(0.25, (f.psk_strength - 20) / 60.0) + reason = f"x^{m} produces a spectral line ({f.psk_strength:.0f} dB)" + if f.baud: + reason += f", ~{f.baud:.0f} baud" + add(name, "psk", conf, reason, "raw") + + # -- wideband digital / noise-like ------------------------------------ + if f.flatness > 0.55 and f.bandwidth > 200_000 and f.carrier_ratio < 0.05: + add("Wideband digital carrier (OFDM/CDMA-like)", "digital", + 0.5 + min(0.25, f.flatness - 0.55), + f"flat, noise-like spectrum {f.bandwidth/1e6:.2f} MHz wide", "raw") + + # -- pulsed / radar-like ---------------------------------------------- + if f.ook_duty < 0.05 and f.ook_contrast_db > 15 and f.bandwidth > 200_000: + add("Pulsed transmission (radar / DME / Mode S-like)", "pulse", + 0.5 + min(0.2, f.ook_contrast_db / 100.0), + f"{f.ook_duty*100:.1f}% duty cycle, wide pulses", "raw") + + if not out: + add("Unidentified signal", "unknown", 0.2, + f"{f.bandwidth/1e3:.1f} kHz wide, SNR {f.snr_db:.0f} dB", "nfm") + return out + + +# --------------------------------------------------------------------------- +# Known-system refinement +# --------------------------------------------------------------------------- + +def _baud_near(f: SignalFeatures, target: float, tol: float = 0.12) -> bool: + return bool(f.baud) and abs(f.baud - target) <= tol * target + + +def _identify_system(freq_hz: float, f: SignalFeatures, + best_family: str) -> tuple[str, float, str] | None: + """Name a specific system when frequency + shape + baud all agree.""" + mhz = freq_hz / 1e6 + + def within(lo, hi): + return lo <= mhz <= hi + + # Aviation surveillance + if within(1089, 1091) and f.ook_duty < 0.2: + return ("ADS-B / Mode S (1090 MHz extended squitter)", 0.9, + "1090 MHz, pulse-position keyed bursts") + if within(977, 979): + return ("UAT ADS-B / FIS-B (978 MHz)", 0.85, "978 MHz UAT channel") + if within(960, 1215) and f.ook_duty < 0.1: + return ("DME / TACAN pulse pairs", 0.7, "pulsed navigation band") + + # Broadcast + if within(87.9, 108.1) and best_family in ("wfm", "fsk", "nfm"): + label = "FM broadcast station" + if f.stereo_pilot: + label += " (stereo)" + return (label, 0.9, "FM broadcast band, wideband FM") + if within(162.39, 162.56) and best_family == "nfm": + return ("NOAA Weather Radio", 0.9, "NWR channel, narrowband FM") + if within(136.9, 138.1) and f.bandwidth > 25_000: + return ("Weather satellite downlink (NOAA APT / Meteor)", 0.7, + "137 MHz satellite band") + + # Marine / maritime + if within(161.96, 162.04) and _baud_near(f, 9600, 0.2): + return ("AIS ship transponder (9600 GMSK)", 0.88, + "AIS channel A/B, 9600 baud GMSK") + if within(156.0, 162.1) and best_family == "nfm": + ch = _marine_channel(freq_hz) + return (f"Marine VHF voice{ch}", 0.75, "marine VHF band, narrowband FM") + + # Aviation voice / data + if within(118.0, 137.0): + if within(129.0, 137.0) and _baud_near(f, 2400, 0.2) and f.ook_duty < 0.6: + return ("ACARS datalink (2400 baud MSK)", 0.85, + "ACARS band, 2400 baud bursts") + if best_family == "am": + return ("VHF airband voice (AM)", 0.85, "118-137 MHz airband, AM") + if within(225.0, 400.0) and best_family == "am": + return ("Military UHF air voice (AM)", 0.7, "225-400 MHz UHF air band") + + # Amateur + if within(144.38, 144.40) and (f.afsk_1200 or _baud_near(f, 1200, 0.2)): + return ("APRS packet (AFSK 1200 baud)", 0.88, "144.390 MHz APRS channel") + ham_hf = any(lo <= mhz <= hi for lo, hi in ( + (1.8, 2.0), (3.5, 4.0), (5.33, 5.41), (7.0, 7.3), (10.1, 10.15), + (14.0, 14.35), (18.068, 18.168), (21.0, 21.45), (24.89, 24.99), + (28.0, 29.7))) + if ham_hf and best_family == "ssb": + return ("Amateur HF SSB voice", 0.75, "inside a US amateur HF phone band") + if ham_hf and best_family == "cw": + return ("Amateur HF CW (Morse)", 0.8, "inside a US amateur HF CW segment") + if within(26.965, 27.405) and best_family in ("am", "ssb"): + return ("CB radio (Citizens Band)", 0.75, "11 m CB channel") + if within(144.0, 148.0) and best_family == "nfm": + return ("2 m amateur FM", 0.7, "2 m band, narrowband FM") + if within(420.0, 450.0) and best_family == "nfm": + return ("70 cm amateur FM", 0.7, "70 cm band, narrowband FM") + + # Land mobile digital voice + if best_family == "fsk" and f.freq_modes == 4: + if _baud_near(f, 4800) and 8_000 < f.bandwidth < 16_000: + if f.is_bursty and 25.0 < f.burst_rate_hz < 45.0: + return ("DMR digital voice (TDMA, 4800 baud C4FM)", 0.8, + "12.5 kHz 4-FSK with ~30 ms TDMA bursts") + return ("P25 Phase 1 C4FM digital voice (4800 baud)", 0.75, + "12.5 kHz 4-level FSK at 4800 baud") + if _baud_near(f, 2400) and f.bandwidth < 8_000: + return ("NXDN digital voice (2400 baud, 6.25 kHz)", 0.72, + "6.25 kHz 4-FSK at 2400 baud") + if best_family == "fsk" and f.freq_modes <= 3: + if _baud_near(f, 4800) and f.bandwidth < 8_000: + return ("D-STAR digital voice (4800 baud GMSK)", 0.65, + "6.25 kHz GMSK at 4800 baud") + for baud, name in ((512, "POCSAG 512"), (1200, "POCSAG 1200"), + (2400, "POCSAG 2400")): + if _baud_near(f, baud) and (within(929, 932) or within(150, 160)): + return (f"{name} pager traffic", 0.75, + f"paging band, {baud} baud 2-FSK") + for baud, name in ((1600, "FLEX 1600"), (3200, "FLEX 3200"), + (6400, "FLEX 6400")): + if _baud_near(f, baud) and within(929, 932): + return (f"{name} pager traffic", 0.72, + f"900 MHz paging, {baud} baud FLEX") + + if best_family == "psk" and f.psk_order == 4 and _baud_near(f, 18000, 0.15): + return ("TETRA (pi/4-DQPSK, 18 kbaud)", 0.7, "25 kHz pi/4-DQPSK") + + # ISM / short range devices + if within(314.5, 315.5) or within(433.0, 434.9) or within(389.5, 390.5): + if best_family in ("ook", "fsk"): + kind = "OOK" if best_family == "ook" else "FSK" + return (f"ISM short-range device ({kind}: TPMS / remote / sensor)", + 0.7, f"ISM band burst, {kind}") + if within(902, 928) and f.bandwidth > 100_000: + return ("902-928 MHz ISM (FHSS / LoRa / smart meter)", 0.6, + "wideband ISM emission") + + # Time signals + if abs(mhz - 2.5) < 0.005 or abs(mhz - 5.0) < 0.005 or \ + abs(mhz - 10.0) < 0.005 or abs(mhz - 15.0) < 0.005 or \ + abs(mhz - 20.0) < 0.005: + return ("WWV/WWVH standard time and frequency station", 0.8, + "exact WWV carrier frequency") + + # Cellular + if (within(824, 894) or within(1850, 1990) or within(614, 698)) \ + and f.bandwidth > 800_000: + return ("Cellular downlink (LTE/5G-NR)", 0.65, + "wide flat carrier in a cellular allocation") + return None + + +def _marine_channel(freq_hz: float) -> str: + """Best-effort marine VHF channel label.""" + known = {156_800_000: " (ch 16 distress)", 156_650_000: " (ch 13 bridge)", + 157_100_000: " (ch 22A USCG)", 156_450_000: " (ch 9)", + 156_600_000: " (ch 12)", 156_700_000: " (ch 14)"} + for hz, name in known.items(): + if abs(freq_hz - hz) < 6_000: + return name + return "" + + +# --------------------------------------------------------------------------- +# Entry point +# --------------------------------------------------------------------------- + +def classify(x: np.ndarray, sample_rate: float, freq_hz: float = 0.0, + snr_db: float = 0.0, analyse_audio: bool = True) -> Classification: + """Identify what kind of signal ``x`` is. + + ``x`` should be complex baseband centred on the signal. ``freq_hz`` is the + real-world centre frequency and is used only for the known-system lookup. + """ + f = extract_features(x, sample_rate, snr_db=snr_db) + filtered = f.extras.pop("_filtered", x) + if analyse_audio: + try: + _analyse_fm_audio(filtered, sample_rate, f) + except Exception: + pass # audio features are a bonus, never a hard failure + + cands = _score_rules(f) + cands.sort(key=lambda c: c[2], reverse=True) + label, family, score, reason, mode = cands[0] + reasons = [reason] + + if family == "ssb": + # Region 2 convention: LSB on 160/80/40 m, USB everywhere else. + lower = freq_hz > 0 and freq_hz < 10_000_000 + mode = "lsb" if lower else "usb" + label = f"SSB voice ({mode.upper()})" + reasons.append(f"{mode.upper()} assumed from the band convention") + + system = _identify_system(freq_hz, f, family) + if system: + sys_label, sys_conf, sys_reason = system + if sys_conf >= score: + label = sys_label + score = min(0.97, 0.5 * sys_conf + 0.5 * score + 0.15) + reasons.insert(0, sys_reason) + else: + reasons.append(f"also consistent with {sys_label}") + + # Low SNR means low trust, whatever the rules said. + if f.snr_db < 8: + score *= 0.65 + reasons.append(f"low SNR ({f.snr_db:.0f} dB) -- treat with caution") + elif f.snr_db < 15: + score *= 0.85 + + alts = [(c[0], round(min(0.99, c[2]), 2)) for c in cands[1:4]] + return Classification( + label=label, family=family, confidence=round(min(0.99, score), 3), + reasons=reasons, alternatives=alts, suggested_mode=mode, features=f, + ) diff --git a/bandsaunter/cli.py b/bandsaunter/cli.py new file mode 100755 index 0000000..ff06d52 --- /dev/null +++ b/bandsaunter/cli.py @@ -0,0 +1,879 @@ +"""Command line interface.""" + +from __future__ import annotations + +import argparse +import json +import signal +import sys +import time +from pathlib import Path + +from rich.console import Console +from rich.prompt import Confirm +from rich.live import Live +from rich.panel import Panel +from rich.table import Table +from rich.text import Text + +from . import __version__ +from .bandplan import CATEGORIES, PRESETS, fmt_hz, in_category, search +from .config import (DEFAULT_CONFIG_DIR, DEFAULT_CONFIG_PATH, ScanConfig, + is_first_run, list_profiles, load_config, load_default, + save_config, save_default) +from .device import RtlSdrError, list_devices, set_driver_messages +from .librtlsdr import load_error +from . import settings as st +from .ranges import (RangeError, ScanRange, build_plan, parse_range_list) +from .scanner import Scanner, ScannerCallbacks +from .tui import TUIAbort, first_run_setup, run_tui, settings_menu +from .ui import KeyReader, ScanDisplay, print_band_table, print_hit + +console = Console() + + +# --------------------------------------------------------------------------- +# argument parsing +# --------------------------------------------------------------------------- + +def build_parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser( + prog="bandsaunter", + description="Scan, record and identify signals with an RTL-SDR.", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog="""\ +examples: + bandsaunter interactive setup + bandsaunter scan -r 144M-148M -r 420M-450M two ranges + bandsaunter scan -b gmrs -b marine-vhf band-plan presets + bandsaunter scan -b 2m --record 30 --hang 3 record 30 s max, 3 s squelch tail + bandsaunter scan -b 2m --record 0 --hang 6 whole conversations, gaps and all + bandsaunter scan -r 14.0M-14.35M --mode cw HF CW (needs direct sampling) + bandsaunter config settings menu + bandsaunter config hang_seconds=5 set one setting and save it + bandsaunter bands --category Aviation browse the US band plan + bandsaunter devices list attached dongles + bandsaunter scan -b 2m --simulate try it without hardware +""") + p.add_argument("--version", action="version", version=f"bandsaunter {__version__}") + sub = p.add_subparsers(dest="command") + + # -- scan ------------------------------------------------------------ + s = sub.add_parser("scan", help="run a scan", + formatter_class=argparse.ArgumentDefaultsHelpFormatter) + g = s.add_argument_group("what to scan") + g.add_argument("-r", "--range", action="append", default=[], metavar="SPEC", + help="frequency range, e.g. 144M-148M or 462M-468M/12.5k@nfm; " + "repeat for as many pairs as you like") + g.add_argument("-b", "--band", action="append", default=[], metavar="KEY", + help="US band plan preset key (see `bandsaunter bands`); repeatable") + g.add_argument("--mode", default=None, + choices=["auto", "nfm", "wfm", "am", "usb", "lsb", "cw", "raw"], + help="force a demodulator for every range") + g.add_argument("-p", "--profile", metavar="NAME", + help="load a saved profile") + g.add_argument("--save-profile", metavar="NAME", + help="save the resulting configuration and exit") + + st.add_arguments(s) + + g = s.add_argument_group("presentation") + g.add_argument("--plain", action="store_true", + help="line-per-hit output instead of the live display") + g.add_argument("--simulate", action="store_true", + help="use a synthetic receiver instead of real hardware") + g.add_argument("--dry-run", action="store_true", + help="show the sweep plan and exit") + g.add_argument("--no-config", action="store_true", + help="ignore the saved settings file and start from defaults") + g.add_argument("--keep-carriers", action="store_true", + help="also record steady unmodulated carriers") + g.add_argument("--settings", action="store_true", + help="open the settings menu before scanning") + g.add_argument("--save", action="store_true", + help="save the resulting settings as the default and exit") + + # -- bands ------------------------------------------------------------ + b = sub.add_parser("bands", help="browse the US band plan") + b.add_argument("term", nargs="?", help="search term") + b.add_argument("-c", "--category", help="show one category") + b.add_argument("--categories", action="store_true", help="list categories") + b.add_argument("--json", action="store_true", help="machine-readable output") + + # -- config ----------------------------------------------------------- + c = sub.add_parser("config", help="view or change the saved settings") + c.add_argument("assignment", nargs="*", metavar="KEY=VALUE", + help="set one or more settings, e.g. hang_seconds=5") + c.add_argument("--show", action="store_true", help="print every setting") + c.add_argument("--path", action="store_true", + help="print where the settings file lives") + c.add_argument("--edit", action="store_true", + help="open the settings menu") + c.add_argument("--reset", action="store_true", + help="delete the saved settings") + c.add_argument("--describe", metavar="KEY", + help="explain one setting in full") + + # -- transcribe --------------------------------------------------------- + tr = sub.add_parser("transcribe", + help="transcribe recordings, or check the recognisers") + tr.add_argument("path", nargs="*", + help="WAV files or directories of them") + tr.add_argument("--engines", action="store_true", + help="list the speech recognisers and which are installed") + tr.add_argument("--engine", default=None, metavar="NAME") + tr.add_argument("--model", default=None, metavar="NAME") + tr.add_argument("--language", default=None, metavar="CODE") + tr.add_argument("--stdout", action="store_true", + help="print instead of writing _transcription.txt files") + + # -- devices ---------------------------------------------------------- + d = sub.add_parser("devices", help="list attached RTL-SDR devices") + d.add_argument("--test", action="store_true", + help="open the device and capture a test block") + + # -- profiles ---------------------------------------------------------- + pr = sub.add_parser("profiles", help="list saved profiles") + pr.add_argument("--show", metavar="NAME", help="print one profile") + + # -- analyse ------------------------------------------------------------ + a = sub.add_parser("analyze", aliases=["analyse"], + help="identify a signal in a recorded file") + a.add_argument("path", help=".cf32/.cs16 IQ file or a .wav from a recording") + a.add_argument("--rate", type=float, help="sample rate of the file (Hz)") + a.add_argument("--freq", type=float, default=0.0, + help="centre frequency in Hz, for band-aware naming") + a.add_argument("--morse", action="store_true", help="force a CW decode") + return p + + +# --------------------------------------------------------------------------- +# helpers +# --------------------------------------------------------------------------- + +def _build_config(args) -> tuple[ScanConfig, Path | None]: + """Layer the configuration: saved settings, then a profile, then flags. + + Returns the config and where its saved settings came from, so the user can + be told what is in force. + """ + source = None + if getattr(args, "no_config", False): + cfg = ScanConfig() + else: + cfg, source = load_default() + if args.profile: + cfg = load_config(args.profile) + source = Path(cfg._source_path) if hasattr(cfg, "_source_path") else None + + ranges: list[ScanRange] = [] + for spec in args.range: + ranges.extend(parse_range_list(spec)) + for key in args.band: + ranges.extend(parse_range_list(key)) + if ranges: + cfg.ranges = ranges + if args.mode and args.mode != "auto": + for r in cfg.ranges: + r.mode = args.mode + + # Everything else comes straight off the shared settings table, so a flag + # cannot exist without the matching entry in the in-app menu. + st.apply_args(cfg, args) + + if getattr(args, "keep_carriers", False) and "carrier" not in cfg.accept: + cfg.accept = list(cfg.accept) + ["carrier"] + return cfg, source + + +def _print_plan(cfg: ScanConfig) -> None: + try: + steps = build_plan(cfg.ranges, cfg.sample_rate, cfg.usable_fraction, + dc_guard=cfg.dc_guard_hz) + except RangeError as exc: + console.print(f"[red]{exc}[/red]") + return + t = Table(title="sweep plan", box=None, header_style="bold") + t.add_column("range") + t.add_column("from", justify="right") + t.add_column("to", justify="right") + t.add_column("mode", justify="center") + t.add_column("steps", justify="right") + counts: dict[int, int] = {} + for s in steps: + counts[s.range_index] = counts.get(s.range_index, 0) + 1 + for i, r in enumerate(cfg.ranges): + if not r.enabled: + continue + t.add_row(r.label, fmt_hz(r.start), fmt_hz(r.stop), r.mode, + str(counts.get(i, 0))) + console.print(t) + cycle = len(steps) * cfg.dwell_seconds + total = sum(r.span for r in cfg.ranges if r.enabled) + gate = (", ".join(cfg.accept) if cfg.require_signal + else "everything above the squelch (content check off)") + combine = "" + if cfg.combine_by_frequency: + combine = "\nOne file per frequency" + if cfg.announce_timestamps: + from .announce import available_engine + chosen = (available_engine() if cfg.announce_engine == "auto" + else cfg.announce_engine) + using = (f"{chosen}" if chosen and chosen != "builtin" + else "the built-in synthesiser") + combine += f", timestamps spoken by {using}" + combine += "." + + console.print( + f"[grey62]{len(steps)} tuner steps, {fmt_hz(total)} of spectrum, " + f"about {cycle:.1f} s per sweep (excluding time spent recording)." + f"\nSquelch +{cfg.threshold_db:g} dB; " + + (f"record up to {cfg.record_seconds:g} s per signal" + if cfg.record_seconds else "record for as long as the signal lasts") + + f"; resume after {cfg.hang_seconds:g} s of quiet " + f"(shorter gaps are recorded through)." + f"\nRecording: {gate}.{combine}[/grey62]") + + +def _maybe_first_run(cfg: ScanConfig, args) -> None: + """Ask where to save, the first time, when there is someone to ask. + + Skipped when the settings file is being ignored, when nothing is going to + be written, and when input is not a terminal -- a script must never block + on a question. + """ + if not is_first_run() or getattr(args, "no_config", False): + return + if getattr(args, "dry_run", False) or getattr(args, "save_profile", None): + return + if not (sys.stdin.isatty() and sys.stdout.isatty()): + return + try: + first_run_setup(console, cfg) + except TUIAbort: + console.print() + + +def _make_device(cfg: ScanConfig, simulate: bool): + if simulate: + from .simulator import SimulatedDevice + console.print("[magenta]Using the built-in simulator " + "(no hardware involved).[/magenta]") + return SimulatedDevice(sample_rate=cfg.sample_rate, realtime=True).open() + from .device import RtlSdrDevice + dev = RtlSdrDevice(index=cfg.device_index, sample_rate=cfg.sample_rate, + gain=cfg.gain, ppm=cfg.ppm, agc=cfg.agc, + bias_tee=cfg.bias_tee, + offset_tuning=cfg.offset_tuning, + direct_sampling=cfg.direct_sampling) + return dev.open() + + +# --------------------------------------------------------------------------- +# commands +# --------------------------------------------------------------------------- + +def cmd_scan(args) -> int: + try: + cfg, source = _build_config(args) + except (RangeError, FileNotFoundError, ValueError, st.SettingError) as exc: + console.print(f"[red]{exc}[/red]") + return 2 + + _maybe_first_run(cfg, args) + + if getattr(args, "settings", False): + try: + settings_menu(console, cfg) + except TUIAbort: + return 0 + + if not cfg.ranges: + cfg = run_tui(console, cfg, source) + if cfg is None: + return 0 + elif source and not cfg.quiet: + console.print(f"[grey62]settings from {source}[/grey62]") + + if args.save_profile: + path = save_config(cfg, args.save_profile) + console.print(f"[green]saved profile to {path}[/green]") + return 0 + if getattr(args, "save", False): + path = save_default(cfg) + console.print(f"[green]saved as the default settings: {path}[/green]") + return 0 + + errs = cfg.validate() + if errs: + for e in errs: + console.print(f"[red]{e}[/red]") + return 2 + + if args.dry_run: + _print_plan(cfg) + return 0 + + if not cfg.quiet: + _print_plan(cfg) + + try: + device = _make_device(cfg, args.simulate) + except RtlSdrError as exc: + console.print(Panel(Text(str(exc)), title="[red]cannot open the receiver", + border_style="red")) + console.print("[grey62]Try `bandsaunter devices` to check what is " + "attached, or `--simulate` to run without " + "hardware.[/grey62]") + return 1 + + scanner = Scanner(cfg, device=device, callbacks=ScannerCallbacks()) + try: + scanner.prepare() + except (ValueError, RangeError, RtlSdrError) as exc: + console.print(f"[red]{exc}[/red]") + device.close() + return 2 + + signal.signal(signal.SIGINT, lambda *a: scanner.stop()) + rc = (_run_plain(scanner, cfg) if (args.plain or cfg.quiet or + not sys.stdout.isatty()) + else _run_live(scanner, cfg)) + _print_summary(scanner) + return rc + + +def _run_plain(scanner: Scanner, cfg: ScanConfig) -> int: + scanner.cb.on_record_end = lambda hit: print_hit(console, hit) + if not cfg.quiet: + scanner.cb.on_status = lambda m: console.print(f"[grey62]{m}[/grey62]") + scanner.cb.on_error = lambda e: console.print(f"[red]{type(e).__name__}: {e}[/red]") + console.print("[grey62]scanning -- Ctrl-C to stop[/grey62]") + try: + scanner.run() + except KeyboardInterrupt: + scanner.stop() + return 0 + + +def _run_live(scanner: Scanner, cfg: ScanConfig) -> int: + display = ScanDisplay(scanner) + display.attach() + import threading + worker = threading.Thread(target=scanner.run, daemon=True, name="scan") + + with KeyReader() as keys: + # crop rather than let an oversized frame scroll: a display taller + # than the terminal cannot be redrawn in place, and every refresh + # would leave another copy behind. + with Live(display.render(), console=console, refresh_per_second=8, + screen=False, transient=False, + vertical_overflow="crop") as live: + worker.start() + try: + while worker.is_alive(): + key = keys.get() + if key: + _handle_key(key, scanner, display) + live.update(display.render()) + time.sleep(0.1) + except KeyboardInterrupt: + scanner.stop() + finally: + scanner.stop() + worker.join(timeout=5.0) + live.update(display.render()) + return 0 + + +def _handle_key(key: str, scanner: Scanner, display: ScanDisplay) -> None: + k = key.lower() + if k == "q": + scanner.stop() + elif k == "p": + scanner.pause(not scanner.paused) + display.on_status("paused" if scanner.paused else "resumed") + elif k == "s": + scanner.skip() + display.on_status("skipping this signal") + elif k == "l": + freq = display._rec.frequency or scanner.stats.current_freq + if freq: + scanner.lockout(freq) + display.on_status(f"locked out {fmt_hz(freq)}") + scanner.skip() + elif k in ("+", "="): + scanner.cfg.threshold_db += 1.0 + display.on_status(f"squelch +{scanner.cfg.threshold_db:g} dB") + elif k == "-": + scanner.cfg.threshold_db = max(1.0, scanner.cfg.threshold_db - 1.0) + display.on_status(f"squelch +{scanner.cfg.threshold_db:g} dB") + + +def _print_summary(scanner: Scanner) -> None: + st = scanner.stats + console.print() + console.rule("[bold]scan summary[/bold]", style="blue") + console.print( + f" ran for {st.elapsed:.0f} s over {st.cycles} sweep(s), " + f"{st.steps_done} tuner steps\n" + f" {st.detections} detection(s), {st.recordings} recording(s) kept, " + f"{st.discarded} discarded\n" + f" {st.seconds_recorded:.0f} s of audio captured") + if st.truncated: + limit = scanner.cfg.record_seconds + console.print( + f"[yellow] {st.truncated} recording(s) were cut off at the " + f"{limit:g} s record limit while the signal was still " + f"transmitting.[/yellow]\n" + f"[grey62] Set 'Record for' to 0 (or --record 0) to follow a " + f"transmission to its end; 'Wait for quiet' then decides when to " + f"move on.[/grey62]") + worker = getattr(scanner, "transcriber", None) + if worker is not None and (worker.written or worker.empty or worker.dropped): + bits = [f"{worker.written} transcript(s) written"] + if worker.empty: + bits.append(f"{worker.empty} with no recognisable speech") + if worker.dropped: + bits.append(f"{worker.dropped} skipped, the recogniser fell behind") + console.print(f"[grey62] {', '.join(bits)}[/grey62]") + if st.rejected_by_category: + drops = ", ".join(f"{n} {cat}" + for cat, n in sorted(st.rejected_by_category.items(), + key=lambda kv: -kv[1]) if cat) + if drops: + console.print(f"[grey62] discarded without recording: {drops}[/grey62]") + if scanner.hits: + counts: dict[str, int] = {} + for h in scanner.hits: + counts[h.classification or "unclassified"] = \ + counts.get(h.classification or "unclassified", 0) + 1 + t = Table(box=None, header_style="bold") + t.add_column("identified as") + t.add_column("count", justify="right") + for label, n in sorted(counts.items(), key=lambda kv: -kv[1]): + t.add_row(label, str(n)) + console.print(t) + out = Path(scanner.cfg.output_dir) + console.print(f"[grey62]recordings in {out.resolve()}, " + f"log in {(out / scanner.cfg.log_file).name} " + f"and .csv[/grey62]") + + +def cmd_config(args) -> int: + cfg, source = load_default() + + if args.path: + print(DEFAULT_CONFIG_PATH) + return 0 + if args.reset: + if DEFAULT_CONFIG_PATH.exists(): + DEFAULT_CONFIG_PATH.unlink() + console.print(f"[green]deleted {DEFAULT_CONFIG_PATH}[/green]") + else: + console.print("[yellow]nothing saved to delete[/yellow]") + return 0 + if args.describe: + setting = st.by_key(args.describe) or next(iter(st.search(args.describe)), None) + if setting is None: + console.print(f"[red]no setting called {args.describe!r}[/red]") + return 1 + from .tui import setting_help + setting_help(console, setting, cfg) + return 0 + + if args.assignment: + changed = [] + for item in args.assignment: + key, _, value = item.partition("=") + setting = st.by_key(key.strip()) + if setting is None: + console.print(f"[red]no setting called {key.strip()!r}[/red]") + matches = st.search(key.strip()) + if matches: + console.print("[grey62]did you mean: " + + ", ".join(m.key for m in matches[:5]) + + "[/grey62]") + return 2 + try: + setattr(cfg, setting.key, st.parse_value(setting, value)) + except st.SettingError as exc: + console.print(f"[red]{setting.key}: {exc}[/red]") + return 2 + changed.append(setting) + errs = [e for e in cfg.validate() if "frequency ranges" not in e] + if errs: + for e in errs: + console.print(f"[red]{e}[/red]") + return 2 + path = save_default(cfg) + for setting in changed: + console.print(f"[green]{setting.key} = " + f"{st.format_value(setting, getattr(cfg, setting.key))}" + f"[/green]") + console.print(f"[grey62]saved to {path}[/grey62]") + return 0 + + if args.edit or not args.show: + try: + settings_menu(console, cfg) + except TUIAbort: + console.print() + return 0 + try: + save = Confirm.ask("save these settings as the default", + default=True) + except (EOFError, KeyboardInterrupt): + save = False + if save: + console.print(f"[green]saved to {save_default(cfg)}[/green]") + return 0 + + default = ScanConfig() + console.print(f"[grey62]{'saved settings: ' + str(source) if source else 'no settings file yet; showing defaults'}[/grey62]") + for group in st.GROUPS: + t = Table(title=group, box=None, header_style="bold", title_justify="left") + t.add_column("key", style="cyan") + t.add_column("value") + t.add_column("default", style="grey62") + t.add_column("what it does", style="grey62", overflow="fold") + for setting in st.in_group(group): + value = st.format_value(setting, getattr(cfg, setting.key)) + dflt = st.format_value(setting, getattr(default, setting.key)) + t.add_row(setting.key, value, "" if value == dflt else dflt, + setting.help) + console.print(t) + console.print() + return 0 + + +def cmd_bands(args) -> int: + if args.categories: + t = Table(title="band plan categories", box=None, header_style="bold") + t.add_column("category") + t.add_column("presets", justify="right") + for c in CATEGORIES: + t.add_row(c, str(len(in_category(c)))) + console.print(t) + return 0 + if args.category: + presets = in_category(args.category) + if not presets: + matches = [c for c in CATEGORIES + if args.category.lower() in c.lower()] + if len(matches) == 1: + presets = in_category(matches[0]) + else: + console.print(f"[red]no such category: {args.category}[/red]") + console.print("[grey62]try: " + ", ".join(CATEGORIES) + "[/grey62]") + return 2 + title = args.category + elif args.term: + presets = search(args.term) + title = f"matching {args.term!r}" + if not presets: + console.print(f"[yellow]nothing matched {args.term!r}[/yellow]") + return 1 + else: + presets = list(PRESETS) + title = f"US band plan ({len(PRESETS)} presets)" + + if args.json: + print(json.dumps([{ + "key": p.key, "name": p.name, "category": p.category, + "start": p.start, "stop": p.stop, "step": p.step, + "mode": p.mode, "bandwidth": p.bandwidth, "note": p.note, + "members": list(p.members), + } for p in presets], indent=2)) + return 0 + print_band_table(console, presets, title) + console.print("[grey62]use a key with: bandsaunter scan -b [/grey62]") + return 0 + + +def cmd_transcribe(args) -> int: + from .transcribe import available_engine, describe_engines, transcribe + from .recorder import read_wav + + if args.engines or not args.path: + t = Table(title="speech recognisers", box=None, header_style="bold") + t.add_column("engine", style="cyan") + t.add_column("installed") + t.add_column("how to get it", style="grey62") + for name, present, how in describe_engines(): + t.add_row(name, + "[green]yes[/green]" if present else "[red]no[/red]", how) + console.print(t) + chosen = available_engine() + console.print(f"[grey62]{'auto would use ' + chosen if chosen else + 'nothing installed — transcription is unavailable'}" + f"[/grey62]") + if not args.path: + return 0 if chosen else 1 + + cfg, _ = load_default() + engine = args.engine or cfg.transcribe_engine + model = args.model or cfg.transcribe_model + language = args.language if args.language is not None \ + else cfg.transcribe_language + + files: list[Path] = [] + for item in args.path: + p = Path(item) + if p.is_dir(): + files.extend(sorted(p.glob("*.wav"))) + elif p.exists(): + files.append(p) + else: + console.print(f"[red]no such file: {p}[/red]") + return 1 + if not files: + console.print("[yellow]nothing to transcribe[/yellow]") + return 1 + + failures = 0 + for wav in files: + try: + audio, rate = read_wav(wav) + except (OSError, ValueError) as exc: + console.print(f"[red]{wav.name}: {exc}[/red]") + failures += 1 + continue + console.print(f"[grey62]{wav.name}: {audio.size / rate:.1f} s[/grey62]") + result = transcribe(audio, rate, engine, model, language) + if result is None: + console.print("[red]no speech recogniser installed — " + "see `bandsaunter transcribe --engines`[/red]") + return 1 + if result.note: + console.print(f"[yellow]{result.note}[/yellow]") + text = result.text.strip() or "[no speech recognised]" + if args.stdout: + console.print(text) + else: + out = wav.with_suffix("") + out = out.with_name(out.name + "_transcription.txt") + out.write_text(text + "\n") + console.print(f" [green]{out.name}[/green]: {text[:70]}") + return 1 if failures else 0 + + +def cmd_devices(args) -> int: + # This is the command people run when something is wrong, so let the + # driver say what it is doing. + set_driver_messages(True) + err = load_error() + if err: + console.print(Panel(Text(err), title="[red]librtlsdr not available", + border_style="red")) + return 1 + devs = list_devices() + if not devs: + console.print("[yellow]No RTL-SDR devices found.[/yellow]") + console.print( + "[grey62]Check `lsusb` for a Realtek RTL2832/RTL2838. If it is " + "listed, the DVB-T kernel driver has probably claimed it:\n" + " echo 'blacklist dvb_usb_rtl28xxu' | " + "sudo tee /etc/modprobe.d/blacklist-rtl.conf\n" + " sudo rmmod dvb_usb_rtl28xxu[/grey62]") + return 1 + t = Table(title="RTL-SDR devices", box=None, header_style="bold") + t.add_column("#", justify="right") + t.add_column("name") + t.add_column("manufacturer") + t.add_column("product") + t.add_column("serial") + for d in devs: + t.add_row(str(d.index), d.name, d.manufacturer, d.product, d.serial) + console.print(t) + + if args.test: + from .device import RtlSdrDevice + import numpy as np + for d in devs: + try: + with RtlSdrDevice(index=d.index) as dev: + dev.tune(100_000_000) + x = dev.read_samples(65536, flush=True) + rms = float(np.sqrt(np.mean(np.abs(x) ** 2))) + gains = dev.available_gains + console.print( + f" [green]device {d.index} works[/green]: tuner " + f"{dev.tuner}, {len(gains)} gain steps " + f"({min(gains):.1f}-{max(gains):.1f} dB), " + f"test capture RMS {rms:.4f}") + except RtlSdrError as exc: + console.print(f" [red]device {d.index}: {exc}[/red]") + return 1 + return 0 + + +def cmd_profiles(args) -> int: + if args.show: + try: + cfg = load_config(args.show) + except (OSError, FileNotFoundError) as exc: + console.print(f"[red]{exc}[/red]") + return 1 + console.print_json(json.dumps(cfg.to_dict(), default=str)) + return 0 + profiles = list_profiles() + if not profiles: + console.print(f"[yellow]no profiles in {DEFAULT_CONFIG_DIR}[/yellow]") + console.print("[grey62]create one with: " + "bandsaunter scan -b 2m --save-profile myscan[/grey62]") + return 0 + t = Table(title=f"profiles in {DEFAULT_CONFIG_DIR}", box=None, + header_style="bold") + t.add_column("name") + t.add_column("ranges", justify="right") + t.add_column("record", justify="right") + t.add_column("hang", justify="right") + for p in profiles: + try: + cfg = load_config(str(p)) + t.add_row(p.stem, str(len(cfg.ranges)), + f"{cfg.record_seconds:g}s", f"{cfg.hang_seconds:g}s") + except Exception: + t.add_row(p.stem, "[red]unreadable[/red]", "", "") + console.print(t) + return 0 + + +def cmd_analyze(args) -> int: + import numpy as np + from .classify import classify + from .morse import decode_morse + + path = Path(args.path) + if not path.exists(): + console.print(f"[red]no such file: {path}[/red]") + return 1 + + rate = args.rate + freq = args.freq + # A recording directory carries its own metadata; use it when present. + meta_file = path.parent / "meta.json" + if meta_file.exists(): + try: + meta = json.loads(meta_file.read_text()) + freq = freq or meta.get("frequency_hz", 0.0) + if rate is None: + rate = (meta.get("iq_rate") if path.suffix != ".wav" + else meta.get("audio_rate")) + except (OSError, ValueError): + pass + + if path.suffix == ".wav": + import wave + with wave.open(str(path)) as w: + rate = rate or w.getframerate() + raw = w.readframes(w.getnframes()) + audio = np.frombuffer(raw, dtype=" int: + parser = build_parser() + args = parser.parse_args(argv) + + if args.command is None: + cfg, source = load_default() + if is_first_run() and sys.stdin.isatty() and sys.stdout.isatty(): + try: + first_run_setup(console, cfg) + except TUIAbort: + console.print() + return 0 + cfg = run_tui(console, cfg, source) + if cfg is None: + return 0 + try: + device = _make_device(cfg, False) + except RtlSdrError as exc: + console.print(Panel(Text(str(exc)), + title="[red]cannot open the receiver", + border_style="red")) + return 1 + scanner = Scanner(cfg, device=device, callbacks=ScannerCallbacks()) + try: + scanner.prepare() + except (ValueError, RangeError, RtlSdrError) as exc: + console.print(f"[red]{exc}[/red]") + return 2 + signal.signal(signal.SIGINT, lambda *a: scanner.stop()) + _run_live(scanner, cfg) + _print_summary(scanner) + return 0 + + handlers = { + "scan": cmd_scan, "bands": cmd_bands, "devices": cmd_devices, + "config": cmd_config, "transcribe": cmd_transcribe, + "profiles": cmd_profiles, "analyze": cmd_analyze, "analyse": cmd_analyze, + } + try: + return handlers[args.command](args) + except KeyboardInterrupt: + console.print("\n[grey62]interrupted[/grey62]") + return 130 + except BrokenPipeError: + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bandsaunter/config.py b/bandsaunter/config.py new file mode 100755 index 0000000..662cfc5 --- /dev/null +++ b/bandsaunter/config.py @@ -0,0 +1,277 @@ +"""Scanner configuration: defaults, validation, and YAML profiles.""" + +from __future__ import annotations + +import os +from dataclasses import dataclass, field, asdict, fields +from pathlib import Path + +import yaml + +from .ranges import ScanRange, parse_range_list + +__all__ = ["ScanConfig", "DEFAULT_CONFIG_DIR", "DEFAULT_CONFIG_PATH", + "load_config", "save_config", "list_profiles", "profile_path", + "load_default", "save_default", "delete_profile", + "is_first_run", "DEFAULT_OUTPUT_DIR"] + +DEFAULT_CONFIG_DIR = Path( + os.environ.get("BANDSAUNTER_CONFIG_DIR", + Path.home() / ".config" / "bandsaunter") +) + +# Settings saved here are picked up by every run, before any command-line +# option is applied. Named profiles live alongside it in the same directory. +DEFAULT_CONFIG_PATH = DEFAULT_CONFIG_DIR / "config.yaml" + +# Suggested on first run. Somewhere visible rather than under a dot +# directory: these are recordings people want to browse and play. +DEFAULT_OUTPUT_DIR = "~/bandsaunter" + + +def is_first_run(directory: Path | None = None) -> bool: + """True when no settings have ever been saved.""" + return not (Path(directory or DEFAULT_CONFIG_DIR) / "config.yaml").exists() + + +@dataclass +class ScanConfig: + """Everything that controls a scan run. + + The two dwell settings the scanner is built around: + + ``record_seconds`` + Record for at most this long on one signal before going back to + scanning, even if the signal is still up. 0 means "no limit", which + is what you want for capturing a whole conversation. + + ``hang_seconds`` + How long the channel must stay quiet before the transmission is + treated as over. Gaps shorter than this are recorded straight + through, so a two-way exchange stays in one file across the natural + pauses between overs. "Quiet" means no real signal: silence, static + and interference all count towards it, so a burst of noise during a + pause does not hold the receiver on a finished conversation. + """ + + # -- what to scan --------------------------------------------------- + ranges: list[ScanRange] = field(default_factory=list) + + # -- the two headline dwell settings -------------------------------- + record_seconds: float = 30.0 # max record time per hit (0 = unlimited) + hang_seconds: float = 2.0 # quiet time before the scan resumes + max_record_seconds: float = 900.0 # absolute ceiling, even when record is 0 + + # -- detection ------------------------------------------------------ + threshold_db: float = 10.0 # dB of margin over the noise + min_record_seconds: float = 0.5 # discard blips shorter than this + dwell_seconds: float = 0.05 # capture time per tuner step while sweeping + probe_seconds: float = 0.4 # look before choosing a demodulator + resolution_hz: float = 3_000.0 # FFT bin width during the sweep + squelch_margin_db: float = 3.0 # hysteresis: drop below thr - this to end + adaptive_floor: bool = True # smooth the per-bin noise baseline + detector: str = "peak" # "peak" holds bursts, "avg" is quieter + detector_bias_db: float | None = None # None = derive it from the detector + + # -- content gate: only keep captures that carry something ---------- + require_signal: bool = True # discard captures with no real content + accept: list[str] = field( + default_factory=lambda: ["voice", "cw", "digital"]) + min_signal_score: float = 0.45 # confidence needed to keep a capture + min_voice_score: float = 0.45 # speech-likeness needed to call it voice + verify_seconds: float = 1.5 # how often to re-check a live capture + verify_max_seconds: float = 6.0 # give up on a contentless capture by here + max_detections_per_step: int = 4 + + # -- radio ---------------------------------------------------------- + device_index: int = 0 + sample_rate: int = 2_048_000 + gain: float | str = "auto" + ppm: int = 0 + agc: bool = False + bias_tee: bool = False + offset_tuning: bool = False + direct_sampling: int | str = "auto" + usable_fraction: float = 0.75 + dc_guard_hz: float = 8_000.0 # LO offset that keeps the DC spike clear + + # -- recording ------------------------------------------------------ + output_dir: str = "~/bandsaunter" + save_audio: bool = True + save_iq: bool = False + audio_rate: int = 16_000 + iq_format: str = "cf32" # cf32 or cs16 + classify: bool = True + decode_morse: bool = True + + # -- one file per frequency ------------------------------------------ + combine_by_frequency: bool = False + combine_tolerance_hz: float = 6_250.0 + combine_keep_individual: bool = False + announce_timestamps: bool = True + announce_frequency: bool = False + announce_engine: str = "auto" + + # -- speech to text --------------------------------------------------- + transcribe: bool = False + transcribe_engine: str = "auto" + transcribe_model: str = "base.en" + transcribe_language: str = "en" + transcribe_min_seconds: float = 1.0 + + # -- behaviour ------------------------------------------------------ + lockout: list[float] = field(default_factory=list) + lockout_width: float = 12_500.0 + revisit_seconds: float = 8.0 # ignore a frequency again this soon + max_cycles: int = 0 # 0 = run forever + max_runtime_seconds: float = 0.0 # 0 = no limit + quiet: bool = False + log_file: str = "scan_log.jsonl" + + # ------------------------------------------------------------------ + def validate(self) -> list[str]: + """Return a list of human-readable problems (empty when config is sane).""" + errs = [] + if not self.ranges: + errs.append("no frequency ranges configured") + if self.record_seconds < 0: + errs.append("record_seconds cannot be negative") + if self.hang_seconds < 0: + errs.append("hang_seconds cannot be negative") + if self.min_record_seconds < 0: + errs.append("min_record_seconds cannot be negative") + if self.max_record_seconds < 0: + errs.append("max_record_seconds cannot be negative") + if self.record_seconds and self.min_record_seconds > self.record_seconds: + errs.append("min_record_seconds is longer than record_seconds, so " + "no recording would ever be kept") + if not (0.005 <= self.dwell_seconds <= 5.0): + errs.append("dwell_seconds should be between 0.005 and 5") + if self.threshold_db <= 0: + errs.append("threshold_db must be positive") + if not (0.1 <= self.usable_fraction <= 0.95): + errs.append("usable_fraction should be between 0.1 and 0.95") + if self.audio_rate < 4000: + errs.append("audio_rate is too low to carry voice") + from .quality import CATEGORIES + bad = [c for c in self.accept if c not in CATEGORIES] + if bad: + errs.append("unknown accept categories: " + ", ".join(bad) + + " (choose from " + ", ".join(CATEGORIES) + ")") + if self.require_signal and not self.accept: + errs.append("require_signal is on but no categories are accepted, " + "so nothing could ever be recorded") + if not (0.0 <= self.min_signal_score <= 1.0): + errs.append("min_signal_score must be between 0 and 1") + if self.detector not in ("peak", "avg"): + errs.append('detector must be "peak" or "avg"') + if self.iq_format not in ("cf32", "cs16"): + errs.append("iq_format must be cf32 or cs16") + for r in self.ranges: + if r.stop <= 0 or r.start <= 0: + errs.append(f"range {r.label!r} has a non-positive frequency") + return errs + + # -- serialisation --------------------------------------------------- + def to_dict(self) -> dict: + d = asdict(self) + d["ranges"] = [r.to_dict() for r in self.ranges] + return d + + @classmethod + def from_dict(cls, d: dict) -> "ScanConfig": + d = dict(d or {}) + raw = d.pop("ranges", []) + known = {f.name for f in fields(cls)} + unknown = set(d) - known + for k in unknown: + d.pop(k) + cfg = cls(**{k: v for k, v in d.items() if k in known}) + out = [] + for item in raw: + if isinstance(item, dict): + out.append(ScanRange.from_dict(item)) + elif isinstance(item, str): + out.extend(parse_range_list(item)) + cfg.ranges = out + if unknown: + cfg._unknown_keys = sorted(unknown) # surfaced by the CLI + return cfg + + def copy(self) -> "ScanConfig": + return ScanConfig.from_dict(self.to_dict()) + + +def profile_path(name: str, directory: Path | None = None) -> Path: + directory = Path(directory or DEFAULT_CONFIG_DIR) + if name.endswith((".yaml", ".yml")): + p = Path(name) + return p if p.is_absolute() or p.parent != Path(".") else directory / p.name + return directory / f"{name}.yaml" + + +def save_config(cfg: ScanConfig, name: str, directory: Path | None = None) -> Path: + path = profile_path(name, directory) + path.parent.mkdir(parents=True, exist_ok=True) + with open(path, "w") as fh: + yaml.safe_dump(cfg.to_dict(), fh, sort_keys=False, default_flow_style=False) + return path + + +def load_config(name_or_path: str, directory: Path | None = None) -> ScanConfig: + path = Path(name_or_path) + if not path.exists(): + path = profile_path(name_or_path, directory) + if not path.exists(): + raise FileNotFoundError(f"no such profile or config file: {name_or_path}") + with open(path) as fh: + data = yaml.safe_load(fh) or {} + cfg = ScanConfig.from_dict(data) + cfg._source_path = str(path) + return cfg + + +def list_profiles(directory: Path | None = None) -> list[Path]: + directory = Path(directory or DEFAULT_CONFIG_DIR) + if not directory.exists(): + return [] + return sorted(p for p in directory.glob("*.yaml")) + + +def load_default(directory: Path | None = None) -> tuple["ScanConfig", Path | None]: + """Load the saved settings, or defaults if none have been saved. + + Returns ``(config, path)`` where ``path`` is None when nothing was loaded, + so the caller can tell the user where its settings came from. + """ + path = Path(directory or DEFAULT_CONFIG_DIR) / "config.yaml" + if not path.exists(): + return ScanConfig(), None + try: + with open(path) as fh: + data = yaml.safe_load(fh) or {} + except (OSError, yaml.YAMLError): + return ScanConfig(), None + cfg = ScanConfig.from_dict(data) + cfg._source_path = str(path) + return cfg, path + + +def save_default(cfg: "ScanConfig", directory: Path | None = None) -> Path: + """Write the settings that every later run should start from.""" + path = Path(directory or DEFAULT_CONFIG_DIR) / "config.yaml" + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(".yaml.tmp") + with open(tmp, "w") as fh: + yaml.safe_dump(cfg.to_dict(), fh, sort_keys=False, + default_flow_style=False) + tmp.replace(path) # atomic, so a crash cannot leave a half file + return path + + +def delete_profile(name: str, directory: Path | None = None) -> bool: + path = profile_path(name, directory) + if path.exists(): + path.unlink() + return True + return False diff --git a/bandsaunter/demod.py b/bandsaunter/demod.py new file mode 100755 index 0000000..5a992c7 --- /dev/null +++ b/bandsaunter/demod.py @@ -0,0 +1,333 @@ +"""Streaming demodulators. + +Each demodulator consumes consecutive blocks of complex baseband at the +device sample rate and emits float32 audio in [-1, 1]. All internal filters +are stateful, so a recording assembled from many blocks has no seams. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass + +import numpy as np +from scipy import signal as sps + +from .dsp import DecimationChain, Mixer + +__all__ = ["make_demodulator", "Demodulator", "MODES", "plan_rates", + "DEEMPHASIS_US"] + +MODES = ("nfm", "wfm", "am", "usb", "lsb", "cw", "dsb", "raw") + +DEEMPHASIS_US = 75e-6 # North America; Europe uses 50 us + + +def _small_prime_divisors(n: int, limit: int = 8192) -> list[int]: + """Divisors of ``n`` built only from 2/3/5 -- the ones that decimate cleanly.""" + out = {1} + for p in (2, 3, 5): + cur = set(out) + for v in cur: + m = v + while m * p <= limit and n % (m * p) == 0: + m *= p + out.add(m) + return sorted(out) + + +def plan_rates(sample_rate: int, bandwidth: float, audio_rate: int, + min_if_factor: float = 2.5) -> tuple[int, int, int, int]: + """Choose IF and audio decimation for one demod chain. + + Returns ``(if_decim, audio_decim, if_rate, actual_audio_rate)``. The IF + rate is kept at least ``min_if_factor`` times the signal bandwidth so the + discriminator is not clipped by its own anti-alias filter. + """ + sample_rate = int(sample_rate) + divisors = _small_prime_divisors(sample_rate) + + total = min(divisors, key=lambda d: abs(sample_rate / d - audio_rate)) + actual_audio = int(round(sample_rate / total)) + + need_if = max(bandwidth * min_if_factor, actual_audio * 2.0) + if_decim = 1 + for d in divisors: + if d > total: + break + if sample_rate / d >= need_if and total % d == 0: + if_decim = d + audio_decim = max(1, total // if_decim) + if_rate = int(round(sample_rate / if_decim)) + return if_decim, audio_decim, if_rate, actual_audio + + +class _DCBlock: + """One-pole DC blocker, keeps voice from riding on a slow offset.""" + + def __init__(self, r: float = 0.995): + self.r = r + self._x1 = 0.0 + self._y1 = 0.0 + + def __call__(self, x: np.ndarray) -> np.ndarray: + y, zf = sps.lfilter([1.0, -1.0], [1.0, -self.r], x, + zi=[self.r * self._y1 - self._x1]) + self._x1 = float(x[-1]) if x.size else self._x1 + self._y1 = float(y[-1]) if y.size else self._y1 + return y + + +class _AGC: + """Slow level control so quiet and loud transmissions land at similar volume.""" + + def __init__(self, target: float = 0.25, attack: float = 0.02, + release: float = 0.002, max_gain: float = 200.0): + self.target = target + self.attack = attack + self.release = release + self.max_gain = max_gain + self.gain = 1.0 + + def __call__(self, x: np.ndarray) -> np.ndarray: + if x.size == 0: + return x + level = float(np.sqrt(np.mean(x.astype(np.float64) ** 2))) + 1e-9 + want = min(self.max_gain, self.target / level) + a = self.attack if want < self.gain else self.release + self.gain = (1.0 - a) * self.gain + a * want + return x * self.gain + + +@dataclass +class Demodulator: + """Base class -- subclasses implement :meth:`_demod`.""" + + sample_rate: int + bandwidth: float + audio_rate: int = 16_000 + mode: str = "raw" + + def __post_init__(self): + self.if_decim, self.audio_decim, self.if_rate, self.audio_rate = plan_rates( + self.sample_rate, self.bandwidth, self.audio_rate, + min_if_factor=self._min_if_factor() + ) + self.if_chain = DecimationChain(self.if_decim, complex_input=True) + self.audio_chain = DecimationChain(self.audio_decim, complex_input=False) + self.dc = _DCBlock() + self.agc = _AGC() + self._setup() + + def _min_if_factor(self) -> float: + return 2.5 + + def _setup(self) -> None: + pass + + def reset(self) -> None: + self.if_chain.reset() + self.audio_chain.reset() + self.dc = _DCBlock() + self.agc = _AGC() + self._setup() + + # -- pipeline ------------------------------------------------------ + def to_if(self, x: np.ndarray) -> np.ndarray: + """Complex baseband at the IF rate -- also what the classifier sees.""" + return self.if_chain(x) + + def _demod(self, x: np.ndarray) -> np.ndarray: + raise NotImplementedError + + def _post(self, audio: np.ndarray) -> np.ndarray: + return audio + + def step(self, x: np.ndarray) -> tuple[np.ndarray, np.ndarray]: + """Return ``(audio, if_iq)``. + + The recorder wants both: audio to write, and the IF-rate complex + baseband to hand to the classifier. + """ + empty_a = np.zeros(0, dtype=np.float32) + empty_i = np.zeros(0, dtype=np.complex64) + if x.size == 0: + return empty_a, empty_i + iq = self.to_if(x) + if iq.size == 0: + return empty_a, empty_i + audio = self._demod(iq) + audio = self._post(audio) + audio = self.audio_chain(audio) + if audio.size == 0: + return empty_a, iq + audio = self.dc(audio) + audio = self.agc(audio) + return np.clip(audio, -1.0, 1.0).astype(np.float32), iq + + def process(self, x: np.ndarray) -> np.ndarray: + return self.step(x)[0] + + +class FMDemod(Demodulator): + """Quadrature discriminator with optional de-emphasis.""" + + deemphasis: float | None = DEEMPHASIS_US + + def _min_if_factor(self) -> float: + return 2.0 + + def _setup(self) -> None: + self._last = np.complex64(0) + tau = getattr(self, "deemphasis", DEEMPHASIS_US) + if tau: + a = math.exp(-1.0 / (tau * self.if_rate)) + self._de_b = [1.0 - a] + self._de_a = [1.0, -a] + self._de_zi = np.zeros(1) + else: + self._de_b = None + + def _demod(self, x: np.ndarray) -> np.ndarray: + prev = np.empty(x.size, dtype=np.complex64) + prev[0] = self._last + prev[1:] = x[:-1] + self._last = x[-1] + # Scale so full deviation maps to roughly +-1. + gain = self.if_rate / (2.0 * math.pi * max(1.0, self.bandwidth / 2.0)) + return np.angle(x * np.conj(prev)).astype(np.float64) * gain + + def _post(self, audio: np.ndarray) -> np.ndarray: + if self._de_b is None: + return audio + out, self._de_zi = sps.lfilter(self._de_b, self._de_a, audio, zi=self._de_zi) + return out + + +class NFMDemod(FMDemod): + pass + + +class WFMDemod(FMDemod): + def _min_if_factor(self) -> float: + return 1.4 # 180 kHz signal in a 256 kHz IF is normal for broadcast FM + + +class AMDemod(Demodulator): + """Envelope detector. Works for airband, shortwave and CB AM.""" + + def _demod(self, x: np.ndarray) -> np.ndarray: + return np.abs(x).astype(np.float64) + + +class SSBDemod(Demodulator): + """Single-sideband via a one-sided complex band-pass, then take the real part.""" + + upper: bool = True + + def _setup(self) -> None: + self._zi = None + self._taps = None + + def _sideband_taps(self): + if self._taps is None: + lo, hi = (200.0, self.bandwidth) if getattr(self, "upper", True) \ + else (-self.bandwidth, -200.0) + centre = 0.5 * (lo + hi) + half = 0.5 * abs(hi - lo) + proto = sps.firwin(129, min(half / (self.if_rate / 2.0), 0.99)) + n = np.arange(proto.size) - proto.size // 2 + self._taps = (proto * np.exp(2j * math.pi * centre * n / self.if_rate)) + self._zi = np.zeros(self._taps.size - 1, dtype=np.complex128) + return self._taps + + def _demod(self, x: np.ndarray) -> np.ndarray: + taps = self._sideband_taps() + y, self._zi = sps.lfilter(taps, [1.0], x, zi=self._zi) + return 2.0 * y.real + + +class USBDemod(SSBDemod): + upper = True + + +class LSBDemod(SSBDemod): + upper = False + + +class CWDemod(SSBDemod): + """SSB with a narrow filter and a beat-frequency offset, so CW is audible.""" + + upper = True + bfo_hz: float = 700.0 + + def _setup(self) -> None: + self._zi = None + self._taps = None + self._mixer = Mixer(-getattr(self, "bfo_hz", 700.0), self.if_rate) + + def _sideband_taps(self): + if self._taps is None: + bfo = getattr(self, "bfo_hz", 700.0) + half = max(150.0, self.bandwidth / 2.0) + lo, hi = bfo - half, bfo + half + centre = 0.5 * (lo + hi) + proto = sps.firwin(255, min((0.5 * (hi - lo)) / (self.if_rate / 2.0), 0.99)) + n = np.arange(proto.size) - proto.size // 2 + self._taps = (proto * np.exp(2j * math.pi * centre * n / self.if_rate)) + self._zi = np.zeros(self._taps.size - 1, dtype=np.complex128) + return self._taps + + def _demod(self, x: np.ndarray) -> np.ndarray: + # Shift the (near-DC) carrier up to the BFO pitch before filtering. + x = self._mixer(x) + taps = self._sideband_taps() + y, self._zi = sps.lfilter(taps, [1.0], x, zi=self._zi) + return 2.0 * y.real + + +class DSBDemod(Demodulator): + """Coherent-ish DSB / product detector; also a usable fallback for raw IQ.""" + + def _demod(self, x: np.ndarray) -> np.ndarray: + return x.real.astype(np.float64) * 2.0 + + +class RawDemod(Demodulator): + """No demodulation -- magnitude only, so 'listening' to data still shows activity.""" + + def _demod(self, x: np.ndarray) -> np.ndarray: + return np.abs(x).astype(np.float64) + + +_CLASSES = { + "nfm": NFMDemod, + "wfm": WFMDemod, + "fm": NFMDemod, + "am": AMDemod, + "usb": USBDemod, + "lsb": LSBDemod, + "ssb": USBDemod, + "cw": CWDemod, + "dsb": DSBDemod, + "raw": RawDemod, +} + + +# Broadcast FM carries content well past the 8 kHz a 16 kHz audio rate can +# hold, so it gets a higher floor than the narrowband modes. +_MIN_AUDIO_RATE = {"wfm": 32_000} + + +def make_demodulator(mode: str, sample_rate: int, bandwidth: float, + audio_rate: int = 16_000, **kw) -> Demodulator: + mode = (mode or "nfm").lower() + audio_rate = max(audio_rate, _MIN_AUDIO_RATE.get(mode, 0)) + cls = _CLASSES.get(mode, RawDemod) + obj = cls(sample_rate=sample_rate, bandwidth=bandwidth, + audio_rate=audio_rate, mode=mode) + for k, v in kw.items(): + setattr(obj, k, v) + if kw: + obj.reset() + return obj diff --git a/bandsaunter/device.py b/bandsaunter/device.py new file mode 100755 index 0000000..da1349a --- /dev/null +++ b/bandsaunter/device.py @@ -0,0 +1,480 @@ +"""High level RTL-SDR device control built on the ctypes binding.""" + +from __future__ import annotations + +import contextlib +import ctypes +import math +import os +import threading +import time +from collections import deque +from dataclasses import dataclass, field + +import numpy as np + +from ._quiet import suppress_stderr +from .librtlsdr import READ_ASYNC_CB, LibRtlSdr, RtlSdrError + +__all__ = ["RtlSdrDevice", "DeviceInfo", "list_devices", "RtlSdrError", + "quiet_driver", "set_driver_messages", + "R820T_MIN_HZ", "R820T_MAX_HZ", "DIRECT_SAMPLING_MAX_HZ"] + +# Practical tuning limits for the common R820T/R828D front ends. Anything +# below this needs direct sampling (bypassing the tuner entirely). +R820T_MIN_HZ = 24_000_000 +R820T_MAX_HZ = 1_766_000_000 +DIRECT_SAMPLING_MAX_HZ = 28_800_000 # usable Q-branch span at 2.4 MSPS-ish clock + +# The RTL2832U only locks these sample-rate windows. +_RATE_WINDOWS = ((225_001, 300_000), (900_001, 3_200_000)) + +# librtlsdr wants read sizes that are a multiple of 512 bytes; 16 kB blocks are +# the size its own async reader uses and behave well over USB. +_READ_GRANULE = 512 + + +# librtlsdr writes its own messages ("Found Rafael Micro R820T tuner", +# "Allocating 15 zero-copy buffers", "[R82XX] PLL not locked!") straight to +# file descriptor 2 from C. Those land in the middle of the live display and +# break its cursor tracking, so the header ends up drawn several times over. +# Set BANDSAUNTER_DRIVER_MESSAGES=1 to see them while debugging. +_SHOW_DRIVER_MESSAGES = os.environ.get("BANDSAUNTER_DRIVER_MESSAGES", "") not in \ + ("", "0", "no", "false") + + +def set_driver_messages(show: bool) -> None: + """Let the driver's own messages through again. + + They are worth seeing while diagnosing a device -- "Detached kernel + driver", the tuner it found -- and only a nuisance during the live + display, which they draw straight over. + """ + global _SHOW_DRIVER_MESSAGES + _SHOW_DRIVER_MESSAGES = bool(show) + + +@contextlib.contextmanager +def quiet_driver(): + """Silence the driver's own chatter for the duration of a C call. + + Only around the call itself, so a Python traceback on stderr is never + swallowed: the window is a few microseconds of library code. + """ + with suppress_stderr(not _SHOW_DRIVER_MESSAGES): + yield + + +@dataclass +class DeviceInfo: + index: int + name: str + manufacturer: str = "?" + product: str = "?" + serial: str = "?" + + def describe(self) -> str: + return (f"[{self.index}] {self.name} " + f"(sn: {self.serial})") + + +def list_devices() -> list[DeviceInfo]: + """Enumerate attached RTL-SDR devices (empty list if none / no driver).""" + try: + lib = LibRtlSdr() + except RtlSdrError: + return [] + out = [] + with quiet_driver(): + for i in range(lib.device_count()): + manu, prod, serial = lib.usb_strings(i) + out.append(DeviceInfo(i, lib.device_name(i), manu, prod, serial)) + return out + + +def clamp_sample_rate(rate: int) -> int: + """Snap a requested sample rate into a window the RTL2832U can actually lock.""" + rate = int(rate) + for lo, hi in _RATE_WINDOWS: + if lo <= rate <= hi: + return rate + # Choose the nearest legal edge. + edges = [e for w in _RATE_WINDOWS for e in w] + return min(edges, key=lambda e: abs(e - rate)) + + +@dataclass +class RtlSdrDevice: + """Owns one dongle and serialises all access to it. + + The scanner retunes constantly, so this class keeps track of the settling + cost of each retune and flushes the USB pipeline so that samples handed + back are guaranteed to have been captured *after* the tune completed. + """ + + index: int = 0 + sample_rate: int = 2_048_000 + gain: float | str = "auto" # dB, or "auto" for the tuner AGC + ppm: int = 0 + agc: bool = False # RTL2832 digital AGC + bias_tee: bool = False + offset_tuning: bool = False + direct_sampling: int | str = "auto" # 0/1/2 or "auto" (enable below 24 MHz) + settle_seconds: float = 0.006 + + _lib: LibRtlSdr | None = field(default=None, init=False, repr=False) + _dev: ctypes.c_void_p | None = field(default=None, init=False, repr=False) + _buf: ctypes.Array | None = field(default=None, init=False, repr=False) + _buf_len: int = field(default=0, init=False, repr=False) + _ds_mode: int = field(default=0, init=False, repr=False) + _center: int = field(default=0, init=False, repr=False) + _gains: list[float] = field(default_factory=list, init=False, repr=False) + tuner: str = field(default="unknown", init=False) + retunes: int = field(default=0, init=False) + samples_read: int = field(default=0, init=False) + _streaming: bool = field(default=False, init=False, repr=False) + _stream_thread: object = field(default=None, init=False, repr=False) + _stream_cb: object = field(default=None, init=False, repr=False) + _stream_chunks: object = field(default=None, init=False, repr=False) + _stream_lock: object = field(default=None, init=False, repr=False) + _stream_ready: object = field(default=None, init=False, repr=False) + _stream_error: object = field(default=None, init=False, repr=False) + _stream_have: int = field(default=0, init=False, repr=False) + _stream_max: int = field(default=0, init=False, repr=False) + _stream_dropped: int = field(default=0, init=False, repr=False) + _partial: object = field(default=None, init=False, repr=False) + + # -- lifecycle ----------------------------------------------------- + def open(self) -> "RtlSdrDevice": + self._lib = LibRtlSdr() + n = self._lib.device_count() + if n == 0: + raise RtlSdrError( + "no RTL-SDR device found.\n" + "Check `lsusb` for a Realtek RTL2832/2838, make sure the DVB-T " + "kernel module is blacklisted, and that you have permission to " + "open the USB device." + ) + if self.index >= n: + raise RtlSdrError(f"device index {self.index} out of range (found {n})") + + with quiet_driver(): + self._dev = self._lib.open(self.index) + self.tuner = self._lib.get_tuner_type(self._dev) + self._gains = self._lib.get_tuner_gains(self._dev) + + self.sample_rate = clamp_sample_rate(self.sample_rate) + self._lib.set_sample_rate(self._dev, self.sample_rate) + self.sample_rate = (self._lib.get_sample_rate(self._dev) + or self.sample_rate) + + if self.ppm: + self._lib.set_freq_correction(self._dev, self.ppm) + self._lib.set_agc_mode(self._dev, self.agc) + if self.bias_tee: + self._lib.set_bias_tee(self._dev, True) + if self.offset_tuning: + self._lib.set_offset_tuning(self._dev, True) + + self.apply_gain(self.gain) + self._lib.reset_buffer(self._dev) + return self + + def close(self) -> None: + self.stop_stream() + if self._dev is not None and self._lib is not None: + try: + if self.bias_tee: + self._lib.set_bias_tee(self._dev, False) + except RtlSdrError: + pass + with quiet_driver(): + self._lib.close(self._dev) + self._dev = None + + def __enter__(self): + return self.open() + + def __exit__(self, *exc): + self.close() + return False + + @property + def is_open(self) -> bool: + return self._dev is not None + + # -- configuration ------------------------------------------------- + @property + def available_gains(self) -> list[float]: + return list(self._gains) + + def apply_gain(self, gain: float | str) -> None: + """Set tuner gain. ``"auto"`` hands control to the tuner AGC.""" + self.gain = gain + if isinstance(gain, str) and gain.lower() in ("auto", "agc", ""): + self._lib.set_tuner_gain_mode(self._dev, False) + return + self._lib.set_tuner_gain_mode(self._dev, True) + target = float(gain) + if self._gains: + target = min(self._gains, key=lambda g: abs(g - target)) + self._lib.set_tuner_gain(self._dev, target) + + @property + def current_gain(self) -> float | str: + if isinstance(self.gain, str): + return "auto" + try: + return self._lib.get_tuner_gain(self._dev) + except RtlSdrError: + return self.gain + + def set_sample_rate(self, rate: int) -> int: + rate = clamp_sample_rate(rate) + if rate != self.sample_rate: + with quiet_driver(): + self._lib.set_sample_rate(self._dev, rate) + self.sample_rate = self._lib.get_sample_rate(self._dev) or rate + self._lib.reset_buffer(self._dev) + return self.sample_rate + + def _wants_direct_sampling(self, hz: float) -> int: + if self.direct_sampling == "auto": + return 2 if hz < R820T_MIN_HZ else 0 + return int(self.direct_sampling) + + def _set_direct_sampling(self, mode: int) -> None: + if mode == self._ds_mode: + return + with quiet_driver(): + ok = self._lib.set_direct_sampling(self._dev, mode) + if ok: + self._ds_mode = mode + # Switching the signal path invalidates whatever is in flight. + self._lib.reset_buffer(self._dev) + elif mode: + raise RtlSdrError( + "this dongle does not support direct sampling, so frequencies " + f"below {R820T_MIN_HZ/1e6:.0f} MHz cannot be tuned" + ) + + def tune(self, hz: float, settle: bool = True) -> int: + """Tune the front end. Returns the frequency the hardware reports.""" + hz = int(round(hz)) + mode = self._wants_direct_sampling(hz) + self._set_direct_sampling(mode) + + if mode == 0 and not (R820T_MIN_HZ <= hz <= R820T_MAX_HZ): + raise RtlSdrError( + f"{hz/1e6:.4f} MHz is outside the tuner's range " + f"({R820T_MIN_HZ/1e6:.0f}-{R820T_MAX_HZ/1e6:.0f} MHz)" + ) + + with quiet_driver(): + self._lib.set_center_freq(self._dev, hz) + self._center = self._lib.get_center_freq(self._dev) or hz + self.retunes += 1 + if settle and self.settle_seconds > 0: + time.sleep(self.settle_seconds) + return self._center + + @property + def center_freq(self) -> int: + return self._center + + @property + def direct_sampling_mode(self) -> int: + """0 = normal tuner path, 1 = I branch, 2 = Q branch.""" + return self._ds_mode + + def can_tune(self, hz: float) -> bool: + mode = self._wants_direct_sampling(hz) + if mode: + return 0 < hz <= DIRECT_SAMPLING_MAX_HZ + return R820T_MIN_HZ <= hz <= R820T_MAX_HZ + + # -- capture ------------------------------------------------------- + def _ensure_buffer(self, n_bytes: int) -> ctypes.Array: + if self._buf is None or self._buf_len < n_bytes: + self._buf = (ctypes.c_ubyte * n_bytes)() + self._buf_len = n_bytes + return self._buf + + def flush(self) -> None: + """Drop anything the RTL2832 already queued (call right after a retune).""" + self._lib.reset_buffer(self._dev) + + def read_samples(self, count: int, flush: bool = False) -> np.ndarray: + """Read ``count`` complex samples, returned as complex64 in [-1, 1). + + ``flush=True`` discards the in-flight USB buffer first so the samples + are known to post-date the most recent tune. + """ + if self._dev is None: + raise RtlSdrError("device is not open") + if flush: + self._lib.reset_buffer(self._dev) + + n_bytes = int(count) * 2 + # Round up to the USB granule; librtlsdr short-reads otherwise. + n_bytes = int(math.ceil(n_bytes / _READ_GRANULE) * _READ_GRANULE) + buf = self._ensure_buffer(n_bytes) + + got = self._lib.read_sync(self._dev, buf, n_bytes) + if got <= 0: + raise RtlSdrError("USB read returned no data") + + raw = np.frombuffer(buf, dtype=np.uint8, count=got) + if raw.size % 2: + raw = raw[:-1] + # 127.4 rather than 127.5 matches the RTL2832's actual DC bias. + iq = raw.astype(np.float32) + iq -= 127.4 + iq *= np.float32(1.0 / 128.0) + out = iq.view(np.complex64) if iq.flags.c_contiguous else iq.copy().view(np.complex64) + out = out[:count] + self.samples_read += out.size + return out + + def read_seconds(self, seconds: float, flush: bool = False) -> np.ndarray: + return self.read_samples(int(self.sample_rate * seconds), flush=flush) + + # -- continuous streaming -------------------------------------------- + # Synchronous reads lose whatever the dongle sends between calls: the USB + # host only moves data while a transfer is outstanding, so every + # microsecond spent demodulating is a microsecond of samples thrown away. + # The recording then holds less than it should and plays back too fast. + # Streaming keeps a ring of transfers queued so the capture is gapless. + + def start_stream(self, buf_num: int = 15, buf_len: int = 262144, + max_seconds: float = 4.0) -> None: + if self._streaming: + return + if self._dev is None: + raise RtlSdrError("device is not open") + self._stream_chunks = deque() + self._stream_have = 0 + self._stream_max = int(max_seconds * self.sample_rate * 2) + self._stream_dropped = 0 + self._stream_lock = threading.Lock() + self._stream_ready = threading.Event() + self._stream_error = None + self._partial = np.zeros(0, dtype=np.complex64) + + def _on_buffer(buf_ptr, length, _ctx): + try: + raw = np.ctypeslib.as_array(buf_ptr, shape=(int(length),)).copy() + except Exception: + return + with self._stream_lock: + if self._stream_have + raw.size > self._stream_max: + # The consumer has fallen behind for real; drop the oldest + # so the capture stays current, and record that it happened. + while self._stream_chunks and \ + self._stream_have + raw.size > self._stream_max: + old = self._stream_chunks.popleft() + self._stream_have -= old.size + self._stream_dropped += old.size + self._stream_chunks.append(raw) + self._stream_have += raw.size + self._stream_ready.set() + + self._stream_cb = READ_ASYNC_CB(_on_buffer) + + def _run(): + try: + with quiet_driver(): + self._lib.read_async(self._dev, self._stream_cb, + buf_num, buf_len) + except RtlSdrError as exc: + self._stream_error = exc + self._stream_ready.set() + + with quiet_driver(): + self._lib.reset_buffer(self._dev) + self._streaming = True + self._stream_thread = threading.Thread(target=_run, daemon=True, + name="rtlsdr-stream") + self._stream_thread.start() + + def stop_stream(self) -> None: + if not self._streaming: + return + self._streaming = False + try: + with quiet_driver(): + self._lib.cancel_async(self._dev) + except Exception: + pass + if self._stream_thread is not None: + self._stream_thread.join(timeout=2.0) + self._stream_thread = None + self._stream_cb = None + + @property + def streaming(self) -> bool: + return self._streaming + + @property + def dropped_samples(self) -> int: + return self._stream_dropped // 2 + + def read_stream(self, count: int, timeout: float = 3.0) -> np.ndarray: + """Pull ``count`` complex samples from the running stream.""" + if not self._streaming: + return self.read_samples(count) + need_bytes = int(count) * 2 + deadline = time.time() + timeout + parts = [] + have = 0 + if self._partial.size: + parts.append(self._partial) + have += self._partial.size * 2 + self._partial = np.zeros(0, dtype=np.complex64) + + while have < need_bytes: + if self._stream_error is not None: + raise self._stream_error + with self._stream_lock: + chunk = self._stream_chunks.popleft() if self._stream_chunks else None + if chunk is not None: + self._stream_have -= chunk.size + elif not self._stream_chunks: + self._stream_ready.clear() + if chunk is None: + if time.time() > deadline: + raise RtlSdrError("timed out waiting for samples") + self._stream_ready.wait(0.2) + continue + parts.append(self._to_complex(chunk)) + have += chunk.size + + out = np.concatenate([p if p.dtype == np.complex64 else p for p in parts]) + if out.size > count: + self._partial = out[count:] + out = out[:count] + self.samples_read += out.size + return out + + @staticmethod + def _to_complex(raw: np.ndarray) -> np.ndarray: + if raw.size % 2: + raw = raw[:-1] + iq = raw.astype(np.float32) + iq -= 127.4 + iq *= np.float32(1.0 / 128.0) + return iq.view(np.complex64) + + # -- diagnostics --------------------------------------------------- + def status(self) -> dict: + return { + "index": self.index, + "tuner": self.tuner, + "sample_rate": self.sample_rate, + "center_freq": self._center, + "gain": self.current_gain, + "ppm": self.ppm, + "direct_sampling": self._ds_mode, + "retunes": self.retunes, + "samples_read": self.samples_read, + } diff --git a/bandsaunter/dsp.py b/bandsaunter/dsp.py new file mode 100755 index 0000000..3d10e0d --- /dev/null +++ b/bandsaunter/dsp.py @@ -0,0 +1,443 @@ +"""Signal processing primitives shared by the sweeper and the recorder. + +Everything here is stateless unless it has to be; the pieces that run across +consecutive blocks of a live capture (decimators, filters, FM discriminator) +carry explicit state so block boundaries do not produce clicks. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass, field + +import numpy as np +from scipy import signal as sps +from scipy import ndimage as ndi + +__all__ = [ + "welch_psd", "psd_dbfs", "noise_floor", "noise_floor_curve", + "detector_bias_db", + "NoiseFloorTracker", + "FIRDecimator", "DecimationChain", "design_decimation", + "frequency_shift", "Mixer", "bandpass_complex", "occupied_bandwidth", + "spectral_flatness", "instantaneous_frequency", "envelope", + "goertzel", "db", "next_fast_len", "estimate_snr", +] + +_EPS = 1e-20 + + +def db(x): + return 10.0 * np.log10(np.maximum(np.asarray(x, dtype=np.float64), _EPS)) + + +def next_fast_len(n: int) -> int: + """Round up to a power of two (fast and predictable for our block sizes).""" + return 1 << int(math.ceil(math.log2(max(2, n)))) + + +# --------------------------------------------------------------------------- +# Spectrum estimation +# --------------------------------------------------------------------------- + +def welch_psd(x: np.ndarray, nfft: int, overlap: float = 0.5, + window: str = "hann", max_segments: int = 64, + combine: str = "mean"): + """Periodogram of a complex baseband block. + + Returns ``(freqs_normalised, power_linear)`` with the spectrum shifted so + index 0 is the most negative frequency. ``freqs_normalised`` runs over + [-0.5, 0.5) and is scaled by the caller's sample rate. + + ``combine="mean"`` averages the segments, which gives the lowest-variance + estimate of a steady signal. ``combine="max"`` keeps the strongest value + seen in each bin instead -- essential while sweeping, because a keyed or + bursty transmission (CW, packet, a short over) is off for part of the + dwell and averaging buries it in the noise. + """ + x = np.asarray(x) + nfft = int(nfft) + if x.size < nfft: + pad = np.zeros(nfft, dtype=np.complex64) + pad[: x.size] = x + x = pad + + step = max(1, int(nfft * (1.0 - overlap))) + n_seg = 1 + (x.size - nfft) // step + if n_seg > max_segments: + # Spread the segments across the whole block rather than truncating. + idx = np.linspace(0, x.size - nfft, max_segments).astype(np.int64) + n_seg = max_segments + else: + idx = np.arange(n_seg, dtype=np.int64) * step + + win = sps.get_window(window, nfft).astype(np.float32) + norm = np.sum(win.astype(np.float64) ** 2) + + peak_hold = combine == "max" + acc = np.zeros(nfft, dtype=np.float64) + for start in idx: + seg = x[start:start + nfft] * win + spec = np.fft.fft(seg, nfft) + mag2 = spec.real.astype(np.float64) ** 2 + spec.imag.astype(np.float64) ** 2 + if peak_hold: + np.maximum(acc, mag2, out=acc) + else: + acc += mag2 + acc /= norm if peak_hold else (len(idx) * norm) + + power = np.fft.fftshift(acc) + freqs = np.fft.fftshift(np.fft.fftfreq(nfft, d=1.0)) + return freqs, power + + +def psd_dbfs(x: np.ndarray, nfft: int, **kw): + """Welch PSD expressed in dB relative to a full-scale sine.""" + freqs, power = welch_psd(x, nfft, **kw) + return freqs, db(power) + + +def noise_floor(psd_db: np.ndarray, percentile: float = 30.0) -> float: + """Robust noise-floor estimate: a low percentile is immune to strong carriers.""" + if psd_db.size == 0: + return -120.0 + return float(np.percentile(psd_db, percentile)) + + +def estimate_snr(psd_db: np.ndarray, floor_db: float | None = None) -> float: + if floor_db is None: + floor_db = noise_floor(psd_db) + return float(np.max(psd_db) - floor_db) + + +def noise_floor_curve(psd_db: np.ndarray, window: int = 0, + percentile: float = 25.0) -> np.ndarray: + """Per-bin noise floor that follows the receiver passband shape. + + A sliding low percentile along the frequency axis tracks the analogue + filter roll-off and any tilt in the band, while stepping straight over + signals: even if half the window is occupied, the 25th percentile still + lands in the noise. Unlike a learned average this has no warm-up sweep + and never hides a station just because it is always transmitting. + """ + psd_db = np.asarray(psd_db, dtype=np.float64) + n = psd_db.size + if n < 8: + return np.full_like(psd_db, float(np.median(psd_db))) + if window <= 0: + window = max(9, n // 4) + window = min(window, n) | 1 + return ndi.percentile_filter(psd_db, percentile, size=window, mode="nearest") + + +def detector_bias_db(n_segments: int, n_bins: int, + combine: str = "max") -> float: + """How far noise alone rides above the measured floor, in dB. + + Peak-hold keeps the largest value each bin reached across the dwell, so a + noise-only bin sits well above the floor that a percentile estimates -- + and a threshold quoted against that floor would fire on static. The + offset is a property of the detector, not of the band: for a periodogram + bin of Gaussian noise (exponentially distributed power), holding the max + of ``n_segments`` and then looking for the largest of ``n_bins`` gives a + predictable excess over the 25th percentile. + + Deliberately not measured from the data. A spread estimated from the + spectrum is inflated wherever signals are dense -- across the broadcast FM + band it reads five times higher than on empty spectrum, which would + suppress exactly the stations the scan is looking for. + """ + if combine != "max" or n_segments < 2: + return 0.0 + k = float(max(2, n_segments)) + m = float(max(8, n_bins)) + p25 = -math.log1p(-0.25 ** (1.0 / k)) # 25th percentile of max-of-k + tail = math.log(k * m) # roughly the largest of m bins + ideal = 10.0 * math.log10(max(tail / p25, 1.0)) + # Real captures fall short of the independence the maths assumes -- the + # segments overlap and the window correlates neighbouring bins -- so + # measured noise runs a few dB above the ideal figure. + return float(min(12.0, ideal + 3.0)) + + +@dataclass +class NoiseFloorTracker: + """Smoothed per-bin noise floor for one tuning step. + + The floor for each sweep comes from :func:`noise_floor_curve`; this class + only averages it across sweeps so the threshold does not jitter with the + randomness of a single short capture. Because the curve already excludes + signals, averaging it cannot make a real transmission disappear. + """ + + alpha: float = 0.3 # weight of each new sweep + window: int = 0 + percentile: float = 25.0 + floor: np.ndarray | None = field(default=None, init=False) + sweeps: int = field(default=0, init=False) + + def update(self, psd_db: np.ndarray) -> np.ndarray: + curve = noise_floor_curve(psd_db, self.window, self.percentile) + if self.floor is None or self.floor.shape != curve.shape: + self.floor = curve + else: + self.floor = (1.0 - self.alpha) * self.floor + self.alpha * curve + self.sweeps += 1 + return self.floor + + def excess(self, psd_db: np.ndarray) -> np.ndarray: + """dB above the noise floor.""" + if self.floor is None: + return np.zeros_like(psd_db) + return np.asarray(psd_db, dtype=np.float64) - self.floor + + +# --------------------------------------------------------------------------- +# Decimation / filtering with state +# --------------------------------------------------------------------------- + +class FIRDecimator: + """Stateful low-pass + integer decimation for complex or real streams. + + Keeps the filter delay line between calls, so feeding it consecutive blocks + of a live capture produces exactly the same result as filtering the whole + capture at once. + + Only the samples that survive decimation are computed. Filtering at the + input rate and then throwing away ``factor - 1`` of every ``factor`` + outputs does ``factor`` times the necessary work, and at 2 MS/s that alone + was enough to stop the capture keeping up with real time -- which shows up + as recordings that play too fast, because the samples the host could not + keep up with are dropped by the driver. + """ + + def __init__(self, factor: int, taps_per_phase: int = 12, + cutoff: float = 0.45, complex_input: bool = True, + window: str = "hamming"): + self.factor = int(factor) + if self.factor < 1: + raise ValueError("decimation factor must be >= 1") + self.complex_input = bool(complex_input) + if self.factor == 1: + self.taps = None + return + ntaps = max(15, int(taps_per_phase * self.factor) | 1) + self.taps = sps.firwin(ntaps, cutoff / self.factor, + window=window).astype(np.float32) + # Reversed once, so each output is a plain dot product. + self._rtaps = np.ascontiguousarray(self.taps[::-1]) + self._dtype = np.complex64 if complex_input else np.float32 + self._tail = np.zeros(ntaps - 1, dtype=self._dtype) + self._phase = 0 + + def reset(self) -> None: + if self.taps is not None: + self._tail[:] = 0 + self._phase = 0 + + def __call__(self, x: np.ndarray) -> np.ndarray: + if self.factor == 1: + return x + x = np.asarray(x, dtype=self._dtype) + ntaps = self._rtaps.size + buf = np.concatenate((self._tail, x)) + # y[n] = dot(buf[n : n + ntaps], reversed_taps) reproduces exactly what + # filtering the whole stream would give, with the tail carrying the + # delay line across the block boundary. + start = (-self._phase) % self.factor + self._tail = buf[-(ntaps - 1):].copy() if ntaps > 1 else self._tail + self._phase = (self._phase + x.size) % self.factor + if start >= x.size: + return np.zeros(0, dtype=self._dtype) + windows = np.lib.stride_tricks.sliding_window_view(buf, ntaps) + return windows[start:x.size:self.factor] @ self._rtaps + + +def design_decimation(factor: int, max_stage: int = 8) -> list[int]: + """Split a decimation factor into cascadable stages (cheapest first).""" + factor = int(factor) + if factor <= 1: + return [] + stages = [] + remaining = factor + for p in (8, 5, 4, 3, 2): + while remaining % p == 0 and p <= max_stage: + stages.append(p) + remaining //= p + if remaining > 1: + stages.append(remaining) + stages.sort(reverse=True) + return stages + + +class DecimationChain: + """Cascade of :class:`FIRDecimator` stages for a large total factor.""" + + def __init__(self, factor: int, complex_input: bool = True, + taps_per_phase: int = 12): + self.factor = int(factor) + self.stages = [ + FIRDecimator(f, taps_per_phase=taps_per_phase, + complex_input=complex_input) + for f in design_decimation(self.factor) + ] + + def reset(self) -> None: + for s in self.stages: + s.reset() + + def __call__(self, x: np.ndarray) -> np.ndarray: + for s in self.stages: + if x.size == 0: + return x + x = s(x) + return x + + +class Mixer: + """Continuous-phase complex mixer (digital LO) for frequency translation. + + A shift of exactly a quarter of the sample rate is special-cased: the + rotation is then the four-step cycle 1, -j, -1, +j, so no trigonometry is + needed at all. The scanner always offsets its local oscillator by exactly + that, to keep the DC spike off the signal, so this is the path that runs. + """ + + _QUARTER = np.array([1.0 + 0.0j, -1.0j, -1.0 + 0.0j, 1.0j], + dtype=np.complex64) + + def __init__(self, shift_hz: float, sample_rate: float): + self.sample_rate = float(sample_rate) + self.shift_hz = float(shift_hz) + self._phase = 0.0 + self._n = 0 + self._cache: dict[tuple[int, int], np.ndarray] = {} + + def reset(self) -> None: + self._phase = 0.0 + self._n = 0 + + def retune(self, shift_hz: float) -> None: + self.shift_hz = float(shift_hz) + self._cache.clear() + + @property + def _is_quarter_rate(self) -> bool: + return abs(abs(self.shift_hz) * 4.0 - self.sample_rate) < 1e-6 + + def _quarter_pattern(self, n: int, phase: int) -> np.ndarray: + key = (n, phase) + pat = self._cache.get(key) + if pat is None: + idx = (np.arange(n, dtype=np.int64) + phase) & 3 + rot = self._QUARTER if self.shift_hz > 0 else self._QUARTER.conj() + pat = rot[idx] + if len(self._cache) < 8: + self._cache[key] = pat + return pat + + def __call__(self, x: np.ndarray) -> np.ndarray: + if self.shift_hz == 0.0: + return x + n = x.size + if self._is_quarter_rate: + out = x * self._quarter_pattern(n, self._n & 3) + self._n = (self._n + n) & 3 + return out.astype(np.complex64, copy=False) + step = -2.0 * math.pi * self.shift_hz / self.sample_rate + ph = self._phase + step * np.arange(n, dtype=np.float64) + self._phase = float((self._phase + step * n) % (2.0 * math.pi)) + return (x * np.exp(1j * ph)).astype(np.complex64) + + +def frequency_shift(x: np.ndarray, shift_hz: float, sample_rate: float) -> np.ndarray: + """One-shot version of :class:`Mixer` for offline blocks.""" + n = np.arange(x.size, dtype=np.float64) + lo = np.exp(-2j * math.pi * shift_hz * n / float(sample_rate)) + return (x * lo).astype(np.complex64) + + +def bandpass_complex(x: np.ndarray, sample_rate: float, + low_hz: float, high_hz: float, ntaps: int = 129) -> np.ndarray: + """Asymmetric (single-sideband capable) complex band-pass. + + Designs a real low-pass of the target width and shifts it to sit between + ``low_hz`` and ``high_hz``, which may both be positive (USB) or both + negative (LSB). + """ + centre = 0.5 * (low_hz + high_hz) + half = max(1.0, 0.5 * abs(high_hz - low_hz)) + nyq = sample_rate / 2.0 + cutoff = min(half / nyq, 0.99) + proto = sps.firwin(ntaps | 1, cutoff).astype(np.float64) + n = np.arange(proto.size) - proto.size // 2 + taps = proto * np.exp(2j * math.pi * centre * n / sample_rate) + return sps.lfilter(taps, [1.0], x).astype(np.complex64) + + +# --------------------------------------------------------------------------- +# Measurements used by the classifier +# --------------------------------------------------------------------------- + +def occupied_bandwidth(psd_lin: np.ndarray, bin_hz: float, + fraction: float = 0.99) -> tuple[float, float]: + """Return ``(bandwidth_hz, centre_offset_hz)`` holding ``fraction`` of power. + + The centre offset is measured from the middle bin, so it doubles as a + frequency-error estimate for retuning onto a detected signal. + """ + p = np.asarray(psd_lin, dtype=np.float64) + if p.size == 0 or not np.any(p > 0): + return 0.0, 0.0 + total = p.sum() + csum = np.cumsum(p) + lo_t = total * (1.0 - fraction) / 2.0 + hi_t = total * (1.0 - (1.0 - fraction) / 2.0) + lo = int(np.searchsorted(csum, lo_t)) + hi = int(np.searchsorted(csum, hi_t)) + hi = min(hi, p.size - 1) + bw = max(bin_hz, (hi - lo + 1) * bin_hz) + + idx = np.arange(p.size, dtype=np.float64) + centroid = float((idx * p).sum() / total) + offset = (centroid - (p.size - 1) / 2.0) * bin_hz + return bw, offset + + +def spectral_flatness(psd_lin: np.ndarray) -> float: + """Wiener entropy: ~1 for noise-like/wideband digital, ~0 for a pure tone.""" + p = np.asarray(psd_lin, dtype=np.float64) + p = p[p > 0] + if p.size == 0: + return 0.0 + gm = np.exp(np.mean(np.log(p))) + am = np.mean(p) + return float(gm / am) if am > 0 else 0.0 + + +def instantaneous_frequency(x: np.ndarray, sample_rate: float) -> np.ndarray: + """Discriminator output in Hz (the derivative of the unwrapped phase).""" + if x.size < 2: + return np.zeros(0, dtype=np.float64) + prod = x[1:] * np.conj(x[:-1]) + return np.angle(prod) * (sample_rate / (2.0 * math.pi)) + + +def envelope(x: np.ndarray) -> np.ndarray: + return np.abs(x).astype(np.float64) + + +def goertzel(x: np.ndarray, sample_rate: float, target_hz: float) -> float: + """Power at a single frequency -- cheap tone detection (CTCSS, pilots).""" + n = x.size + if n == 0: + return 0.0 + k = int(0.5 + (n * target_hz) / sample_rate) + w = 2.0 * math.pi * k / n + coeff = 2.0 * math.cos(w) + s1 = s2 = 0.0 + xr = np.asarray(x, dtype=np.float64) + for v in xr: + s0 = v + coeff * s1 - s2 + s2, s1 = s1, s0 + return float(s1 * s1 + s2 * s2 - coeff * s1 * s2) / n diff --git a/bandsaunter/librtlsdr.py b/bandsaunter/librtlsdr.py new file mode 100755 index 0000000..1217efb --- /dev/null +++ b/bandsaunter/librtlsdr.py @@ -0,0 +1,338 @@ +"""Minimal ctypes binding to librtlsdr. + +Only the subset of the API that the scanner needs is bound. Kept separate from +``device.py`` so that the raw C surface stays easy to audit. +""" + +from __future__ import annotations + +import ctypes +import ctypes.util +import os + +__all__ = ["LibRtlSdr", "RtlSdrError", "load", "is_available", "load_error"] + + +class RtlSdrError(RuntimeError): + """Raised when librtlsdr reports a failure or is unusable.""" + + +# librtlsdr returns negative errno-ish codes; these are the ones worth naming. +_ERRNO_TEXT = { + -1: "device handle is invalid", + -2: "device not found or already claimed", + -3: "operation not supported by this tuner", + -5: "USB transfer error (cable, power, or driver problem)", + -6: "device is busy", + -12: "out of memory", +} + + +def _describe(code: int) -> str: + return _ERRNO_TEXT.get(code, f"librtlsdr error {code}") + + +_CANDIDATES = ( + "librtlsdr.so.2", + "librtlsdr.so.0", + "librtlsdr.so", + "librtlsdr.dylib", + "rtlsdr.dll", + "librtlsdr.dll", +) + +_lib = None +_load_error: str | None = None + + +def _try_load(): + """Locate and dlopen librtlsdr, returning (handle, error_message).""" + tried = [] + + env = os.environ.get("BANDSAUNTER_LIBRTLSDR") + names = ([env] if env else []) + list(_CANDIDATES) + + found = ctypes.util.find_library("rtlsdr") + if found: + names.append(found) + + for name in names: + try: + return ctypes.CDLL(name), None + except OSError as exc: # pragma: no cover - platform dependent + tried.append(f"{name}: {exc}") + + hint = ( + "librtlsdr was not found. On Debian/Ubuntu install it with:\n" + " sudo apt install rtl-sdr librtlsdr0\n" + "On Fedora: sudo dnf install rtl-sdr\n" + "On macOS: brew install librtlsdr\n" + "Or set BANDSAUNTER_LIBRTLSDR=/path/to/librtlsdr.so" + ) + return None, hint + "\n\nTried:\n " + "\n ".join(tried) + + +# Callback librtlsdr invokes from its own thread for each filled buffer. +READ_ASYNC_CB = ctypes.CFUNCTYPE(None, ctypes.POINTER(ctypes.c_ubyte), + ctypes.c_uint32, ctypes.c_void_p) + + +def _bind(lib): + c = ctypes.c_int + u32 = ctypes.c_uint32 + p = ctypes.c_void_p + cp = ctypes.c_char_p + + sig = { + "rtlsdr_get_device_count": ([], ctypes.c_uint32), + "rtlsdr_get_device_name": ([u32], cp), + "rtlsdr_get_device_usb_strings": ([u32, cp, cp, cp], c), + "rtlsdr_get_index_by_serial": ([cp], c), + "rtlsdr_open": ([ctypes.POINTER(p), u32], c), + "rtlsdr_close": ([p], c), + "rtlsdr_set_center_freq": ([p, u32], c), + "rtlsdr_get_center_freq": ([p], u32), + "rtlsdr_set_freq_correction": ([p, c], c), + "rtlsdr_get_freq_correction": ([p], c), + "rtlsdr_get_tuner_type": ([p], c), + "rtlsdr_get_tuner_gains": ([p, ctypes.POINTER(c)], c), + "rtlsdr_set_tuner_gain": ([p, c], c), + "rtlsdr_get_tuner_gain": ([p], c), + "rtlsdr_set_tuner_gain_mode": ([p, c], c), + "rtlsdr_set_tuner_bandwidth": ([p, u32], c), + "rtlsdr_set_sample_rate": ([p, u32], c), + "rtlsdr_get_sample_rate": ([p], u32), + "rtlsdr_set_agc_mode": ([p, c], c), + "rtlsdr_set_direct_sampling": ([p, c], c), + "rtlsdr_get_direct_sampling": ([p], c), + "rtlsdr_set_offset_tuning": ([p, c], c), + "rtlsdr_get_offset_tuning": ([p], c), + "rtlsdr_set_bias_tee": ([p, c], c), + "rtlsdr_reset_buffer": ([p], c), + "rtlsdr_read_sync": ([p, p, c, ctypes.POINTER(c)], c), + "rtlsdr_read_async": ([p, READ_ASYNC_CB, p, u32, u32], c), + "rtlsdr_cancel_async": ([p], c), + } + + missing = [] + for name, (argtypes, restype) in sig.items(): + fn = getattr(lib, name, None) + if fn is None: + missing.append(name) + continue + fn.argtypes = argtypes + fn.restype = restype + + # A few entry points only exist in newer librtlsdr; the wrapper degrades + # gracefully for those rather than refusing to run. + hard_required = { + "rtlsdr_open", + "rtlsdr_close", + "rtlsdr_set_center_freq", + "rtlsdr_set_sample_rate", + "rtlsdr_read_sync", + "rtlsdr_reset_buffer", + } + fatal = hard_required.intersection(missing) + if fatal: + raise RtlSdrError( + "librtlsdr is missing required symbols: " + ", ".join(sorted(fatal)) + ) + return lib + + +def load(): + """Return the loaded librtlsdr handle, raising RtlSdrError if unavailable.""" + global _lib, _load_error + if _lib is not None: + return _lib + if _load_error is not None: + raise RtlSdrError(_load_error) + lib, err = _try_load() + if lib is None: + _load_error = err + raise RtlSdrError(err) + _lib = _bind(lib) + return _lib + + +def is_available() -> bool: + try: + load() + return True + except RtlSdrError: + return False + + +def load_error() -> str | None: + """The dlopen failure message, or None if the library loaded fine.""" + try: + load() + return None + except RtlSdrError as exc: + return str(exc) + + +class LibRtlSdr: + """Thin, checked wrapper around the C calls. + + Every method raises :class:`RtlSdrError` on a negative return code so the + layers above can assume success. + """ + + TUNER_NAMES = { + 0: "unknown", + 1: "Elonics E4000", + 2: "Fitipower FC0012", + 3: "Fitipower FC0013", + 4: "FCI FC2580", + 5: "Rafael Micro R820T/R820T2", + 6: "Rafael Micro R828D", + } + + def __init__(self): + self.lib = load() + + # -- enumeration --------------------------------------------------- + def device_count(self) -> int: + return int(self.lib.rtlsdr_get_device_count()) + + def device_name(self, index: int) -> str: + name = self.lib.rtlsdr_get_device_name(index) + return name.decode("utf-8", "replace") if name else "?" + + def usb_strings(self, index: int): + buf = [ctypes.create_string_buffer(256) for _ in range(3)] + rc = self.lib.rtlsdr_get_device_usb_strings(index, *buf) + if rc < 0: + return ("?", "?", "?") + return tuple(b.value.decode("utf-8", "replace") for b in buf) + + # -- lifecycle ----------------------------------------------------- + def open(self, index: int) -> ctypes.c_void_p: + handle = ctypes.c_void_p() + rc = self.lib.rtlsdr_open(ctypes.byref(handle), index) + if rc < 0: + extra = "" + if rc == -2 or rc == -6: + extra = ( + "\nThe DVB-T kernel driver may have claimed it. Blacklist it:\n" + " echo 'blacklist dvb_usb_rtl28xxu' | " + "sudo tee /etc/modprobe.d/blacklist-rtl.conf\n" + " sudo rmmod dvb_usb_rtl28xxu\n" + "Also make sure your user can access the USB device " + "(udev rule / plugdev group)." + ) + raise RtlSdrError(f"could not open device {index}: {_describe(rc)}{extra}") + return handle + + def close(self, dev) -> None: + if dev: + self.lib.rtlsdr_close(dev) + + # -- checked setters ----------------------------------------------- + def _check(self, rc: int, what: str, soft: bool = False) -> bool: + if rc < 0: + if soft: + return False + raise RtlSdrError(f"{what} failed: {_describe(rc)}") + return True + + def set_center_freq(self, dev, hz: int) -> None: + self._check(self.lib.rtlsdr_set_center_freq(dev, int(hz)), + f"tuning to {hz/1e6:.6f} MHz") + + def get_center_freq(self, dev) -> int: + return int(self.lib.rtlsdr_get_center_freq(dev)) + + def set_sample_rate(self, dev, hz: int) -> None: + self._check(self.lib.rtlsdr_set_sample_rate(dev, int(hz)), + f"setting sample rate {hz}") + + def get_sample_rate(self, dev) -> int: + return int(self.lib.rtlsdr_get_sample_rate(dev)) + + def set_freq_correction(self, dev, ppm: int) -> bool: + # librtlsdr returns -2 when the value is already set; that is benign. + rc = self.lib.rtlsdr_set_freq_correction(dev, int(ppm)) + return rc >= 0 or rc == -2 + + def get_tuner_type(self, dev) -> str: + try: + return self.TUNER_NAMES.get(int(self.lib.rtlsdr_get_tuner_type(dev)), + "unknown") + except Exception: + return "unknown" + + def get_tuner_gains(self, dev) -> list[float]: + n = self.lib.rtlsdr_get_tuner_gains(dev, None) + if n <= 0: + return [] + arr = (ctypes.c_int * n)() + self.lib.rtlsdr_get_tuner_gains(dev, arr) + return [v / 10.0 for v in arr] + + def set_tuner_gain_mode(self, dev, manual: bool) -> None: + self._check(self.lib.rtlsdr_set_tuner_gain_mode(dev, 1 if manual else 0), + "setting tuner gain mode") + + def set_tuner_gain(self, dev, db: float) -> None: + self._check(self.lib.rtlsdr_set_tuner_gain(dev, int(round(db * 10))), + f"setting tuner gain {db} dB") + + def get_tuner_gain(self, dev) -> float: + return self.lib.rtlsdr_get_tuner_gain(dev) / 10.0 + + def set_tuner_bandwidth(self, dev, hz: int) -> bool: + fn = getattr(self.lib, "rtlsdr_set_tuner_bandwidth", None) + if fn is None: + return False + return self._check(fn(dev, int(hz)), "setting tuner bandwidth", soft=True) + + def set_agc_mode(self, dev, on: bool) -> bool: + return self._check(self.lib.rtlsdr_set_agc_mode(dev, 1 if on else 0), + "setting RTL2832 AGC", soft=True) + + def set_direct_sampling(self, dev, mode: int) -> bool: + """0 = off, 1 = I branch, 2 = Q branch.""" + return self._check(self.lib.rtlsdr_set_direct_sampling(dev, int(mode)), + "setting direct sampling", soft=True) + + def get_direct_sampling(self, dev) -> int: + try: + return int(self.lib.rtlsdr_get_direct_sampling(dev)) + except Exception: + return 0 + + def set_offset_tuning(self, dev, on: bool) -> bool: + return self._check(self.lib.rtlsdr_set_offset_tuning(dev, 1 if on else 0), + "setting offset tuning", soft=True) + + def set_bias_tee(self, dev, on: bool) -> bool: + fn = getattr(self.lib, "rtlsdr_set_bias_tee", None) + if fn is None: + return False + return self._check(fn(dev, 1 if on else 0), "setting bias tee", soft=True) + + def reset_buffer(self, dev) -> None: + self._check(self.lib.rtlsdr_reset_buffer(dev), "resetting USB buffer") + + def read_async(self, dev, callback, buf_num: int = 15, + buf_len: int = 262144) -> None: + """Stream continuously until :meth:`cancel_async`. Blocks the caller.""" + rc = self.lib.rtlsdr_read_async(dev, callback, None, + int(buf_num), int(buf_len)) + if rc < 0: + raise RtlSdrError(f"async read failed: {_describe(rc)}") + + def cancel_async(self, dev) -> None: + self.lib.rtlsdr_cancel_async(dev) + + def read_sync(self, dev, buf, count: int) -> int: + n_read = ctypes.c_int(0) + rc = self.lib.rtlsdr_read_sync( + dev, ctypes.cast(buf, ctypes.c_void_p), count, ctypes.byref(n_read) + ) + if rc < 0: + raise RtlSdrError(f"USB read failed: {_describe(rc)}") + return int(n_read.value) diff --git a/bandsaunter/morse.py b/bandsaunter/morse.py new file mode 100755 index 0000000..db198c8 --- /dev/null +++ b/bandsaunter/morse.py @@ -0,0 +1,345 @@ +"""CW / Morse detection and decoding. + +Takes demodulated audio (or a complex baseband block), finds the keyed tone, +recovers the element timing without being told the speed, and decodes the +text. Also reports a confidence so the scanner can tell real Morse from any +other on-off keyed carrier. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass, field + +import numpy as np +from scipy import signal as sps + +__all__ = ["decode_morse", "MorseResult", "MORSE_TABLE", "encode_morse"] + + +MORSE_TABLE: dict[str, str] = { + ".-": "A", "-...": "B", "-.-.": "C", "-..": "D", ".": "E", + "..-.": "F", "--.": "G", "....": "H", "..": "I", ".---": "J", + "-.-": "K", ".-..": "L", "--": "M", "-.": "N", "---": "O", + ".--.": "P", "--.-": "Q", ".-.": "R", "...": "S", "-": "T", + "..-": "U", "...-": "V", ".--": "W", "-..-": "X", "-.--": "Y", + "--..": "Z", + "-----": "0", ".----": "1", "..---": "2", "...--": "3", "....-": "4", + ".....": "5", "-....": "6", "--...": "7", "---..": "8", "----.": "9", + ".-.-.-": ".", "--..--": ",", "..--..": "?", ".----.": "'", + "-.-.--": "!", "-..-.": "/", "-.--.": "(", "-.--.-": ")", + ".-...": "&", "---...": ":", "-.-.-.": ";", "-...-": "=", + ".-.-.": "+", "-....-": "-", "..--.-": "_", ".-..-.": '"', + "...-..-": "$", ".--.-.": "@", + # Common prosigns, rendered between angle brackets. + "...-.-": "", "-.-.-": "", ".-.-": "", "...-.": "", + "-...-.-": "", "........": "", ".-.-.-.-": "", +} + +_REVERSE = {v: k for k, v in MORSE_TABLE.items() if len(v) == 1} + +# Morse timing is defined against PARIS = 50 dot units per word. +_DOTS_PER_WORD = 50.0 + + +@dataclass +class MorseResult: + text: str = "" + wpm: float = 0.0 + tone_hz: float = 0.0 + dot_seconds: float = 0.0 + confidence: float = 0.0 + n_elements: int = 0 + n_characters: int = 0 + timing_fit: float = 0.0 + undecoded: int = 0 + snr_db: float = 0.0 + notes: list[str] = field(default_factory=list) + + @property + def is_morse(self) -> bool: + # Nobody sends below 5 or above 60 WPM. A "decode" outside that range + # is the timing estimator latching onto something that is not Morse -- + # speech syllables, for instance. + return (self.confidence >= 0.5 and self.n_characters >= 3 + and 5.0 <= self.wpm <= 60.0 + # Enough elements to be a transmission rather than a handful + # of noise crossings, and timing that genuinely fits Morse. + and self.n_elements >= 8 and self.timing_fit >= 0.7 + and self.undecoded <= 0.3 * max(1, self.n_characters)) + + def summary(self) -> str: + if not self.text: + return "keyed carrier, no readable Morse" + return f"{self.wpm:.0f} WPM: {self.text.strip()}" + + +def encode_morse(text: str) -> str: + """Encode text to Morse -- handy for generating test signals.""" + out = [] + for ch in text.upper(): + if ch == " ": + out.append("/") + elif ch in _REVERSE: + out.append(_REVERSE[ch]) + return " ".join(out) + + +# --------------------------------------------------------------------------- + +def _find_tone(audio: np.ndarray, fs: float, + lo: float = 200.0, hi: float = 3000.0) -> tuple[float, float]: + """Locate the keyed tone. Returns ``(freq_hz, prominence_db)``.""" + n = 1 << int(math.floor(math.log2(max(1024, min(audio.size, 1 << 16))))) + if audio.size < n: + return 0.0, 0.0 + x = audio[:n].astype(np.float64) + x -= x.mean() + spec = np.abs(np.fft.rfft(x * np.hanning(n), n)) + freqs = np.fft.rfftfreq(n, 1.0 / fs) + band = (freqs >= lo) & (freqs <= min(hi, fs / 2.2)) + if not np.any(band): + return 0.0, 0.0 + sub, subf = spec[band], freqs[band] + k = int(np.argmax(sub)) + prom = 20.0 * math.log10((sub[k] + 1e-12) / (np.median(sub) + 1e-12)) + return float(subf[k]), float(prom) + + +def _tone_envelope(audio: np.ndarray, fs: float, tone_hz: float, + width: float = 150.0) -> tuple[np.ndarray, float]: + """Band-pass around the tone and return its envelope at a reduced rate.""" + nyq = fs / 2.0 + lo = max(20.0, tone_hz - width) / nyq + hi = min(0.99, (tone_hz + width) / nyq) + if hi <= lo: + return np.abs(audio), fs + taps = sps.firwin(255, [lo, hi], pass_zero=False) + filtered = sps.lfilter(taps, [1.0], audio)[255:] + env = np.abs(sps.hilbert(filtered)) + + # 500 Hz of envelope bandwidth resolves elements down to ~2 ms. + dec = max(1, int(fs // 1000)) + if dec > 1: + env = sps.decimate(env, dec, ftype="fir", zero_phase=True) + return env, fs / dec + + +def _threshold(env: np.ndarray) -> float: + """Split the envelope into key-down and key-up populations.""" + lo = float(np.percentile(env, 15)) + hi = float(np.percentile(env, 85)) + if hi <= lo: + return float(np.median(env)) + # Midway in the log domain: keying contrast is multiplicative, not additive. + return float(math.sqrt(max(lo, 1e-12) * max(hi, 1e-12))) + + +def _despeckle(on: np.ndarray, min_len: int) -> np.ndarray: + """Remove key transitions too short to be real elements. + + Filter ring at the edges of each element crosses the threshold for a + sample or two. Left in, those specks become the shortest "element" and + drag the dot-length estimate down to nothing, so flatten anything below + the fastest plausible keying speed into its neighbours. + """ + if min_len <= 1 or on.size == 0: + return on + out = on.copy() + for _ in range(4): + changed = False + for state, start, length in [(s, a, b) for s, a, b in _run_spans(out)]: + if length < min_len: + out[start:start + length] = not state + changed = True + if not changed: + break + return out + + +def _run_spans(on: np.ndarray): + if on.size == 0: + return + change = np.flatnonzero(np.diff(on.astype(np.int8))) + bounds = np.concatenate(([0], change + 1, [on.size])) + for i in range(bounds.size - 1): + yield bool(on[bounds[i]]), int(bounds[i]), int(bounds[i + 1] - bounds[i]) + + +def _runs(on: np.ndarray) -> list[tuple[bool, int]]: + if on.size == 0: + return [] + change = np.flatnonzero(np.diff(on.astype(np.int8))) + bounds = np.concatenate(([0], change + 1, [on.size])) + return [(bool(on[bounds[i]]), int(bounds[i + 1] - bounds[i])) + for i in range(bounds.size - 1)] + + +def _estimate_dot(on_lengths: list[int], off_lengths: list[int]) -> float: + """Two-means clustering in the log domain gives a speed-agnostic dot length.""" + vals = np.array([v for v in on_lengths if v > 0], dtype=np.float64) + if vals.size == 0: + return 0.0 + if vals.size < 4: + return float(np.min(vals)) + + logs = np.log(vals) + lo, hi = float(logs.min()), float(logs.max()) + if hi - lo < 0.35: # all one length -- everything is a dot + return float(np.median(vals)) + c0, c1 = lo, hi + for _ in range(30): + d0 = np.abs(logs - c0) + d1 = np.abs(logs - c1) + g0, g1 = logs[d0 <= d1], logs[d1 < d0] + if g0.size == 0 or g1.size == 0: + break + n0, n1 = float(g0.mean()), float(g1.mean()) + if abs(n0 - c0) < 1e-6 and abs(n1 - c1) < 1e-6: + c0, c1 = n0, n1 + break + c0, c1 = n0, n1 + dot = math.exp(c0) + dash = math.exp(c1) + # A dash is three dots; if the two clusters are close to that ratio, use + # both to refine the estimate. + if 2.0 <= dash / max(dot, 1e-9) <= 4.5: + dot = 0.5 * (dot + dash / 3.0) + # The inter-element gap is also one dot long -- a useful cross-check. + offs = np.array([v for v in off_lengths if v > 0], dtype=np.float64) + if offs.size >= 3: + short = offs[offs <= np.percentile(offs, 40)] + if short.size and 0.5 * dot <= np.median(short) <= 2.0 * dot: + dot = 0.5 * (dot + float(np.median(short))) + return dot + + +def decode_morse(audio: np.ndarray, sample_rate: float, + min_elements: int = 6) -> MorseResult: + """Decode CW from a block of demodulated audio. + + ``audio`` should be real audio containing the beat note (what the ``cw`` + demodulator produces). Speed is estimated from the signal itself, so no + WPM setting is needed. + """ + res = MorseResult() + audio = np.asarray(audio, dtype=np.float64).ravel() + if audio.size < int(sample_rate * 0.3): + res.notes.append("too short to decode") + return res + + tone, prom = _find_tone(audio, sample_rate) + res.tone_hz = tone + res.snr_db = prom + if tone <= 0 or prom < 6.0: + res.notes.append("no steady tone found") + return res + + env, fs_env = _tone_envelope(audio, sample_rate, tone) + if env.size < 32: + res.notes.append("envelope too short") + return res + + thr = _threshold(env) + on = env > thr + # 6 ms is a dot at about 200 WPM -- far beyond any real operator or + # machine, so anything shorter is filter ring, not keying. + on = _despeckle(on, max(2, int(0.006 * fs_env))) + runs = _runs(on) + # Drop the leading and trailing partial runs -- they are cut off by the + # capture window and would corrupt the timing estimate. + if len(runs) >= 3: + runs = runs[1:-1] + if len(runs) < min_elements: + res.notes.append("not enough keying transitions") + return res + + on_lengths = [n for state, n in runs if state] + off_lengths = [n for state, n in runs if not state] + if not on_lengths: + res.notes.append("carrier never keys down") + return res + + dot = _estimate_dot(on_lengths, off_lengths) + if dot <= 0: + res.notes.append("could not estimate element length") + return res + res.dot_seconds = dot / fs_env + if not (0.008 <= res.dot_seconds <= 0.5): + res.notes.append(f"implausible element length ({res.dot_seconds*1e3:.0f} ms)") + return res + res.wpm = 1.2 / res.dot_seconds * (_DOTS_PER_WORD / 50.0) + + # ---- element decision -------------------------------------------- + symbols: list[str] = [] + text_parts: list[str] = [] + current = "" + undecoded = 0 + + def flush(): + nonlocal current, undecoded + if not current: + return + ch = MORSE_TABLE.get(current) + if ch is None: + undecoded += 1 + text_parts.append("") + else: + text_parts.append(ch) + current = "" + + for state, n in runs: + units = n / dot + if state: + symbols.append("." if units < 2.0 else "-") + current += symbols[-1] + else: + if units < 2.0: + continue # gap between elements of a letter + flush() + if units >= 5.0: + text_parts.append(" ") # word gap + flush() + + text = "".join(text_parts) + # Collapse runs of spaces the timing may have produced. + text = " ".join(text.split(" ")) if text else "" + res.text = text + res.n_elements = len(symbols) + res.n_characters = sum(1 for p in text_parts if p not in (" ",)) + res.undecoded = undecoded + + # ---- confidence ---------------------------------------------------- + if res.n_characters == 0: + res.confidence = 0.0 + res.notes.append("no characters resolved") + return res + + good_ratio = 1.0 - (undecoded / max(1, res.n_characters)) + # Real Morse has a clean bimodal element histogram: mostly 1 and 3 units. + units = np.array([n / dot for state, n in runs if state], dtype=np.float64) + # Real keying lands tightly on one and three units. A loose tolerance + # here lets the random run lengths of a noisy channel "fit" Morse. + fit = float(np.mean(np.minimum(np.abs(units - 1.0), np.abs(units - 3.0)) < 0.35)) + res.timing_fit = round(fit, 3) + speed_ok = 1.0 if 5.0 <= res.wpm <= 60.0 else 0.4 + conf = 0.45 * good_ratio + 0.35 * fit + 0.20 * speed_ok + if res.n_characters < 3: + conf *= 0.5 + + # A uniform pulse train decodes to "EEEE" or "TTTT" and looks perfect by + # the metrics above. Real text mixes dots and dashes and uses more than + # one or two letters, so require both. + dash_ratio = float(np.mean(units > 2.0)) if units.size else 0.0 + res.notes.append(f"{dash_ratio*100:.0f}% dashes") + if dash_ratio < 0.10 or dash_ratio > 0.80: + conf *= 0.35 + res.notes.append("element mix is not typical of Morse text") + distinct = {p for p in text_parts if p not in (" ", "")} + if len(distinct) < 2: + conf *= 0.35 + res.notes.append("only one distinct character decoded") + res.confidence = round(min(0.99, max(0.0, conf)), 3) + if res.confidence < 0.5: + res.notes.append("timing does not fit Morse cleanly") + return res diff --git a/bandsaunter/quality.py b/bandsaunter/quality.py new file mode 100644 index 0000000..713ae28 --- /dev/null +++ b/bandsaunter/quality.py @@ -0,0 +1,377 @@ +"""Deciding whether a capture actually carries a signal worth keeping. + +Power alone cannot tell a transmission from a hump of interference, so every +capture is assessed for *content* before it is kept: + +* **voice** -- speech structure in the demodulated audio +* **cw** -- a keyed carrier whose timing resolves as Morse +* **digital** -- a symbol rate, discrete FSK levels, or an M-PSK phase line +* **carrier** -- a steady unmodulated carrier (real, but carries nothing) +* **noise** -- no structure at all: static, interference, receiver artefacts + +Only the categories the user asked for are recorded. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass, field + +import numpy as np + +__all__ = ["VoiceMetrics", "voice_metrics", "Assessment", "assess", + "CATEGORIES", "RAYLEIGH_CV"] + +CATEGORIES = ("voice", "cw", "digital", "carrier", "noise") + +# Envelope coefficient of variation for complex Gaussian noise: |x| is +# Rayleigh distributed, so std/mean is exactly sqrt(4/pi - 1). +RAYLEIGH_CV = math.sqrt(4.0 / math.pi - 1.0) # 0.5227 + + +@dataclass +class VoiceMetrics: + """Speech-likeness measurements taken on demodulated audio.""" + + voiced_fraction: float = 0.0 # frames with a clear pitch period + voiced_frames: int = 0 # how much evidence that fraction rests on + pitch_hz: float = 0.0 # median pitch of the voiced frames + pitch_variation: float = 0.0 # speech pitch drifts; a buzz does not + dynamic_range_db: float = 0.0 # speech pauses between phrases + syllabic: float = 0.0 # envelope modulation in the 2-8 Hz band + band_concentration: float = 0.0 # energy inside 250-3400 Hz + spectral_flux: float = 0.0 # formants moving between frames + active_fraction: float = 0.0 # how much of the clip is above silence + score: float = 0.0 + + def describe(self) -> str: + return (f"voiced {self.voiced_fraction:.2f}, pitch " + f"{self.pitch_hz:.0f} Hz +-{self.pitch_variation*100:.0f}%, " + f"dynamics {self.dynamic_range_db:.0f} dB, " + f"syllabic {self.syllabic:.2f}") + + +def _frames(x: np.ndarray, fs: float, win_s: float = 0.032, + hop_s: float = 0.010): + n = int(win_s * fs) + hop = max(1, int(hop_s * fs)) + if x.size < n * 3: + return np.zeros((0, n)), hop + count = 1 + (x.size - n) // hop + idx = np.arange(n)[None, :] + hop * np.arange(count)[:, None] + return x[idx], hop + + +def _pitch_track(frames: np.ndarray, fs: float, + f_lo: float = 70.0, f_hi: float = 400.0): + """Normalised autocorrelation peak per frame, over the human pitch range. + + Computed for every frame at once through the FFT. A direct + autocorrelation is quadratic in the frame length, and running it frame by + frame was slow enough to stall the capture loop -- which costs samples, + and shows up as a recording that plays too fast. + + Returns ``(strengths, pitches_hz)``. + """ + n = frames.shape[1] + if frames.shape[0] == 0 or n < 16: + return np.zeros(0), np.zeros(0) + f = frames - frames.mean(axis=1, keepdims=True) + energy = np.einsum("ij,ij->i", f, f) + + nfft = 1 << int(math.ceil(math.log2(2 * n))) + spec = np.fft.rfft(f, nfft, axis=1) + ac = np.fft.irfft(spec * np.conj(spec), nfft, axis=1)[:, :n] + + lo = max(1, int(fs / f_hi)) + hi = min(n - 1, int(fs / f_lo)) + if hi <= lo + 4: + return np.zeros(frames.shape[0]), np.zeros(frames.shape[0]) + + seg = ac[:, lo:hi] + k = np.argmax(seg, axis=1) + peak = seg[np.arange(seg.shape[0]), k] + strength = np.where(energy > 1e-12, peak / np.maximum(energy, 1e-12), 0.0) + # A maximum sitting on the edge of the search range is the search running + # out of room, not a periodicity. Noise does this constantly, and the + # result is a stream of "pitches" pinned at exactly the limit. + edge = (k <= 1) | (k >= seg.shape[1] - 2) + strength = np.where(edge, 0.0, strength) + pitch = np.where(edge, 0.0, fs / (lo + k)) + return strength, pitch + + +def voice_metrics(audio: np.ndarray, fs: float) -> VoiceMetrics: + """Measure how speech-like a block of audio is. + + Speech has four properties that static, hum, tones and data bursts do not + have together: a pitch period in the 70-400 Hz range during voiced sounds, + pauses between phrases, an envelope that varies at the syllable rate, and + formants that move from frame to frame. + """ + m = VoiceMetrics() + audio = np.asarray(audio, dtype=np.float64).ravel() + if audio.size < int(fs * 0.4): + return m + audio = audio - audio.mean() + peak = float(np.abs(audio).max()) + if peak < 1e-6: + return m + audio = audio / peak + + frames, hop = _frames(audio, fs) + if frames.shape[0] < 16: + return m + win = np.hanning(frames.shape[1]) + + energy = np.sqrt(np.mean((frames * win) ** 2, axis=1)) + 1e-12 + e_db = 20.0 * np.log10(energy) + + # Speech pauses show up as a wide spread between loud and quiet frames. + m.dynamic_range_db = float(np.percentile(e_db, 95) - np.percentile(e_db, 15)) + + # Only look for pitch where there is something to look at. + gate = e_db > (np.percentile(e_db, 95) - 25.0) + m.active_fraction = float(gate.mean()) + + idx = np.flatnonzero(gate)[:400] + if idx.size: + strengths, pitches = _pitch_track(frames[idx], fs) + voiced = strengths > 0.38 + m.voiced_fraction = float(voiced.mean()) + m.voiced_frames = int(voiced.sum()) + if np.any(voiced): + track = np.asarray(pitches)[voiced] + m.pitch_hz = float(np.median(track)) + if m.pitch_hz > 0 and track.size > 3: + # Intonation: a talker's pitch wanders, mains hum and a + # switching-supply buzz sit on exactly one frequency. + m.pitch_variation = float(np.std(track) / m.pitch_hz) + + # Syllable-rate envelope modulation, 2-8 Hz. + frame_rate = fs / hop + env = energy - energy.mean() + if env.size >= 64 and frame_rate > 24: + n = 1 << int(math.floor(math.log2(env.size))) + spec = np.abs(np.fft.rfft(env[:n] * np.hanning(n), n)) + mf = np.fft.rfftfreq(n, 1.0 / frame_rate) + syl = (mf >= 2.0) & (mf <= 8.0) + ref = (mf >= 0.5) & (mf <= 20.0) + if np.any(syl) and np.any(ref): + total = float(np.sum(spec[ref] ** 2)) + if total > 0: + m.syllabic = float(np.sum(spec[syl] ** 2) / total) + + # Energy inside the voice band, and how much the spectrum moves. + mag = np.abs(np.fft.rfft(frames * win, axis=1)) + freqs = np.fft.rfftfreq(frames.shape[1], 1.0 / fs) + band = (freqs >= 250.0) & (freqs <= 3400.0) + tot = np.sum(mag ** 2) + 1e-12 + m.band_concentration = float(np.sum(mag[:, band] ** 2) / tot) + + norm = mag / (np.linalg.norm(mag, axis=1, keepdims=True) + 1e-12) + if norm.shape[0] > 1: + m.spectral_flux = float(np.mean(np.linalg.norm(np.diff(norm, axis=0), + axis=1))) + + # ---- combine ------------------------------------------------------- + # A pitch track that moves is the one measurement here that only a voice + # produces, so it gates the score rather than contributing a share of it. + # Dynamics, syllable-rate modulation and energy landing in the voice band + # are all things that static does too: weighted alongside voicing they + # were enough to carry noise over the line on their own, which is exactly + # how hiss ended up being recorded as speech. + core = min(1.0, m.voiced_fraction / 0.30) + # Pitch drift is the guard against tones and hum, but measuring drift + # needs several voiced frames to measure it across. A short over -- a + # two-second "QSL, 73" -- cannot supply them, so the requirement eases + # when the evidence is thin; the steady-tone guard below still applies. + if m.voiced_frames >= 20: + core *= min(1.0, m.pitch_variation / 0.04) + else: + core *= min(1.0, 0.45 + m.pitch_variation / 0.04) + + dyn_term = min(1.0, max(0.0, (m.dynamic_range_db - 6.0) / 20.0)) + syl_term = min(1.0, max(0.0, (m.syllabic - 0.18) / 0.30)) + band_term = min(1.0, max(0.0, (m.band_concentration - 0.25) / 0.45)) + support = 0.40 * dyn_term + 0.35 * syl_term + 0.25 * band_term + + score = core * (0.55 + 0.45 * support) + + # A steady tone is perfectly periodic and would score full marks on pitch; + # what it does not have is a spectrum that changes or any dynamics. + if m.spectral_flux < 0.045 and m.dynamic_range_db < 6.0: + score *= 0.25 + m.score = float(min(1.0, score)) + return m + + +# --------------------------------------------------------------------------- + +@dataclass +class Assessment: + category: str = "noise" + score: float = 0.0 + accept: bool = False + reason: str = "" + voice: VoiceMetrics = field(default_factory=VoiceMetrics) + noise_likeness: float = 0.0 + + def describe(self) -> str: + return f"{self.category} ({self.score:.2f}): {self.reason}" + + +def noise_likeness(f) -> float: + """0 = structured signal, 1 = indistinguishable from receiver noise. + + Gaussian noise has a known envelope statistic, a flat spectrum, no + carrier, no discrete frequency levels and no symbol rate. Scoring all of + those together is far more reliable than any one of them. + """ + votes = [] + # Envelope statistics sitting on the Rayleigh value. + votes.append(max(0.0, 1.0 - abs(f.env_cv - RAYLEIGH_CV) / 0.16)) + # Flat, featureless spectrum. + votes.append(min(1.0, max(0.0, (f.flatness - 0.35) / 0.45))) + # No carrier line worth the name. + votes.append(min(1.0, max(0.0, (14.0 - f.papr_spectral) / 10.0))) + # No symbol rate. + votes.append(1.0 if f.baud_strength < 11.0 else 0.0) + # No discrete discriminator levels. + votes.append(1.0 if f.freq_modes <= 1 else 0.0) + # No keying. + votes.append(min(1.0, max(0.0, (9.0 - f.ook_contrast_db) / 9.0))) + return float(np.mean(votes)) + + +def digital_structure(f) -> tuple[float, list[str]]: + """Score how much symbol structure a signal shows, with the evidence. + + A symbol *rate* on its own proves nothing -- the cyclostationary estimator + will always return its best peak, and noise and bare carriers both produce + one. So an identified keying scheme is required first, and the baud + figure only corroborates it. + """ + score = 0.0 + bits: list[str] = [] + + if f.freq_modes >= 2 and f.mode_spacing > 0 and f.level_dwell > 0.32: + score += 0.45 + bits.append(f"{f.freq_modes} discrete frequency levels " + f"{f.mode_spacing:.0f} Hz apart") + # A phase line needs a symbol rate behind it that holds still; without + # one, an Mth-power peak is just the strongest thing in a noisy spectrum. + stable_baud = (f.baud_strength > 14 and 50.0 <= f.baud <= 100_000.0 + and f.baud_stability > 0.80) + if f.psk_strength > 22 and f.psk_order and f.ifreq_kurtosis > 3.0 \ + and f.psk_strength > f.papr_spectral + 8.0 \ + and f.carrier_ratio < 0.10 and stable_baud: + score += 0.40 + bits.append(f"{f.psk_order}-PSK phase line") + # On/off contrast on its own is not evidence of keying: a signal fading + # in and out at the squelch edge produces plenty of it. Real OOK data + # also has a symbol rate, so require both. + plausible_baud = stable_baud + # Keying means the on and off runs land on a common symbol grid. Contrast + # and a baud estimate are not enough on their own: a signal fading across + # the squelch produces both, with run lengths that fit no grid at all. + if (f.ook_contrast_db > 16 and 0.05 < f.ook_duty < 0.95 + and plausible_baud and f.keying_regularity > 0.55): + score += 0.35 + bits.append(f"on/off keying, {f.ook_contrast_db:.0f} dB contrast, " + f"runs on a {f.keying_regularity*100:.0f}% regular grid") + + if score <= 0: + return 0.0, [] + + if plausible_baud: + score += 0.20 + bits.append(f"{f.baud:.0f} baud symbol rate") + return score, bits + + +def assess(classification, audio: np.ndarray, audio_rate: float, + morse=None, min_voice: float = 0.45, freq_hz: float = 0.0, + accept: tuple[str, ...] = ("voice", "cw", "digital"), + min_score: float = 0.45) -> Assessment: + """Decide what a capture contains and whether it should be kept. + + Deliberately does not route on the classifier's label. The label is a + best guess that can be wrong -- speech on a quiet FM channel can look like + two-level FSK, for instance -- and a capture should be kept or dropped on + what is measurably *in* it, not on what it was called. + """ + f = classification.features + a = Assessment() + if f is None: + a.reason = "no measurements available" + return a + + a.noise_likeness = noise_likeness(f) + fam = classification.family + vm = voice_metrics(audio, audio_rate) + a.voice = vm + dig_score, dig_bits = digital_structure(f) + + # 1. Morse that actually decoded. + if morse is not None and morse.is_morse: + a.category = "cw" + a.score = float(morse.confidence) + a.reason = f'Morse decoded at {morse.wpm:.0f} WPM: "{morse.text.strip()[:40]}"' + + # 2. Speech, whatever the modulation was called. + elif vm.score >= min_voice: + a.category = "voice" + a.score = vm.score + a.reason = f"speech in the audio ({vm.describe()})" + + # 3. Symbol structure. + elif dig_score > 0.3: + a.category = "digital" + a.score = min(0.95, 0.35 + dig_score) + a.reason = "digital modulation: " + ", ".join(dig_bits) + + # 4. A keyed carrier whose timing would not resolve as Morse. + elif fam == "cw" and f.ook_contrast_db > 12: + a.category = "cw" + a.score = 0.55 + a.reason = (f"keyed carrier, {f.ook_contrast_db:.0f} dB on/off contrast " + "(timing did not resolve as Morse)") + + # 5. Broadcast FM gets a lenient path because most of it is music, which + # has no speech pitch track to find. That leniency is confined to + # signals that really are broadcast: in the FM band, or carrying a + # 19 kHz stereo pilot. Without the restriction, any wideband hump -- + # a clock harmonic, say -- walks straight through it. + elif (f.stereo_pilot or 87.5e6 <= freq_hz <= 108.1e6) \ + and f.bandwidth > 50_000 and f.fdev_rms > 2_000 \ + and (vm.spectral_flux > 0.15 or vm.dynamic_range_db > 8): + a.category = "voice" + a.score = max(0.55, vm.score) + a.reason = ("programme audio on a wideband FM carrier" + + (" with a 19 kHz stereo pilot" if f.stereo_pilot else "")) + + # 6. A carrier that is really just a carrier. + elif fam == "carrier" or (f.am_depth < 0.004 and f.fdev_rms < 150 + and f.ook_contrast_db < 6): + a.category = "carrier" + a.score = 0.6 + a.reason = "steady unmodulated carrier" + + else: + a.category = "noise" + a.score = vm.score + a.reason = (f"no speech, no symbol structure and no keying " + f"({vm.describe()})") + + # A strong noise verdict overrides any label, however confident. A + # score bar here left a hole: a single piece of "digital" evidence scored + # 0.80 and sailed past a check that only applied below 0.75. + if a.category != "noise" and a.noise_likeness > 0.82: + a.category = "noise" + a.reason = (f"statistics match receiver noise " + f"(noise likeness {a.noise_likeness:.2f})") + a.score = 0.0 + + a.accept = a.category in accept and a.score >= min_score + return a diff --git a/bandsaunter/ranges.py b/bandsaunter/ranges.py new file mode 100755 index 0000000..9909c7c --- /dev/null +++ b/bandsaunter/ranges.py @@ -0,0 +1,315 @@ +"""Frequency ranges, parsing, and turning them into a sweep plan.""" + +from __future__ import annotations + +import math +import re +from dataclasses import asdict, dataclass + +from .bandplan import BandPreset, by_key, fmt_hz, presets_covering + +__all__ = ["ScanRange", "TuneStep", "parse_frequency", "parse_range", + "parse_range_list", "build_plan", "fmt_hz", "RangeError"] + + +class RangeError(ValueError): + """Raised for un-parseable user frequency input.""" + + +_SUFFIX = { + "": 1.0, "hz": 1.0, + "k": 1e3, "khz": 1e3, + "m": 1e6, "mhz": 1e6, + "g": 1e9, "ghz": 1e9, +} + +_NUM = re.compile(r"^\s*([0-9]*\.?[0-9]+)\s*([a-zA-Z]*)\s*$") + + +def parse_frequency(text: str, default_unit: str = "") -> float: + """Parse ``146.52M``, ``146520000``, ``433.92 MHz``, ``14074k`` -> Hz. + + A bare number with no suffix is read as MHz when it is small enough to be + unambiguous (under 10000), otherwise as Hz -- which is how people + actually write frequencies. + """ + if text is None: + raise RangeError("empty frequency") + s = str(text).strip().replace(",", "").replace("_", "") + if not s: + raise RangeError("empty frequency") + m = _NUM.match(s) + if not m: + raise RangeError(f"cannot read {text!r} as a frequency") + value, suffix = float(m.group(1)), m.group(2).lower() + + if not suffix: + suffix = default_unit.lower() + if not suffix: + # No unit anywhere: guess from magnitude. + return value * 1e6 if value < 10_000 else value + if suffix not in _SUFFIX: + raise RangeError(f"unknown frequency unit {m.group(2)!r} in {text!r}") + return value * _SUFFIX[suffix] + + +@dataclass +class ScanRange: + """One start/stop pair the user wants swept.""" + + start: float + stop: float + step: float = 12_500.0 + mode: str = "auto" + bandwidth: float = 0.0 + label: str = "" + preset_key: str = "" + enabled: bool = True + threshold_db: float | None = None + + def __post_init__(self): + self.start = float(self.start) + self.stop = float(self.stop) + if self.stop < self.start: + self.start, self.stop = self.stop, self.start + if self.stop == self.start: + # A single frequency: give it enough width to actually measure. + pad = max(self.step, 12_500.0) / 2.0 + self.start -= pad + self.stop += pad + if self.step <= 0: + self.step = 12_500.0 + if not self.label: + self.label = f"{fmt_hz(self.start)}-{fmt_hz(self.stop)}" + + @property + def span(self) -> float: + return self.stop - self.start + + @classmethod + def from_preset(cls, preset: BandPreset, **over) -> "ScanRange": + kw = dict(start=preset.start, stop=preset.stop, step=preset.step, + mode=preset.mode, bandwidth=preset.bandwidth, + label=preset.name, preset_key=preset.key) + kw.update(over) + return cls(**kw) + + def _covering(self, hz: float, need_bandwidth: bool = False): + """Band-plan segments to resolve against at one frequency. + + A range that came from a band-plan preset resolves against segments of + the same service first. Without that, scanning the whole of 70 cm + would hand 433-435 MHz to the ISM preset that overlaps it there, and + an amateur repeater would be treated as an unlicensed device. + """ + covering = [p for p in presets_covering(hz) + if (p.bandwidth > 0 if need_bandwidth else p.mode != "auto")] + own = by_key(self.preset_key) if self.preset_key else None + if own is not None: + same = [p for p in covering if p.category == own.category] + if same: + return same + return covering + + def resolved_mode(self, freq: float | None = None) -> str: + """The demodulator to use, resolving ``auto`` against the band plan.""" + if self.mode and self.mode != "auto": + return self.mode + hz = freq if freq is not None else 0.5 * (self.start + self.stop) + # Whole-band entries are skipped: they defer to the segments beneath + # them, so resolving to one would hand "auto" straight back. + covering = self._covering(hz) + if covering: + return min(covering, key=lambda p: p.span).mode + return "nfm" + + def resolved_bandwidth(self, freq: float | None = None) -> float: + if self.bandwidth > 0: + return self.bandwidth + hz = freq if freq is not None else 0.5 * (self.start + self.stop) + covering = self._covering(hz, need_bandwidth=True) + if covering: + return min(covering, key=lambda p: p.span).bandwidth + return max(self.step, 12_500.0) + + def to_dict(self) -> dict: + return asdict(self) + + @classmethod + def from_dict(cls, d: dict) -> "ScanRange": + known = {f for f in cls.__dataclass_fields__} + return cls(**{k: v for k, v in d.items() if k in known}) + + def describe(self) -> str: + state = "" if self.enabled else " (disabled)" + return (f"{self.label}: {fmt_hz(self.start)} - {fmt_hz(self.stop)} " + f"step {fmt_hz(self.step)} mode {self.mode}{state}") + + +_RANGE_SPLIT = re.compile(r"\s*(?:-{1,2}|\.{2}|to|:{2})\s*", re.I) + + +def parse_range(text: str) -> ScanRange: + """Parse one range specification. + + Accepted forms:: + + 144M-148M explicit start and stop + 144-148M unit carried over to both ends + 146.52M a single frequency + 144M-148M/25k with an explicit step + 144M-148M/25k@nfm with a step and a demodulator + gmrs any band-plan preset key + + """ + if text is None: + raise RangeError("empty range") + s = str(text).strip() + if not s: + raise RangeError("empty range") + + mode = "auto" + if "@" in s: + s, _, mode = s.partition("@") + mode = mode.strip().lower() or "auto" + + step = None + if "/" in s: + s, _, step_s = s.partition("/") + step = parse_frequency(step_s, default_unit="khz") + + s = s.strip() + preset = by_key(s) + if preset is not None and preset.is_group: + raise RangeError( + f"{preset.key!r} covers {len(preset.expand())} separate ranges; " + "it can only be used where a list of ranges is accepted") + if preset is not None: + r = ScanRange.from_preset(preset) + if step: + r.step = step + if mode != "auto": + r.mode = mode + return r + + parts = [p for p in _RANGE_SPLIT.split(s) if p.strip()] + if len(parts) == 1: + hz = parse_frequency(parts[0]) + r = ScanRange(hz, hz, step or 12_500.0, mode) + elif len(parts) == 2: + # "144-148M": the unit on the right end applies to the left too. + right_unit = _NUM.match(parts[1]) + unit = right_unit.group(2) if right_unit else "" + lo = parse_frequency(parts[0], default_unit=unit) + hi = parse_frequency(parts[1]) + r = ScanRange(lo, hi, step or 12_500.0, mode) + else: + raise RangeError(f"cannot read {text!r} as a frequency range") + return r + + +def parse_range_list(text: str | list[str]) -> list[ScanRange]: + """Parse a comma/newline separated list into ranges (unlimited count). + + A band-plan key that stands for a set of others -- ``all-cw``, say -- + expands here into one range per member. + """ + if isinstance(text, (list, tuple)): + items = list(text) + else: + items = re.split(r"[,\n;]+", str(text)) + out = [] + for item in items: + item = item.strip() + if not item: + continue + preset = by_key(item) + if preset is not None and preset.is_group: + out.extend(ScanRange.from_preset(m) for m in preset.expand()) + else: + out.append(parse_range(item)) + return out + + +# --------------------------------------------------------------------------- +# Sweep planning +# --------------------------------------------------------------------------- + +@dataclass +class TuneStep: + """One tuner position, plus the slice of its spectrum we trust.""" + + center: float + low: float # first frequency this step is responsible for + high: float # last frequency this step is responsible for + range_index: int + range_label: str = "" + mode: str = "auto" + index: int = 0 # position in the plan; keys the noise-floor memory + + @property + def width(self) -> float: + return self.high - self.low + + def describe(self) -> str: + return f"{fmt_hz(self.center)} covering {fmt_hz(self.low)}-{fmt_hz(self.high)}" + + +def build_plan(ranges: list[ScanRange], sample_rate: float, + usable_fraction: float = 0.75, + dc_guard: float = 8_000.0, + max_steps: int = 200_000, + tunable=None) -> list[TuneStep]: + """Split every enabled range into tuner steps. + + The tuner is deliberately *not* centred on the span it covers. An RTL2832 + always shows a DC spike at whatever it is tuned to, so a centred step would + blank a hole in the middle of every range -- and for a range narrower than + one capture, that hole is the whole point of interest. Instead each step + parks the local oscillator ``dc_guard`` below the span it is responsible + for, so the covered frequencies sit entirely on one side of DC, clear of + both the spike and the anti-alias roll-off. + + ``tunable`` is an optional predicate; when the offset LO would fall outside + the hardware's range the step falls back to centring. + """ + if sample_rate <= 0 or usable_fraction <= 0: + raise RangeError("sample rate and usable fraction must be positive") + # Half the usable width, because only one side of DC is used per step. + usable = float(sample_rate) * float(usable_fraction) / 2.0 + if usable <= dc_guard: + raise RangeError("usable bandwidth is too small for the DC guard band") + + steps: list[TuneStep] = [] + for i, r in enumerate(ranges): + if not r.enabled: + continue + n = max(1, int(math.ceil(r.span / usable))) + if len(steps) + n > max_steps: + raise RangeError( + f"plan would need over {max_steps} tuner steps; " + "narrow the ranges or raise the sample rate" + ) + width = r.span / n + for k in range(n): + low = r.start + k * width + high = low + width + center = low - dc_guard + if tunable is not None and not tunable(center): + center = 0.5 * (low + high) # fall back rather than lose the step + steps.append(TuneStep( + center=center, + low=low, high=high, + range_index=i, range_label=r.label, + mode=r.mode, index=len(steps), + )) + return steps + + +def plan_summary(ranges: list[ScanRange], steps: list[TuneStep], + dwell_seconds: float) -> str: + total = sum(r.span for r in ranges if r.enabled) + n_en = sum(1 for r in ranges if r.enabled) + cycle = len(steps) * dwell_seconds + return (f"{n_en} range(s), {fmt_hz(total)} total span, {len(steps)} tuner " + f"steps, ~{cycle:.1f} s per full sweep") diff --git a/bandsaunter/recorder.py b/bandsaunter/recorder.py new file mode 100755 index 0000000..14cf55f --- /dev/null +++ b/bandsaunter/recorder.py @@ -0,0 +1,591 @@ +"""Writing hits to disk: audio, raw IQ, metadata sidecars and the run log.""" + +from __future__ import annotations + +import json +import time +import wave +from dataclasses import dataclass, field, asdict +from datetime import datetime, timezone +from pathlib import Path + +import numpy as np + +from .bandplan import fmt_hz, presets_covering + +__all__ = ["Recording", "ScanLog", "HitRecord", "safe_freq_name", + "build_stem"] + + +def safe_freq_name(hz: float) -> str: + """A filename-friendly frequency: 146520000 Hz -> ``0146.520000MHz``. + + Padded to four digits so that names sort by frequency as text. Without + it a directory listing puts 1090 MHz before 146 MHz, since "1" sorts + before "4". + """ + return f"{float(hz) / 1e6:011.6f}MHz" + + +def build_stem(started_at: float, frequency: float, modulation: str) -> str: + """``146.520000MHz--2026-08-20_14_23_05-nfm`` -- one flat name per capture. + + Every artefact of a capture (audio, IQ, metadata, transcript) shares this + stem, so a recording and its sidecars sit next to each other. Frequency + leads, so a directory listing groups everything heard on one channel + together, and the timestamp orders each group. + """ + when = datetime.fromtimestamp(started_at).strftime("%Y-%m-%d_%H_%M_%S") + mod = "".join(c if (c.isalnum() or c in "+") else "-" + for c in (modulation or "unknown").lower()).strip("-") + return f"{safe_freq_name(frequency)}--{when}-{mod or 'unknown'}" + + +@dataclass +class HitRecord: + """One detection, from first trigger to final classification.""" + + frequency: float + started_at: float + started_iso: str = "" + duration: float = 0.0 + peak_dbfs: float = -999.0 + snr_db: float = 0.0 + bandwidth: float = 0.0 + mode: str = "nfm" + range_label: str = "" + band_labels: list[str] = field(default_factory=list) + + classification: str = "" + family: str = "" + confidence: float = 0.0 + reasons: list[str] = field(default_factory=list) + alternatives: list = field(default_factory=list) + + morse_text: str = "" + morse_wpm: float = 0.0 + ctcss_hz: float = 0.0 + baud: float = 0.0 + + category: str = "" # voice / cw / digital / carrier / noise + signal_score: float = 0.0 + voice_score: float = 0.0 + noise_likeness: float = 0.0 + content_reason: str = "" + + filename: str = "" + audio_path: str = "" + combined_path: str = "" + transcript_path: str = "" + iq_path: str = "" + meta_path: str = "" + directory: str = "" + stop_reason: str = "" + kept: bool = True + features: dict = field(default_factory=dict) + + def to_dict(self) -> dict: + return asdict(self) + + def one_line(self) -> str: + bits = [f"{fmt_hz(self.frequency):>14s}", + f"{self.duration:5.1f}s", + f"SNR {self.snr_db:5.1f} dB", + self.classification or "unclassified"] + if self.morse_text: + bits.append(f'CW "{self.morse_text.strip()[:40]}"') + elif self.ctcss_hz: + bits.append(f"CTCSS {self.ctcss_hz:.1f}") + return " ".join(bits) + + +class Recording: + """Streams one detection to disk. + + Audio and IQ are appended block by block, so a long recording never has to + fit in memory. A bounded slice of IQ is kept in RAM for the classifier. + """ + + def __init__(self, root: Path, frequency: float, mode: str, + audio_rate: int, iq_rate: float | None = None, + save_audio: bool = True, save_iq: bool = False, + iq_format: str = "cf32", + classify_seconds: float = 10.0, + range_label: str = ""): + self.frequency = float(frequency) + self.mode = mode + self.audio_rate = int(audio_rate) + self.iq_rate = float(iq_rate or 0.0) + self.save_audio = save_audio + self.save_iq = save_iq + self.iq_format = iq_format + self.range_label = range_label + + self.started_at = time.time() + self.dir = Path(root) + self.dir.mkdir(parents=True, exist_ok=True) + # Provisional name: the modulation is not known until the capture has + # been analysed, so the files are renamed when the hit is finalised. + self.stem = build_stem(self.started_at, frequency, mode) + + self._wav: wave.Wave_write | None = None + self._iq_fh = None + self.audio_samples = 0 + self.iq_samples = 0 + + # Bounded buffer of IF-rate IQ for classification. Capped by sample + # count as well as duration so a wideband capture cannot balloon. + self._class_budget = min( + 2_000_000, int(max(1.0, classify_seconds) * (self.iq_rate or 1))) + self._class_chunks: list[np.ndarray] = [] + self._class_have = 0 + + # Bounded buffer of audio for the Morse decoder. + self._audio_budget = int(30.0 * self.audio_rate) + self._audio_chunks: list[np.ndarray] = [] + self._audio_have = 0 + # The same audio, but only the stretches where the squelch said a + # signal was present. A short over followed by the hang tail is half + # silence, and judging speech across the silence dilutes it away. + self._active_chunks: list[np.ndarray] = [] + self._active_marks: list[float] = [] # elapsed time at each chunk's end + self._active_have = 0 + + self.peak_dbfs = -999.0 + self.snr_db = 0.0 + self.bandwidth = 0.0 + + # -- paths ----------------------------------------------------------- + @property + def audio_path(self) -> Path: + return self.dir / f"{self.stem}.wav" + + @property + def iq_path(self) -> Path: + ext = "cf32" if self.iq_format == "cf32" else "cs16" + return self.dir / f"{self.stem}.{ext}" + + @property + def meta_path(self) -> Path: + return self.dir / f"{self.stem}.json" + + @property + def sigmf_path(self) -> Path: + return self.dir / f"{self.stem}.sigmf-meta" + + def rename_for(self, modulation: str) -> str: + """Rename everything written so far to reflect the identified mode.""" + new_stem = build_stem(self.started_at, self.frequency, modulation) + if new_stem == self.stem: + return self.stem + # Never clobber an existing capture: two signals can land in the same + # second on the same frequency when a scan re-visits quickly. + candidate, n = new_stem, 2 + while (self.dir / f"{candidate}.wav").exists() and candidate != self.stem: + candidate = f"{new_stem}_{n}" + n += 1 + for suffix in (".wav", ".cf32", ".cs16", ".sigmf-meta", ".json"): + old_p = self.dir / f"{self.stem}{suffix}" + if old_p.exists(): + try: + old_p.rename(self.dir / f"{candidate}{suffix}") + except OSError: + return self.stem + self.stem = candidate + return self.stem + + # -- writing --------------------------------------------------------- + def _open_wav(self): + if self._wav is None: + self._wav = wave.open(str(self.audio_path), "wb") + self._wav.setnchannels(1) + self._wav.setsampwidth(2) + self._wav.setframerate(self.audio_rate) + return self._wav + + def write_audio(self, audio: np.ndarray, present: bool = True, + at: float = -1.0) -> None: + if audio.size == 0: + return + if self.save_audio: + pcm = np.clip(audio, -1.0, 1.0) + pcm = (pcm * 32767.0).astype("= 0 else self.audio_samples / max(self.audio_rate, 1)) + self._active_have += audio.size + + def write_iq(self, iq: np.ndarray) -> None: + """Store IF-rate complex samples (for the file and the classifier).""" + if iq.size == 0: + return + if self.save_iq: + if self._iq_fh is None: + self._iq_fh = open(self.iq_path, "wb") + if self.iq_format == "cs16": + inter = np.empty(iq.size * 2, dtype=" np.ndarray: + """Buffered IF-rate IQ, optionally only the most recent ``limit``. + + The limit matters: the periodic content check runs inside the capture + loop, and analysing several megasamples there stalls the reader long + enough for the dongle to drop what arrives meanwhile. + """ + if not self._class_chunks: + return np.zeros(0, dtype=np.complex64) + if limit and self._class_have > limit: + taken, total = [], 0 + for chunk in reversed(self._class_chunks): + taken.append(chunk) + total += chunk.size + if total >= limit: + break + return np.concatenate(list(reversed(taken)))[-limit:] + return np.concatenate(self._class_chunks) + + def active_since(self, t0: float) -> np.ndarray: + """Audio the squelch called signal, from ``t0`` seconds onward. + + Judging a live capture on everything heard so far means the speech at + the start keeps vouching for the channel long after it has gone quiet. + A recent window is what answers "is there still something here". + """ + if not self._active_chunks: + return np.zeros(0, dtype=np.float32) + keep = [c for c, mark in zip(self._active_chunks, self._active_marks) + if mark >= t0] + if not keep: + return np.zeros(0, dtype=np.float32) + return np.concatenate(keep) + + def classification_audio(self, active_only: bool = True, + limit: int = 0) -> np.ndarray: + """Audio for content analysis. + + Defaults to only the stretches the squelch called signal, falling back + to everything when that is too short to judge. + """ + chunks = (self._active_chunks + if active_only and self._active_have >= int(0.6 * self.audio_rate) + else self._audio_chunks) + if not chunks: + return np.zeros(0, dtype=np.float32) + out = np.concatenate(chunks) + return out[-limit:] if limit and out.size > limit else out + + @property + def duration(self) -> float: + if self.audio_rate and self.audio_samples: + return self.audio_samples / self.audio_rate + if self.iq_rate and self.iq_samples: + return self.iq_samples / self.iq_rate + return time.time() - self.started_at + + # -- finishing -------------------------------------------------------- + def close(self) -> None: + if self._wav is not None: + self._wav.close() + self._wav = None + if self._iq_fh is not None: + self._iq_fh.close() + self._iq_fh = None + + def discard(self) -> None: + """Delete everything this recording wrote (too short, or squelched out).""" + self.close() + for p in (self.audio_path, self.iq_path, self.meta_path, + self.sigmf_path): + try: + if p.exists(): + p.unlink() + except OSError: + pass + + def write_metadata(self, hit: HitRecord) -> Path: + meta = { + "bandsaunter_version": 1, + "frequency_hz": self.frequency, + "frequency": fmt_hz(self.frequency), + "mode": self.mode, + "started_utc": datetime.fromtimestamp( + self.started_at, timezone.utc).isoformat(), + "started_local": datetime.fromtimestamp(self.started_at).isoformat(), + "duration_seconds": round(self.duration, 3), + "audio_rate": self.audio_rate, + "iq_rate": self.iq_rate, + "audio_samples": self.audio_samples, + "iq_samples": self.iq_samples, + "range_label": self.range_label, + "bands": [p.name for p in presets_covering(self.frequency)], + "hit": hit.to_dict(), + } + with open(self.meta_path, "w") as fh: + json.dump(meta, fh, indent=2, default=_json_default) + if self.save_iq and self.iq_samples: + self._write_sigmf(hit) + return self.meta_path + + def _write_sigmf(self, hit: HitRecord) -> None: + """A SigMF sidecar so the IQ opens in standard SDR tooling.""" + sigmf = { + "global": { + "core:datatype": "cf32_le" if self.iq_format == "cf32" else "ci16_le", + "core:sample_rate": self.iq_rate, + "core:version": "1.0.0", + "core:description": hit.classification or "bandsaunter capture", + "core:recorder": "bandsaunter", + }, + "captures": [{ + "core:sample_start": 0, + "core:frequency": self.frequency, + "core:datetime": datetime.fromtimestamp( + self.started_at, timezone.utc).isoformat().replace("+00:00", "Z"), + }], + "annotations": [{ + "core:sample_start": 0, + "core:sample_count": self.iq_samples, + "core:freq_lower_edge": self.frequency - hit.bandwidth / 2, + "core:freq_upper_edge": self.frequency + hit.bandwidth / 2, + "core:label": hit.classification or "signal", + }], + } + with open(self.sigmf_path, "w") as fh: + json.dump(sigmf, fh, indent=2, default=_json_default) + + +def _json_default(o): + if isinstance(o, (np.floating, np.integer)): + return o.item() + if isinstance(o, np.ndarray): + return o.tolist() + if isinstance(o, (set, tuple)): + return list(o) + return str(o) + + +class ScanLog: + """Append-only JSONL log plus a human-readable CSV of every hit.""" + + CSV_COLUMNS = ("timestamp", "frequency_hz", "frequency", "duration_s", + "snr_db", "bandwidth_hz", "mode", "category", + "classification", "confidence", "signal_score", + "morse_text", "ctcss_hz", "baud", "range", "filename") + + def __init__(self, root: Path, jsonl_name: str = "scan_log.jsonl"): + self.root = Path(root) + self.root.mkdir(parents=True, exist_ok=True) + self.jsonl = self.root / jsonl_name + self.csv = self.root / (Path(jsonl_name).stem + ".csv") + if not self.csv.exists(): + with open(self.csv, "w") as fh: + fh.write(",".join(self.CSV_COLUMNS) + "\n") + + def append(self, hit: HitRecord) -> None: + with open(self.jsonl, "a") as fh: + fh.write(json.dumps(hit.to_dict(), default=_json_default) + "\n") + row = [ + datetime.fromtimestamp(hit.started_at).isoformat(timespec="seconds"), + f"{hit.frequency:.0f}", fmt_hz(hit.frequency), + f"{hit.duration:.2f}", f"{hit.snr_db:.1f}", f"{hit.bandwidth:.0f}", + hit.mode, hit.category, hit.classification, + f"{hit.confidence:.2f}", f"{hit.signal_score:.2f}", + hit.morse_text.strip(), f"{hit.ctcss_hz:.1f}", f"{hit.baud:.0f}", + hit.range_label, hit.filename, + ] + with open(self.csv, "a") as fh: + fh.write(",".join(_csv_escape(str(v)) for v in row) + "\n") + + +def _csv_escape(v: str) -> str: + if any(c in v for c in ',"\n'): + return '"' + v.replace('"', '""') + '"' + return v + + +# --------------------------------------------------------------------------- +# One file per frequency, appended to as the scan goes on +# --------------------------------------------------------------------------- + +_WAV_HEADER = 44 # canonical PCM header: RIFF + fmt + data + + +def append_wav(path: Path, audio: np.ndarray, rate: int) -> int: + """Append mono audio to a WAV, creating it if needed. + + The file is left valid after every append -- sizes in the header are + rewritten each time -- so a long-running scan can be interrupted at any + point and the recording so far still plays. The ``wave`` module cannot + do this: it only writes its header when the file is closed. + """ + pcm = (np.clip(np.asarray(audio, dtype=np.float32), -1.0, 1.0) + * 32767.0).astype(" bytes: + byte_rate = rate * channels * bits // 8 + block_align = channels * bits // 8 + return b"".join(( + b"RIFF", (36 + data_bytes).to_bytes(4, "little"), b"WAVE", + b"fmt ", (16).to_bytes(4, "little"), (1).to_bytes(2, "little"), + channels.to_bytes(2, "little"), rate.to_bytes(4, "little"), + byte_rate.to_bytes(4, "little"), block_align.to_bytes(2, "little"), + bits.to_bytes(2, "little"), + b"data", data_bytes.to_bytes(4, "little"), + )) + + +def read_wav(path: Path) -> tuple[np.ndarray, int]: + """Read a mono 16-bit WAV back as floats.""" + with wave.open(str(path)) as w: + rate = w.getframerate() + channels = w.getnchannels() + raw = w.readframes(w.getnframes()) + audio = np.frombuffer(raw, dtype=" 1: + audio = audio.reshape(-1, channels).mean(axis=1) + return audio, rate + + +def _resample(audio: np.ndarray, src: int, dst: int) -> np.ndarray: + if src == dst or audio.size == 0: + return audio + from math import gcd + from scipy.signal import resample_poly + g = gcd(int(src), int(dst)) + return resample_poly(audio, int(dst) // g, int(src) // g).astype(np.float32) + + +class FrequencyLog: + """Collects every transmission heard on one frequency into one file. + + Each capture is appended after a spoken timestamp, so the result plays + back as a running commentary of what was heard and when. + """ + + def __init__(self, root: Path, tolerance_hz: float = 6_250.0, + announce: bool = True, announce_frequency: bool = False, + engine: str = "auto", gap_seconds: float = 0.4): + self.root = Path(root) + self.root.mkdir(parents=True, exist_ok=True) + self.tolerance = float(tolerance_hz) + self.announce = announce + self.announce_frequency = announce_frequency + self.engine = engine + self.gap_seconds = gap_seconds + # centre frequency -> (path, sample rate) + self._groups: list[tuple[float, Path, int]] = [] + # The date only has to be said when it changes, per file. + self._last_date: dict[Path, object] = {} + self.appended = 0 + + # -- grouping --------------------------------------------------------- + def _group_for(self, frequency: float, rate: int) -> tuple[Path, int]: + best = None + best_delta = self.tolerance + for i, (centre, path, existing_rate) in enumerate(self._groups): + delta = abs(centre - frequency) + if delta <= best_delta: + best, best_delta = i, delta + if best is not None: + centre, path, existing_rate = self._groups[best] + return path, existing_rate + + path = self.root / f"{safe_freq_name(round(frequency / 100) * 100)}.wav" + if path.exists(): + # Carry on with a file from an earlier run rather than clashing + # with it: its sample rate is what everything must match. + try: + with wave.open(str(path)) as w: + rate = w.getframerate() + except (wave.Error, OSError): + pass + self._groups.append((frequency, path, rate)) + return path, rate + + # -- appending --------------------------------------------------------- + def add(self, frequency: float, audio: np.ndarray, rate: int, + when: datetime | None = None) -> Path: + """Append one transmission, preceded by its spoken timestamp.""" + when = when or datetime.now() + path, target_rate = self._group_for(frequency, int(rate)) + + pieces = [] + if self.announce: + from .announce import speak_timestamp + # Say the full date on the first transmission of each day and the + # time alone after that. Repeating the date before every short + # over would take longer to say than most of them last. + say_date = self._last_date.get(path) != when.date() + self._last_date[path] = when.date() + spoken = speak_timestamp( + when, float(target_rate), + frequency=frequency if self.announce_frequency else None, + engine=self.engine, with_date=say_date) + if spoken.size: + pieces.append(spoken) + pieces.append(np.zeros(int(self.gap_seconds * target_rate), + dtype=np.float32)) + + body = _resample(np.asarray(audio, dtype=np.float32), int(rate), + target_rate) + pieces.append(body) + pieces.append(np.zeros(int(self.gap_seconds * target_rate), + dtype=np.float32)) + + append_wav(path, np.concatenate(pieces), target_rate) + self.appended += 1 + return path + + def add_file(self, frequency: float, wav_path: Path, + when: datetime | None = None) -> Path: + audio, rate = read_wav(wav_path) + return self.add(frequency, audio, rate, when) + + @property + def files(self) -> list[Path]: + return [p for _, p, _ in self._groups] diff --git a/bandsaunter/scanner.py b/bandsaunter/scanner.py new file mode 100755 index 0000000..7b3f125 --- /dev/null +++ b/bandsaunter/scanner.py @@ -0,0 +1,1073 @@ +"""The scan engine. + +Sweeps the configured ranges, detects energy above a learned noise floor, +drops onto each hit to record and identify it, then carries on. The two dwell +rules from the configuration decide when to leave a signal: + +* ``record_seconds`` -- hard cap on how long one hit may hold the receiver +* ``hang_seconds`` -- how long the channel must stay quiet before resuming +""" + +from __future__ import annotations + +import math +import threading +import time +from datetime import datetime +from dataclasses import dataclass, field +from pathlib import Path + +import numpy as np + +from . import dsp +from .bandplan import fmt_hz, presets_covering +from .classify import classify +from .config import ScanConfig +from .demod import make_demodulator +from .device import RtlSdrDevice, RtlSdrError +from .morse import decode_morse +from .quality import Assessment, assess +from .ranges import TuneStep, build_plan +from .recorder import FrequencyLog, HitRecord, Recording, ScanLog, read_wav +from .transcribe import TranscriptionWorker, available_engine + +__all__ = ["Scanner", "Detection", "ScanStats", "ScannerCallbacks"] + + +@dataclass +class Detection: + """A candidate signal found in one tuner step's spectrum.""" + + frequency: float + peak_dbfs: float + snr_db: float + bandwidth: float + step: TuneStep + range_index: int = 0 + + def describe(self) -> str: + return (f"{fmt_hz(self.frequency)} SNR {self.snr_db:.1f} dB " + f"BW {fmt_hz(self.bandwidth)}") + + +@dataclass +class ScanStats: + started_at: float = field(default_factory=time.time) + steps_done: int = 0 + cycles: int = 0 + detections: int = 0 + recordings: int = 0 + discarded: int = 0 + seconds_recorded: float = 0.0 + current_freq: float = 0.0 + current_range: str = "" + state: str = "idle" + dropped_samples: int = 0 + truncated: int = 0 # captures cut off mid-transmission + rejected_by_category: dict = field(default_factory=dict) + + @property + def elapsed(self) -> float: + return time.time() - self.started_at + + +@dataclass +class ScannerCallbacks: + """Hooks the UI plugs into. All are optional and must not raise.""" + + on_step: callable = None # (step_index, n_steps, TuneStep, psd_db, freqs) + on_detection: callable = None # (Detection) + on_record_start: callable = None # (Detection, Recording) + on_record_tick: callable = None # (Recording, elapsed, signal_present, snr) + on_record_end: callable = None # (HitRecord) + on_cycle: callable = None # (cycle_number) + on_status: callable = None # (str) + on_error: callable = None # (Exception) + + +class Scanner: + """Runs a scan on one device. Drive it with :meth:`run`.""" + + def __init__(self, config: ScanConfig, device: RtlSdrDevice | None = None, + callbacks: ScannerCallbacks | None = None): + self.cfg = config + self.device = device + self._own_device = device is None + self.cb = callbacks or ScannerCallbacks() + self.stats = ScanStats() + + self.plan: list[TuneStep] = [] + self._floors: dict[int, dsp.NoiseFloorTracker] = {} + self._recent: dict[int, float] = {} # frequency bucket -> last visit + self._lockouts: set[int] = set() + + self._stop = threading.Event() + self._pause = threading.Event() + self._skip = threading.Event() + self._lock = threading.Lock() + + self.log: ScanLog | None = None + self.frequency_log: FrequencyLog | None = None + self.transcriber: TranscriptionWorker | None = None + self.hits: list[HitRecord] = [] + self.nfft = 1024 + self.detector_bias = 0.0 + self._warned_record_limit = False + self._bucket = max(1.0, self.cfg.lockout_width) + + # -- control --------------------------------------------------------- + def stop(self) -> None: + self._stop.set() + + def pause(self, on: bool = True) -> None: + self._pause.set() if on else self._pause.clear() + + @property + def paused(self) -> bool: + return self._pause.is_set() + + def skip(self) -> None: + """Abandon the current recording and resume sweeping.""" + self._skip.set() + + def lockout(self, freq_hz: float) -> None: + with self._lock: + self._lockouts.add(self._key(freq_hz)) + self.cfg.lockout.append(float(freq_hz)) + + def _key(self, freq_hz: float) -> int: + return int(round(freq_hz / self._bucket)) + + def _is_locked(self, freq_hz: float) -> bool: + return self._key(freq_hz) in self._lockouts + + # -- setup ------------------------------------------------------------- + def prepare(self) -> None: + errs = self.cfg.validate() + if errs: + raise ValueError("configuration problems:\n - " + "\n - ".join(errs)) + + if self.device is None: + self.device = RtlSdrDevice( + index=self.cfg.device_index, + sample_rate=self.cfg.sample_rate, + gain=self.cfg.gain, + ppm=self.cfg.ppm, + agc=self.cfg.agc, + bias_tee=self.cfg.bias_tee, + offset_tuning=self.cfg.offset_tuning, + direct_sampling=self.cfg.direct_sampling, + ) + self.device.open() + self.cfg.sample_rate = self.device.sample_rate + + self.plan = build_plan(self.cfg.ranges, self.device.sample_rate, + self.cfg.usable_fraction, + dc_guard=self.cfg.dc_guard_hz, + tunable=self.device.can_tune) + if not self.plan: + raise ValueError("no tunable steps: every range is disabled or empty") + + # Drop steps the hardware cannot reach rather than failing mid-sweep. + reachable = [s for s in self.plan if self.device.can_tune(s.center)] + skipped = len(self.plan) - len(reachable) + if not reachable: + raise ValueError( + "none of the configured frequencies are within this device's " + "tuning range" + ) + if skipped: + self._status(f"skipping {skipped} step(s) outside the tuner's range") + self.plan = reachable + + self.nfft = dsp.next_fast_len( + max(64, int(self.device.sample_rate / max(1.0, self.cfg.resolution_hz))) + ) + # Offset that noise alone clears with this detector, so the configured + # threshold means real margin over static rather than over the floor. + n_samples = self._sweep_samples() + n_seg = min(64, 1 + max(0, (n_samples - self.nfft)) // max(1, self.nfft // 2)) + self.detector_bias = (self.cfg.detector_bias_db + if self.cfg.detector_bias_db is not None + else dsp.detector_bias_db( + n_seg, self.nfft, + "max" if self.cfg.detector == "peak" else "mean")) + for f in self.cfg.lockout: + self._lockouts.add(self._key(f)) + + root = Path(self.cfg.output_dir).expanduser() + root.mkdir(parents=True, exist_ok=True) + self.log = ScanLog(root, self.cfg.log_file) + if self.cfg.transcribe: + engine = (available_engine() + if self.cfg.transcribe_engine in ("", "auto") + else self.cfg.transcribe_engine) + if engine is None: + self._status( + "transcription is on but no speech recogniser is " + "installed — see `bandsaunter transcribe --engines`") + else: + self.transcriber = TranscriptionWorker( + engine=self.cfg.transcribe_engine, + model=self.cfg.transcribe_model, + language=self.cfg.transcribe_language, + on_error=self._error) + self.transcriber.start() + self._status(f"transcribing speech with {engine}") + + if self.cfg.combine_by_frequency: + self.frequency_log = FrequencyLog( + root, tolerance_hz=self.cfg.combine_tolerance_hz, + announce=self.cfg.announce_timestamps, + announce_frequency=self.cfg.announce_frequency, + engine=self.cfg.announce_engine) + + def _status(self, msg: str) -> None: + if self.cb.on_status: + try: + self.cb.on_status(msg) + except Exception: + pass + + def _error(self, exc: Exception) -> None: + if self.cb.on_error: + try: + self.cb.on_error(exc) + except Exception: + pass + + # -- sweeping ---------------------------------------------------------- + @property + def bin_hz(self) -> float: + return self.device.sample_rate / self.nfft + + def _sweep_samples(self) -> int: + return max(self.nfft * 4, + int(self.device.sample_rate * self.cfg.dwell_seconds)) + + def measure_step(self, step: TuneStep): + """Tune, capture and measure one step. + + Returns ``(freqs_hz, psd_db, excess_db, margin_db)`` where ``excess`` + is dB above the noise floor (what gets reported as SNR) and ``margin`` + is the extra headroom a detection must clear on top of the configured + threshold, derived from how much the noise in that bin scatters. + """ + self.device.tune(step.center) + x = self.device.read_samples(self._sweep_samples(), flush=True) + # Strip the RTL2832's constant DC offset before it becomes a spike. + x = x - x.mean() + freqs_norm, psd = dsp.welch_psd( + x, self.nfft, combine="max" if self.cfg.detector == "peak" else "mean") + psd_db = dsp.db(psd) + freqs = step.center + freqs_norm * self.device.sample_rate + + if self.cfg.adaptive_floor: + tracker = self._floors.get(step.index) + if tracker is None: + tracker = self._floors[step.index] = dsp.NoiseFloorTracker() + tracker.update(psd_db) + excess = tracker.excess(psd_db) + else: + excess = psd_db - dsp.noise_floor_curve(psd_db) + return freqs, psd_db, excess, self.detector_bias + + def find_detections(self, step: TuneStep, freqs: np.ndarray, + psd_db: np.ndarray, excess: np.ndarray, + threshold_db: float, + margin: np.ndarray | float = 0.0) -> list[Detection]: + """Group bins above threshold into distinct signals.""" + n = freqs.size + mask = np.ones(n, dtype=bool) + + # Only trust the flat middle of the capture, and only the slice of it + # this step is responsible for. + mask &= np.abs(freqs - step.center) <= 0.45 * self.device.sample_rate + mask &= (freqs >= step.low) & (freqs <= step.high) + # Belt and braces: the planner already parks the LO outside the covered + # span, but never trust bins next to DC. + dc_guard = max(2.0 * self.bin_hz, 0.5 * self.cfg.dc_guard_hz) + mask &= np.abs(freqs - step.center) > dc_guard + + hot = mask & (excess > (threshold_db + margin)) + if not np.any(hot): + return [] + + idx = np.flatnonzero(hot) + groups: list[list[int]] = [[idx[0]]] + for i in idx[1:]: + if i - groups[-1][-1] <= 2: # bridge single-bin dropouts + groups[-1].append(i) + else: + groups.append([i]) + + dets = [] + for g in groups: + lo_i, hi_i = g[0], g[-1] + band = slice(lo_i, hi_i + 1) + weights = np.maximum(excess[band], 0.0) + if weights.sum() <= 0: + continue + centroid = float(np.average(freqs[band], weights=weights)) + peak_i = lo_i + int(np.argmax(psd_db[band])) + bw = max(self.bin_hz, (hi_i - lo_i + 1) * self.bin_hz) + dets.append(Detection( + frequency=centroid, + peak_dbfs=float(psd_db[peak_i]), + snr_db=float(excess[peak_i]), + bandwidth=bw, + step=step, + range_index=step.range_index, + )) + + dets.sort(key=lambda d: d.snr_db, reverse=True) + return dets[: max(1, self.cfg.max_detections_per_step)] + + def _should_visit(self, det: Detection) -> bool: + if self._is_locked(det.frequency): + return False + last = self._recent.get(self._key(det.frequency)) + if last is not None and (time.time() - last) < self.cfg.revisit_seconds: + return False + return True + + # -- capture ------------------------------------------------------------ + def _measure_bandwidth(self, x: np.ndarray, hint_bw: float) -> float: + """Fine-resolution occupied bandwidth of a signal already at DC. + + The sweep runs at a few kHz per bin, which cannot tell a CW carrier + from an SSB channel. One short probe at ~60 Hz resolution can. + """ + nfft = min(32768, dsp.next_fast_len( + int(self.device.sample_rate / 100.0))) + if x.size < nfft: + return hint_bw + _, psd = dsp.welch_psd(x, nfft) + psd_db = dsp.db(psd) + bin_hz = self.device.sample_rate / nfft + freqs = np.fft.fftshift( + np.fft.fftfreq(nfft, 1.0 / self.device.sample_rate)) + win = np.abs(freqs) <= max(hint_bw * 3.0, 30_000.0) + if not np.any(win): + return hint_bw + floor = float(np.percentile(psd_db[win], 25.0)) + peak = float(psd_db[win].max()) + # Relative to the noise floor *and* to the peak. On a strong signal a + # floor-relative threshold alone follows the skirts a long way out and + # reports a channel several times wider than it is -- which then opens + # the demodulator far too wide and fills the audio with noise. + hot = win & (psd_db > max(floor + 10.0, peak - 26.0)) + if not np.any(hot): + return hint_bw + idx = np.flatnonzero(hot) + # Percentiles of the occupied positions, not their extremes: a single + # stray bin out at the edge of the window would otherwise stretch the + # measurement by an order of magnitude, and the demodulator opened to + # match would hear nothing but the carrier. + lo = float(np.percentile(idx, 2.0)) + hi = float(np.percentile(idx, 98.0)) + return max(bin_hz, float((hi - lo + 1.0) * bin_hz)) + + def _capture_plan(self, det: Detection, + fine_bw: float | None = None) -> tuple[str, float]: + """Pick the demodulator and bandwidth for a hit.""" + rng = self.cfg.ranges[det.range_index] if \ + det.range_index < len(self.cfg.ranges) else None + if rng is not None: + mode = rng.resolved_mode(det.frequency) + bw = rng.resolved_bandwidth(det.frequency) + else: + covering = presets_covering(det.frequency) + best = min(covering, key=lambda p: p.span) if covering else None + mode = best.mode if best else "nfm" + bw = best.bandwidth if best else 12_500.0 + measured = fine_bw if fine_bw is not None else det.bandwidth + # A very narrow signal inside an SSB segment is a CW carrier, and an + # SSB filter would reject it outright (it sits at DC once centred). + # Switching to the CW demodulator puts a beat note back on it. + if mode in ("usb", "lsb", "ssb") and measured < 1_200.0: + return "cw", 800.0 + # Never demodulate narrower than the signal actually measured. + bw = max(bw, min(measured * 1.5, self.device.sample_rate * 0.4)) + return mode, bw + + def _demod_from_signal(self, probe: np.ndarray, freq_hz: float, + fallback: str, bw: float, + signal_bw: float = 0.0, + snr_db: float = 0.0) -> str: + """Choose the demodulator from what the signal physically is. + + Three measurements settle it, and all three are ratios that hold + steady over a fraction of a second regardless of what is being said: + + * how much the envelope varies, + * how far the instantaneous frequency swings, + * how much of the power sits in a single carrier bin. + + Running the full classifier on the probe was tried and is not + reliable here -- over a fraction of a second, speech makes any + modulation look bursty, and AM came back as on-off keying while FM + came back as AM. The band plan is kept whenever the evidence is not + decisive, since it is usually right about what a band carries. + """ + if probe.size < 4096 or snr_db < 12.0: + return fallback + + # Narrow to the signal's own bandwidth first. Across the full 2 MHz + # capture a narrow channel is a sliver, and the measurements describe + # the surrounding noise instead of the signal. + rate = float(self.device.sample_rate) + target = max(4.0 * (signal_bw or bw), 16_000.0) + factor = 1 + while factor * 2 <= 256 and rate / (factor * 2) >= target: + factor *= 2 + if factor > 1: + try: + probe = dsp.DecimationChain(factor)(probe) + rate = rate / factor + except Exception: + return fallback + if probe.size < 4096: + return fallback + + env = np.abs(probe) + mean_env = float(env.mean()) + if mean_env <= 0: + return fallback + env_cv = float(env.std() / mean_env) + + ifreq = dsp.instantaneous_frequency(probe, rate) + strong = env[1:] > 0.5 * mean_env + sel = ifreq[strong] if strong.sum() > 64 else ifreq + fdev_rms = float(np.std(sel)) + + _, psd = dsp.welch_psd(probe, 1024) + total = float(psd.sum()) + carrier_ratio = float(psd.max() / total) if total > 0 else 0.0 + + measured_bw = signal_bw or bw + # Deliberately absolute, not scaled by the measured bandwidth: that + # measurement is an estimate, and dividing by it turned a stable + # decision into one that moved with the estimate's error. + if env_cv < 0.15 and fdev_rms > 200.0: + # Constant envelope with the tone swinging: frequency modulation, + # which is also how FSK is conventionally listened to. + return "wfm" if measured_bw > 50_000 else "nfm" + if 0.05 < env_cv < 0.6 and carrier_ratio > 0.25 and fdev_rms < 200.0: + # A surviving carrier, a varying envelope, a tone that stays put. + return "am" + if carrier_ratio < 0.15 and env_cv > 0.6: + # No carrier left, and the envelope carrying the whole signal. + return "lsb" if freq_hz < 10_000_000 else "usb" + return fallback + + def _lo_offset(self, freq: float, bw: float) -> float: + """How far to offset the LO so the DC spike misses the signal.""" + if self.device.direct_sampling_mode != 0: + return 0.0 + offset = self.device.sample_rate * 0.25 + if not self.device.can_tune(freq - offset): + return 0.0 + if offset < bw: + return 0.0 + return offset + + def _band_snr(self, x: np.ndarray, bw: float) -> float: + """In-band vs guard-band power, in dB -- the squelch metric. + + Comparing the signal against the noise *right next to it* makes the + squelch immune to gain changes and to the overall band noise level. + """ + nfft = 512 + if x.size < nfft * 2: + return 0.0 + _, psd = dsp.welch_psd(x, nfft, max_segments=16) + bin_hz = self.device.sample_rate / nfft + freqs = np.fft.fftshift(np.fft.fftfreq(nfft, 1.0 / self.device.sample_rate)) + half = max(bin_hz, bw / 2.0) + inband = np.abs(freqs) <= half + guard = (np.abs(freqs) > 2.0 * half) & \ + (np.abs(freqs) <= self.device.sample_rate * 0.45) + if not np.any(inband) or not np.any(guard): + return 0.0 + sig = float(np.mean(psd[inband])) + noise = float(np.median(psd[guard])) + if noise <= 0: + return 0.0 + return float(10.0 * math.log10(max(sig / noise, 1e-12))) + + def capture(self, det: Detection) -> HitRecord | None: + """Drop onto a detection: record it, then work out what it is.""" + cfg = self.cfg + mode, bw = self._capture_plan(det) + offset = self._lo_offset(det.frequency, bw) + + self.device.tune(det.frequency - offset) + mixer = dsp.Mixer(offset, self.device.sample_rate) if offset else None + + # Probe before committing to a demodulator. The sweep's bin width is + # far too coarse to tell narrow modes apart, and the band plan only + # says what a frequency is *usually* used for -- an AM signal inside a + # band listed as FM would otherwise be recorded through the wrong + # detector, producing audio that is useless to listen to and + # impossible to judge for content. + # Stream from here on, so the probe and the recording are one + # continuous capture with no gap between them. + streamed = False + try: + self.device.start_stream() + streamed = True + except (RtlSdrError, AttributeError) as exc: + if isinstance(exc, RtlSdrError): + self._error(exc) + + probe = None + try: + n_probe = int(self.device.sample_rate * cfg.probe_seconds) + probe = (self.device.read_stream(n_probe) if streamed + else self.device.read_samples(n_probe, flush=True)) + if mixer is not None: + probe = mixer(probe) + # Strip the receiver's DC offset for the *measurements* only, and + # do it on the pre-mix signal. That offset sits at the tuned + # frequency, which the mixer moves a quarter of the sample rate + # away; subtracting the mean after mixing would instead delete + # whatever landed at DC -- the signal's own carrier, which is the + # very thing that identifies AM. + measured = probe - probe.mean() if mixer is None else probe + fine_bw = self._measure_bandwidth(measured, bw) + mode, bw = self._capture_plan(det, fine_bw) + det.bandwidth = fine_bw + mode = self._demod_from_signal(measured, det.frequency, mode, bw, + signal_bw=fine_bw, + snr_db=det.snr_db) + except RtlSdrError as exc: + self._error(exc) + + demod = make_demodulator(mode, self.device.sample_rate, bw, + cfg.audio_rate) + + rec = Recording( + root=Path(cfg.output_dir).expanduser(), + frequency=det.frequency, mode=mode, + audio_rate=demod.audio_rate, iq_rate=demod.if_rate, + save_audio=cfg.save_audio, save_iq=cfg.save_iq, + iq_format=cfg.iq_format, range_label=det.step.range_label, + ) + if self.cb.on_record_start: + try: + self.cb.on_record_start(det, rec) + except Exception: + pass + + block = max(4096, int(self.device.sample_rate * 0.05)) + block -= block % 512 + squelch_on = cfg.threshold_db + squelch_off = max(1.0, cfg.threshold_db - cfg.squelch_margin_db) + + # Elapsed time is counted from samples consumed, not the wall clock: + # "record for X seconds" then means X seconds of signal even if the + # host stalls, and the recorded file length matches the setting exactly. + fs = float(self.device.sample_rate) + start = time.time() + elapsed = 0.0 + last_signal_at = 0.0 + last_content_at = 0.0 + ever_signal = False + content_ever = False + checks_done = 0 + peak_snr = det.snr_db + stop_reason = "unknown" + next_check = cfg.verify_seconds + rejected_early = None + quiet_for = 0.0 + cut_short = False + self.stats.state = "recording" + self._recent[self._key(det.frequency)] = start + + # The probe is real, contiguous signal -- play it into the recording + # rather than discarding it. Throwing it away costs the opening of + # every transmission, which for a short over is most of it. + pending = [] + if probe is not None and probe.size: + pending = [probe[i:i + block] for i in range(0, probe.size, block)] + + try: + while True: + if self._stop.is_set(): + stop_reason = "scan stopped" + break + if self._skip.is_set(): + self._skip.clear() + stop_reason = "skipped by operator" + break + + if pending: + x = pending.pop(0) + else: + x = (self.device.read_stream(block) if streamed + else self.device.read_samples(block)) + if mixer is not None: + x = mixer(x) + elapsed += x.size / fs + + snr = self._band_snr(x, bw) + present = snr >= (squelch_off if ever_signal else squelch_on) + if present: + last_signal_at = elapsed + ever_signal = True + peak_snr = max(peak_snr, snr) + + audio, iq = demod.step(x) + rec.write_audio(audio, present=present, at=elapsed) + rec.write_iq(iq) + + if self.cb.on_record_tick: + try: + self.cb.on_record_tick(rec, elapsed, present, snr, + quiet_for) + except Exception: + pass + + # Judge the content while it is still arriving so static and + # interference are dropped after a second or two instead of + # holding the receiver for the full record time. + if (cfg.require_signal and cfg.classify and ever_signal + and elapsed >= next_check): + # Check more often once a gap has opened, so a channel + # held open by static is released promptly instead of + # waiting out a full verify interval. + in_gap = content_ever and \ + (elapsed - last_content_at) > cfg.hang_seconds + next_check = elapsed + (0.5 if in_gap else cfg.verify_seconds) + window = max(2.0, cfg.verify_seconds) + _, _, verdict = self._analyse( + rec, demod, det.frequency, peak_snr, + since=max(0.0, elapsed - window)) + if verdict is not None: + checks_done += 1 + if verdict.accept: + content_ever = True + last_content_at = elapsed + elif not content_ever: + # Static has no structure at all -- abandon at + # once. Anything else gets until + # verify_max_seconds to show some content. + # Once a capture *has* produced content it is + # never abandoned this way: a two-way exchange + # goes quiet between overs, and cutting it off + # mid-conversation is exactly wrong. + if (verdict.noise_likeness > 0.85 + or elapsed >= cfg.verify_max_seconds): + rejected_early = verdict + stop_reason = (f"no signal content: " + f"{verdict.reason}") + break + + if cfg.record_seconds and elapsed >= cfg.record_seconds: + stop_reason = f"reached the {cfg.record_seconds:g} s record limit" + if present: + # Cut off mid-transmission. This is the one case where + # the record limit is almost certainly not what the + # operator wanted, and it is invisible otherwise: the + # file simply ends. + cut_short = True + break + if elapsed >= cfg.max_record_seconds > 0: + stop_reason = (f"hit the {cfg.max_record_seconds:g} s safety " + "limit") + break + + # The gap that ends a capture is measured from the last real + # signal, not merely from the last time the squelch opened. + # Static and interference break squelch too, and letting them + # reset the timer would hold the receiver on a dead channel. + quiet_for = elapsed - last_signal_at + if content_ever and checks_done: + quiet_for = min(quiet_for, + elapsed - last_content_at + + min(cfg.verify_seconds, 1.0)) + content_gap = elapsed - last_content_at + if content_gap >= cfg.hang_seconds + min(cfg.verify_seconds, + 1.0): + stop_reason = (f"no signal for {cfg.hang_seconds:g} s " + "(only silence, static or interference)") + break + if ever_signal and (elapsed - last_signal_at) >= cfg.hang_seconds: + stop_reason = f"channel quiet for {cfg.hang_seconds:g} s" + break + if not ever_signal and elapsed >= max(0.4, min(2.0, cfg.hang_seconds)): + stop_reason = "signal gone before recording started" + break + except RtlSdrError as exc: + stop_reason = f"device error: {exc}" + self._error(exc) + finally: + if streamed: + dropped_before = getattr(self.device, "dropped_samples", 0) + try: + self.device.stop_stream() + except Exception: + pass + if dropped_before: + self.stats.dropped_samples += dropped_before + self._status( + f"host fell behind: {dropped_before} samples dropped " + "during this capture") + rec.close() + self.stats.state = "sweeping" + if cut_short: + self.stats.truncated += 1 + if not self._warned_record_limit: + self._warned_record_limit = True + self._status( + f"cut off at the {cfg.record_seconds:g} s record limit " + f"while still transmitting — set 'Record for' to 0 " + f"(--record 0) to follow a transmission to its end") + + duration = rec.duration + hit = HitRecord( + frequency=det.frequency, + started_at=rec.started_at, + started_iso=time.strftime("%Y-%m-%dT%H:%M:%S", + time.localtime(rec.started_at)), + duration=round(duration, 3), + peak_dbfs=round(det.peak_dbfs, 2), + snr_db=round(peak_snr, 2), + bandwidth=det.bandwidth, + mode=mode, + range_label=det.step.range_label, + band_labels=[p.name for p in presets_covering(det.frequency)], + directory=str(rec.dir), + filename=rec.stem, + stop_reason=stop_reason, + ) + + too_short = duration < cfg.min_record_seconds + if too_short or not ever_signal or rejected_early is not None: + hit.kept = False + if rejected_early is not None: + hit.category = rejected_early.category + hit.signal_score = round(rejected_early.score, 3) + hit.content_reason = rejected_early.reason + hit.noise_likeness = round(rejected_early.noise_likeness, 3) + hit.voice_score = round(rejected_early.voice.score, 3) + elif not ever_signal: + hit.category = "vanished" + hit.stop_reason = "false trigger (no signal after retuning)" + elif too_short: + hit.category = hit.category or "too short" + self._reject(rec, hit) + return hit + + verdict = self._identify(rec, hit, demod) + if cfg.require_signal and verdict is not None and not verdict.accept: + hit.kept = False + hit.stop_reason = f"no signal content: {verdict.reason}" + self._reject(rec, hit) + return hit + + # Name the files after what the signal turned out to be, not after the + # demodulator that was guessed before listening to it. + rec.rename_for(hit.family or hit.mode) + hit.filename = rec.stem + hit.audio_path = str(rec.audio_path) if cfg.save_audio else "" + hit.iq_path = str(rec.iq_path) if cfg.save_iq else "" + + # Add this transmission to the running file for its frequency, after + # a spoken timestamp, so one channel plays back as one recording. + if self.frequency_log is not None and cfg.save_audio and \ + rec.audio_path.exists(): + try: + combined = self.frequency_log.add_file( + hit.frequency, rec.audio_path, + when=datetime.fromtimestamp(rec.started_at)) + hit.combined_path = str(combined) + if not cfg.combine_keep_individual: + rec.audio_path.unlink() + hit.audio_path = "" + except (OSError, ValueError) as exc: + self._error(exc) + + hit.meta_path = str(rec.write_metadata(hit)) + # Queued after the sidecar exists, so the transcriber can record the + # result in it -- and record nothing when there was no speech. + self._submit_transcription(rec, hit) + self.stats.recordings += 1 + self.stats.seconds_recorded += duration + self.hits.append(hit) + if self.log: + try: + self.log.append(hit) + except OSError as exc: + self._error(exc) + if self.cb.on_record_end: + try: + self.cb.on_record_end(hit) + except Exception: + pass + return hit + + def _reject(self, rec: Recording, hit: HitRecord) -> None: + """Throw away a capture and everything it wrote.""" + rec.discard() + self.stats.discarded += 1 + self.stats.rejected_by_category[hit.category] = \ + self.stats.rejected_by_category.get(hit.category, 0) + 1 + if self.cb.on_record_end: + try: + self.cb.on_record_end(hit) + except Exception: + pass + + def _analyse(self, rec: Recording, demod, freq_hz: float, snr_db: float, + final: bool = False, since: float | None = None): + """Classify what has been captured so far and judge whether to keep it. + + Returns ``(classification, morse, assessment)``. Called periodically + during a capture so that static and interference can be abandoned + early, and once more at the end to make the final decision. + """ + # While the capture is running the check has to be cheap; once it has + # finished the stream is stopped and a longer look costs nothing. + iq_limit = 0 if final else min(int(2.0 * demod.if_rate), 200_000) + iq = rec.classification_iq(limit=iq_limit) + if iq.size < 2048: + return None, None, None + try: + cls = classify(iq, demod.if_rate, freq_hz=freq_hz, snr_db=snr_db) + except Exception as exc: + self._error(exc) + return None, None, None + + morse = None + if self.cfg.decode_morse and cls.family in ("cw", "ook", "carrier"): + morse = self._decode_cw(iq, demod) + + # Judge the audio that was actually recorded, and only that. An + # earlier version also demodulated a second way and kept whichever + # scored higher; that is cherry-picking, and on noise one of the two + # will flatter it by chance. It was the main source of false keeps. + if final or since is None: + audio = rec.classification_audio( + limit=0 if final else min(int(4.0 * rec.audio_rate), 96_000)) + else: + audio = rec.active_since(since) + return cls, morse, self._verdict(cls, audio, rec.audio_rate, morse, + freq_hz) + + def _verdict(self, cls, audio, audio_rate, morse, + freq_hz: float = 0.0) -> Assessment: + cfg = self.cfg + return assess(cls, audio, audio_rate, morse=morse, freq_hz=freq_hz, + min_voice=cfg.min_voice_score, + accept=tuple(cfg.accept), + min_score=cfg.min_signal_score) + + def _decode_cw(self, iq: np.ndarray, demod): + """Run a dedicated CW detector over the captured IQ. + + Independent of the recording mode: a keyed carrier is inaudible + through an FM or SSB detector, so Morse would otherwise be missed. + """ + try: + cw = make_demodulator("cw", int(demod.if_rate), 800.0, + self.cfg.audio_rate) + return decode_morse(cw.process(iq), cw.audio_rate) + except Exception as exc: + self._error(exc) + return None + + def _submit_transcription(self, rec: Recording, hit: HitRecord) -> None: + """Queue a voice capture for speech recognition. + + Only voice: running a recogniser over Morse or a data burst wastes + seconds per capture and produces nothing. + """ + worker = self.transcriber + if worker is None or hit.category != "voice": + return + if hit.duration < self.cfg.transcribe_min_seconds: + return + + audio, rate = None, rec.audio_rate + if rec.audio_path.exists(): + try: + audio, rate = read_wav(rec.audio_path) + except (OSError, ValueError) as exc: + self._error(exc) + if audio is None or not audio.size: + audio = rec.classification_audio(active_only=False) + if audio is None or not audio.size: + return + + # Beside the recording, sharing its name. When captures are being + # combined by frequency there is one recording per frequency, so the + # transcripts are appended to one file per frequency too. + if self.frequency_log is not None and hit.combined_path: + path = Path(hit.combined_path).with_suffix("") + path = path.with_name(path.name + "_transcription.txt") + append = True + else: + path = rec.dir / f"{rec.stem}_transcription.txt" + append = False + worker.submit(audio, rate, path, + datetime.fromtimestamp(rec.started_at), hit.frequency, + append=append, + meta_path=Path(hit.meta_path) if hit.meta_path else None) + + def _identify(self, rec: Recording, hit: HitRecord, demod, + analysis=None) -> Assessment | None: + """Attach the classification, Morse text and content verdict to a hit.""" + if not self.cfg.classify: + return None + cls, morse, verdict = analysis if analysis is not None else \ + self._analyse(rec, demod, hit.frequency, hit.snr_db, final=True) + if cls is None: + return None + + hit.classification = cls.label + hit.family = cls.family + hit.confidence = cls.confidence + hit.reasons = list(cls.reasons) + hit.alternatives = list(cls.alternatives) + f = cls.features + if f is not None: + hit.ctcss_hz = f.ctcss_hz + hit.baud = f.baud + hit.bandwidth = f.bandwidth or hit.bandwidth + hit.features = { + k: (round(v, 4) if isinstance(v, float) else v) + for k, v in f.__dict__.items() + if k != "extras" and not k.startswith("_") + } + + if morse is not None and morse.is_morse: + hit.morse_text = morse.text + hit.morse_wpm = round(morse.wpm, 1) + hit.classification = f"CW / Morse at {morse.wpm:.0f} WPM" + hit.family = "cw" + hit.confidence = max(hit.confidence, morse.confidence) + hit.reasons.insert(0, f'decoded Morse: "{morse.text.strip()}"') + elif morse is not None and cls.family == "cw": + hit.reasons.append( + "keyed carrier but the Morse timing did not resolve") + + if verdict is not None and verdict.category == "voice" and \ + hit.family not in ("nfm", "wfm", "am", "ssb"): + # The content check found speech, so the modulation label was + # wrong. Speech on a quiet FM channel is easy to mistake for + # two-level FSK -- during the pauses between phrases the only + # thing left modulating the carrier is the CTCSS tone, whose + # discriminator output has a peak at each end of its swing. + # Name it after what was actually heard. + spoken = {"nfm": ("Narrowband FM voice", "nfm"), + "wfm": ("Wideband FM broadcast", "wfm"), + "am": ("AM voice", "am"), + "usb": ("SSB voice (USB)", "ssb"), + "lsb": ("SSB voice (LSB)", "ssb")} + label, fam = spoken.get(hit.mode, ("Voice transmission", "nfm")) + if hit.ctcss_hz: + label += f" (CTCSS {hit.ctcss_hz:.1f} Hz)" + hit.reasons.insert(0, f"identified from its audio; the modulation " + f"measurements alone suggested " + f"{hit.classification.lower()}") + hit.classification = label + hit.family = fam + + if verdict is not None: + hit.category = verdict.category + hit.signal_score = round(verdict.score, 3) + hit.content_reason = verdict.reason + hit.voice_score = round(verdict.voice.score, 3) + hit.noise_likeness = round(verdict.noise_likeness, 3) + return verdict + + # -- main loop ----------------------------------------------------------- + def run(self) -> ScanStats: + if not self.plan: + self.prepare() + self.stats = ScanStats() + self.stats.state = "sweeping" + cfg = self.cfg + deadline = (self.stats.started_at + cfg.max_runtime_seconds + if cfg.max_runtime_seconds else None) + + try: + while not self._stop.is_set(): + for i, step in enumerate(self.plan): + if self._stop.is_set(): + break + while self._pause.is_set() and not self._stop.is_set(): + self.stats.state = "paused" + time.sleep(0.1) + if self._stop.is_set(): + break + self.stats.state = "sweeping" + + if deadline and time.time() >= deadline: + self._status("runtime limit reached") + self._stop.set() + break + + try: + freqs, psd_db, excess, margin = self.measure_step(step) + except RtlSdrError as exc: + self._error(exc) + time.sleep(0.2) + continue + + self.stats.steps_done += 1 + self.stats.current_freq = step.center + self.stats.current_range = step.range_label + if self.cb.on_step: + try: + self.cb.on_step(i, len(self.plan), step, psd_db, freqs) + except Exception: + pass + + rng = cfg.ranges[step.range_index] \ + if step.range_index < len(cfg.ranges) else None + thr = (rng.threshold_db if rng and rng.threshold_db is not None + else cfg.threshold_db) + + for det in self.find_detections(step, freqs, psd_db, + excess, thr, margin): + if self._stop.is_set(): + break + if not self._should_visit(det): + continue + self.stats.detections += 1 + if self.cb.on_detection: + try: + self.cb.on_detection(det) + except Exception: + pass + try: + self.capture(det) + except RtlSdrError as exc: + self._error(exc) + # Carry on through the rest of this step's detections. + # Their spectrum reading is now stale, but each one is + # re-checked against the squelch after retuning, so a + # signal that has since dropped is discarded cheaply + # instead of starving the weaker hits in this step. + + self.stats.cycles += 1 + if self.cb.on_cycle: + try: + self.cb.on_cycle(self.stats.cycles) + except Exception: + pass + if cfg.max_cycles and self.stats.cycles >= cfg.max_cycles: + break + finally: + self.stats.state = "stopped" + if self.transcriber is not None: + pending = self.transcriber.pending + if pending: + self._status(f"finishing {pending} transcription(s)") + self.transcriber.close() + if self._own_device and self.device is not None: + self.device.close() + return self.stats diff --git a/bandsaunter/settings.py b/bandsaunter/settings.py new file mode 100644 index 0000000..a7ae511 --- /dev/null +++ b/bandsaunter/settings.py @@ -0,0 +1,666 @@ +"""One description of every setting, shared by the command line and the TUI. + +Both front ends are generated from this table, so a setting cannot exist in +one and be missing from the other, and the help text is written once. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass, fields + +from .bandplan import fmt_hz +from .quality import CATEGORIES as CONTENT_CATEGORIES + +__all__ = ["Setting", "SETTINGS", "GROUPS", "SettingError", "by_key", + "in_group", "parse_value", "format_value", "add_arguments", + "apply_args", "search"] + + +class SettingError(ValueError): + """Raised when a value the user typed cannot be used.""" + + +@dataclass(frozen=True) +class Setting: + key: str # attribute on ScanConfig + label: str + group: str + kind: str # see _parse dispatch below + help: str # one line, shown in lists and --help + detail: str = "" # paragraph, shown by the built-in help + choices: tuple[str, ...] = () + unit: str = "" + minimum: float | None = None + maximum: float | None = None + flags: tuple[str, ...] = () # value flags, or the "on" flags for a bool + off_flags: tuple[str, ...] = () # bool flags that turn it off + metavar: str = "" + example: str = "" + + @property + def dest(self) -> str: + return self.key + + def describe_range(self) -> str: + if self.choices: + return "one of: " + ", ".join(self.choices) + bits = [] + if self.minimum is not None: + bits.append(f"at least {self.minimum:g}") + if self.maximum is not None: + bits.append(f"at most {self.maximum:g}") + return ", ".join(bits) + + +S = Setting + +# --------------------------------------------------------------------------- +# The table. Order within a group is the order the TUI shows them. +# --------------------------------------------------------------------------- + +SETTINGS: tuple[Setting, ...] = ( + + # -- dwell ------------------------------------------------------------- + S("record_seconds", "Record for", "Dwell and recording", "float", + "longest one signal may hold the receiver (0 = no limit)", + "The hard cap on a single capture. Reached even if the signal is still " + "transmitting. Set it to 0 to stay for as long as the transmission " + "lasts, which is what you want for capturing a whole conversation; " + "'Absolute limit' still bounds it.", + unit="s", minimum=0.0, flags=("--record", "--record-seconds"), + metavar="SEC", example="30"), + S("hang_seconds", "Wait for quiet", "Dwell and recording", "float", + "quiet time before the sweep resumes", + "How long the channel must stay quiet before the transmission is " + "treated as over. Gaps shorter than this are recorded straight " + "through, so a two-way exchange stays in one file across the pauses " + "between overs. 'Quiet' means no real signal: silence, static and " + "interference all count towards it.", + unit="s", minimum=0.0, flags=("--hang", "--hang-seconds"), + metavar="SEC", example="3"), + S("max_record_seconds", "Absolute limit", "Dwell and recording", "float", + "ceiling on one capture, even when 'Record for' is 0", + "A safety stop so an unlimited capture cannot run until the disk is " + "full. 0 removes it entirely.", + unit="s", minimum=0.0, flags=("--max-record",), metavar="SEC", + example="900"), + S("min_record_seconds", "Discard shorter than", "Dwell and recording", + "float", "throw away captures shorter than this", + "Brief noise spikes that open the squelch for a moment leave nothing " + "behind on disk.", + unit="s", minimum=0.0, flags=("--min-record",), metavar="SEC", + example="0.5"), + S("revisit_seconds", "Ignore again for", "Dwell and recording", "float", + "hold-off before the same frequency can be recorded again", + "Stops one busy repeater monopolising the sweep.", + unit="s", minimum=0.0, flags=("--revisit",), metavar="SEC", + example="8"), + + # -- detection ---------------------------------------------------------- + S("threshold_db", "Squelch threshold", "Detection", "float", + "margin over the noise before a signal counts as present", + "Measured in dB above the noise, not above the noise floor: the offset " + "that noise alone clears with the current detector is worked out and " + "added automatically, so this number is real headroom. Raise it if the " + "scan stops on too much; lower it to catch weaker signals.", + unit="dB", minimum=0.5, flags=("-t", "--threshold"), metavar="DB", + example="10"), + S("dwell_seconds", "Dwell per step", "Detection", "float", + "how long the sweep listens at each tuner position", + "Longer dwells catch briefer transmissions but slow the sweep. The " + "time for one full pass is roughly this times the number of steps.", + unit="s", minimum=0.005, maximum=5.0, flags=("--dwell",), + metavar="SEC", example="0.05"), + S("resolution_hz", "Sweep resolution", "Detection", "float", + "FFT bin width while sweeping", + "Finer resolution separates close signals and lowers the noise in each " + "bin, at the cost of a larger transform per step.", + unit="Hz", minimum=100.0, flags=("--resolution",), metavar="HZ", + example="3000"), + S("detector", "Detector", "Detection", "choice", + "peak-hold catches bursts; averaging is quieter", + "Peak-hold keeps the strongest value each bin reached during the " + "dwell, so a keyed or bursty transmission that is off for part of the " + "dwell is still found. Averaging gives a smoother estimate of a steady " + "signal.", + choices=("peak", "avg"), flags=("--detector",), metavar="MODE"), + S("detector_bias_db", "Detector bias", "Detection", "opt_float", + "override the computed noise-peak offset (blank = automatic)", + "Peak-hold makes noise alone ride several dB above the measured floor. " + "That offset is derived from the detector and added to the threshold " + "automatically. Set a number here only to override it.", + unit="dB", minimum=0.0, flags=("--detector-bias",), metavar="DB"), + S("squelch_margin_db", "Squelch hysteresis", "Detection", "float", + "how far below the threshold a signal may drop before it counts as gone", + "Stops a signal sitting exactly on the threshold from chattering the " + "squelch open and shut.", + unit="dB", minimum=0.0, flags=("--squelch-margin",), metavar="DB"), + S("adaptive_floor", "Smooth the noise floor", "Detection", "bool", + "average the measured floor across sweeps", + "Averaging the per-bin floor over successive passes stops the " + "threshold jittering with the randomness of a single short capture. " + "The floor itself comes from a percentile that steps over signals, so " + "this cannot hide a station.", + flags=("--adaptive-floor",), off_flags=("--no-adaptive-floor",)), + S("max_detections_per_step", "Signals per step", "Detection", "int", + "how many separate signals one tuner position may report", + "A step covering a busy stretch of spectrum can hold several signals. " + "Each is visited in turn, strongest first.", + minimum=1, flags=("--max-detections",), metavar="N"), + S("probe_seconds", "Probe length", "Detection", "float", + "look at a signal for this long before choosing a demodulator", + "The demodulator is chosen from what the signal physically is rather " + "than from the band plan alone. The probe is played into the recording " + "rather than discarded, so lengthening it does not lose audio.", + unit="s", minimum=0.05, maximum=3.0, flags=("--probe",), metavar="SEC"), + + # -- content gate -------------------------------------------------------- + S("require_signal", "Check for content", "What counts as a signal", "bool", + "only keep captures that carry voice, CW or data", + "A power threshold cannot tell a transmission from a hump of " + "interference. With this on, every capture is checked for content and " + "anything that turns out to be static, hum or a bare carrier is " + "deleted. Turning it off records anything above the squelch.", + flags=("--require-signal",), + off_flags=("--keep-everything", "--no-require-signal")), + S("accept", "Record these", "What counts as a signal", "accept_list", + "which kinds of content are worth keeping", + "voice - speech structure in the demodulated audio. " + "cw - a keyed carrier whose timing resolves as Morse. " + "digital - an identified keying scheme with a steady symbol rate. " + "carrier - a steady unmodulated carrier. " + "noise - everything else.", + choices=CONTENT_CATEGORIES, flags=("--accept",), metavar="LIST", + example="voice,cw,digital"), + S("min_signal_score", "Minimum confidence", "What counts as a signal", + "float", "confidence needed to keep a capture", + "How sure the content check must be before a recording is kept. " + "Between 0 and 1.", + minimum=0.0, maximum=1.0, flags=("--min-signal-score",), metavar="N"), + S("min_voice_score", "Minimum speech score", "What counts as a signal", + "float", "how speech-like audio must be to count as voice", + "Speech is recognised by a pitch track that drifts the way intonation " + "does, pauses between phrases, syllable-rate modulation and moving " + "formants. Lower this if real voice is being missed; raise it if " + "noise is getting through. Between 0 and 1.", + minimum=0.0, maximum=1.0, flags=("--min-voice-score",), metavar="N"), + S("verify_seconds", "Re-check every", "What counts as a signal", "float", + "how often a live capture is re-examined", + "The content check runs while the capture is still going, so " + "interference is dropped after a second or two rather than holding the " + "receiver for the whole record time.", + unit="s", minimum=0.2, flags=("--verify-every",), metavar="SEC"), + S("verify_max_seconds", "Give up after", "What counts as a signal", + "float", "how long a capture has to show any content", + "A capture that has produced no recognisable content by this point is " + "abandoned. Once a capture *has* produced content it is never " + "abandoned this way, so a pause between overs cannot cut a " + "conversation short.", + unit="s", minimum=0.5, flags=("--verify-max",), metavar="SEC"), + + # -- receiver ------------------------------------------------------------- + S("device_index", "Device index", "Receiver", "int", + "which dongle to use when more than one is attached", + "See `bandsaunter devices` for the list.", + minimum=0, flags=("-d", "--device"), metavar="N"), + S("sample_rate", "Sample rate", "Receiver", "int", + "capture rate; sets how much spectrum one step covers", + "The RTL2832 only locks 225001-300000 Hz and 900001-3200000 Hz. " + "2.048 MS/s is a good default: wide enough to sweep quickly and slow " + "enough that the host keeps up.", + unit="Hz", flags=("--sample-rate",), metavar="HZ", example="2048000"), + S("gain", "Tuner gain", "Receiver", "gain", + "gain in dB, or 'auto' for the tuner's own control", + "Manual gain is usually better for scanning: automatic gain rises in " + "quiet parts of the band and amplifies noise until it breaks squelch. " + "`bandsaunter devices --test` lists the steps this tuner supports.", + unit="dB", flags=("-g", "--gain"), metavar="DB|auto"), + S("ppm", "Frequency correction", "Receiver", "int", + "crystal error correction", + "Cheap dongles can be tens of ppm off. A wrong value shifts every " + "measured frequency.", + unit="ppm", flags=("--ppm",), metavar="N"), + S("agc", "RTL2832 AGC", "Receiver", "bool", + "the demodulator chip's own automatic gain", + "Separate from the tuner gain above. Usually best left off.", + flags=("--agc",), off_flags=("--no-agc",)), + S("bias_tee", "Bias tee", "Receiver", "bool", + "put DC on the antenna port to power an external amplifier", + "Only enable this if you know the hardware supports it and nothing on " + "the antenna port would be damaged by DC.", + flags=("--bias-tee",), off_flags=("--no-bias-tee",)), + S("offset_tuning", "Offset tuning", "Receiver", "bool", + "shift the tuner's own local oscillator (E4000 tuners only)", + "Has no effect on the common R820T. The scanner already offsets its " + "local oscillator digitally to keep the DC spike off the signal.", + flags=("--offset-tuning",), off_flags=("--no-offset-tuning",)), + S("direct_sampling", "Direct sampling", "Receiver", "direct", + "HF reception below 24 MHz: auto, 0 off, 1 I branch, 2 Q branch", + "Below about 24 MHz the tuner cannot reach, so the signal is fed " + "straight into the digitiser. 'auto' switches it on for frequencies " + "the tuner cannot reach and off again above them. Most dongles use the " + "Q branch. There is no front-end filtering or gain in this mode.", + choices=("auto", "0", "1", "2"), flags=("--direct-sampling",), + metavar="MODE"), + S("usable_fraction", "Usable bandwidth", "Receiver", "float", + "fraction of the sample rate each step covers", + "The edges of a capture sit in the anti-alias roll-off, so only the " + "middle is searched. Raising this sweeps faster but reaches further " + "into the roll-off.", + minimum=0.1, maximum=0.95, flags=("--usable-fraction",), metavar="N"), + S("dc_guard_hz", "DC guard", "Receiver", "float", + "how far the local oscillator is parked below each step", + "An RTL2832 always shows a spike at whatever it is tuned to. Parking " + "the oscillator below the span being searched keeps that spike out of " + "the results.", + unit="Hz", minimum=0.0, flags=("--dc-guard",), metavar="HZ"), + + # -- output --------------------------------------------------------------- + S("output_dir", "Output directory", "Output", "path", + "where recordings and logs are written", + "Files are named yyyy-mm-dd_hh.mm.ss_frequency_modulation and all live " + "in this one directory. A relative path is taken from wherever the " + "scan is started.", + flags=("-o", "--output"), metavar="DIR"), + S("save_audio", "Save audio", "Output", "bool", + "write a WAV of the demodulated audio", + "16-bit mono PCM at the demodulator's natural rate.", + flags=("--audio",), off_flags=("--no-audio",)), + S("save_iq", "Save raw IQ", "Output", "bool", + "also write the raw complex samples and a SigMF sidecar", + "Large, but lets a capture be re-analysed later or opened in other SDR " + "tools. Roughly 8 bytes per sample at the intermediate rate.", + flags=("--iq", "--save-iq"), off_flags=("--no-iq",)), + S("iq_format", "IQ format", "Output", "choice", + "sample format for raw IQ files", + "cf32 is 32-bit floats, the easiest to work with. cs16 is 16-bit " + "integers and half the size.", + choices=("cf32", "cs16"), flags=("--iq-format",), metavar="FMT"), + S("audio_rate", "Audio rate", "Output", "int", + "preferred audio sample rate", + "The nearest rate the decimation chain can reach exactly is used, so " + "the recorded rate may differ slightly. Broadcast FM is given at least " + "32 kHz regardless.", + unit="Hz", minimum=4000, flags=("--audio-rate",), metavar="HZ"), + S("classify", "Identify signals", "Output", "bool", + "work out what kind of signal each capture is", + "Turning this off also turns off the content check, since that is what " + "decides whether a capture is worth keeping.", + flags=("--classify",), off_flags=("--no-classify",)), + S("decode_morse", "Decode CW to text", "Output", "bool", + "decode keyed carriers as Morse", + "Speed is measured from the signal, so nothing needs configuring. " + "Anything from about 8 to 40 WPM reads reliably.", + flags=("--morse",), off_flags=("--no-morse",)), + S("combine_by_frequency", "Combine by frequency", "Combining", "bool", + "collect every transmission on one frequency into a single file", + "Instead of a file per transmission, each frequency gets one file that " + "later receptions are appended to as the scan continues, so a whole " + "watch on a channel plays back as one recording. The file stays valid " + "while the scan runs, so it can be opened at any time.", + flags=("--combine",), off_flags=("--no-combine",)), + S("announce_timestamps", "Speak the time", "Combining", "bool", + "insert a spoken date and time before each transmission", + "Each recording is preceded by its date and time read aloud, so the " + "combined file says when everything was heard. Uses an installed " + "text-to-speech program if there is one, and a built-in synthesiser " + "otherwise, so it works with nothing else installed.", + flags=("--announce",), off_flags=("--no-announce",)), + S("announce_frequency", "Speak the frequency", "Combining", "bool", + "also read the frequency out with the timestamp", + "Useful when several combined files are played back together, or when " + "the tuned frequency drifts between receptions.", + flags=("--announce-frequency",), off_flags=("--no-announce-frequency",)), + S("announce_engine", "Speech engine", "Combining", "text", + "which text-to-speech to use: auto, builtin, or a program name", + "'auto' uses the first of espeak-ng, espeak, pico2wave, flite or say " + "that is installed, and falls back to the built-in formant " + "synthesiser. 'builtin' always uses the built-in one.", + flags=("--announce-engine",), metavar="NAME", example="auto"), + S("combine_tolerance_hz", "Same-frequency tolerance", "Combining", "float", + "how far apart two receptions may be and still count as one frequency", + "Detections wander by a few hundred hertz, and a channel is usually " + "wider than that, so receptions within this distance of each other are " + "written to the same file.", + unit="Hz", minimum=0.0, flags=("--combine-tolerance",), metavar="HZ"), + S("combine_keep_individual", "Keep separate files too", "Combining", + "bool", "also keep the one-file-per-transmission recordings", + "Off by default when combining, so each transmission exists in one " + "place. The .json describing each capture is written either way.", + flags=("--keep-individual",), off_flags=("--no-keep-individual",)), + + S("transcribe", "Transcribe speech", "Transcription", "bool", + "write out what was said in each voice transmission", + "Recordings the content check identified as voice are passed to a " + "speech recogniser, and the text is written beside the recording as " + "_transcription.txt. Recognition needs an installed engine and " + "takes seconds per capture, so it runs on its own thread and never " + "holds up the scan.", + flags=("--transcribe",), off_flags=("--no-transcribe",)), + S("transcribe_engine", "Recogniser", "Transcription", "text", + "which speech recogniser to use, or auto", + "'auto' picks the first installed of faster-whisper, whisper, " + "whisper-cli, vosk, pocketsphinx. Whisper handles the noise and " + "clipping of radio audio far better than the smaller recognisers, " + "which were trained on clean speech. Run `bandsaunter transcribe --engines` " + "to see what is installed.", + flags=("--transcribe-engine",), metavar="NAME", example="auto"), + S("transcribe_model", "Model", "Transcription", "text", + "model the recogniser should load", + "For whisper this is a size: tiny.en, base.en, small.en, medium.en. " + "Larger is more accurate and slower. For vosk it is a path to an " + "unpacked model directory.", + flags=("--transcribe-model",), metavar="NAME", example="base.en"), + S("transcribe_language", "Language", "Transcription", "text", + "language to expect, or blank to detect it", + "Fixing the language is worth doing: on a short noisy clip automatic " + "detection often guesses wrong and the transcript comes back as " + "nonsense in another language.", + flags=("--transcribe-language",), metavar="CODE", example="en"), + S("transcribe_min_seconds", "Skip clips shorter than", "Transcription", + "float", "do not bother transcribing very short captures", + "A fragment of a word costs as much to recognise as a sentence and " + "rarely produces anything useful.", + unit="s", minimum=0.0, flags=("--transcribe-min",), metavar="SEC"), + + S("log_file", "Log file", "Output", "text", + "name of the run log inside the output directory", + "Written as JSON lines, with a matching .csv alongside it.", + flags=("--log-file",), metavar="NAME"), + + # -- run control ---------------------------------------------------------- + S("max_cycles", "Stop after sweeps", "Run control", "int", + "stop after this many full passes (0 = run until stopped)", + "", minimum=0, flags=("--cycles",), metavar="N"), + S("max_runtime_seconds", "Stop after time", "Run control", "float", + "stop after this long (0 = run until stopped)", + "", unit="s", minimum=0.0, flags=("--duration",), metavar="SEC"), + S("lockout", "Locked-out frequencies", "Run control", "freq_list", + "never stop on these frequencies", + "Useful for a local pager transmitter or a birdie the receiver makes " + "itself. Frequencies may be written 162.55M, 162550000 or 162.55 MHz. " + "The lock-out key during a scan adds the current frequency here for " + "the rest of the run.", + flags=("--lockout",), metavar="FREQ"), + S("lockout_width", "Lock-out width", "Run control", "float", + "how wide a locked-out frequency is", + "A signal within half this distance of a locked-out frequency is " + "ignored.", + unit="Hz", minimum=1.0, flags=("--lockout-width",), metavar="HZ"), + S("quiet", "Quiet output", "Run control", "bool", + "print errors only", + "", flags=("--quiet",), off_flags=("--no-quiet",)), +) + +GROUPS: tuple[str, ...] = tuple(dict.fromkeys(s.group for s in SETTINGS)) + +_BY_KEY = {s.key: s for s in SETTINGS} + + +def by_key(key: str) -> Setting | None: + return _BY_KEY.get(key) + + +def in_group(group: str) -> list[Setting]: + return [s for s in SETTINGS if s.group == group] + + +def search(term: str) -> list[Setting]: + """Find settings by name, flag, or any words from their description. + + Every word has to appear somewhere, so "voice score" finds the speech + threshold even though those two words are never adjacent in its text. + """ + tokens = [t for t in term.strip().lower().split() if t] + if not tokens: + return [] + scored = [] + for s in SETTINGS: + hay = " ".join((s.key, s.key.replace("_", " "), s.label, s.help, + s.detail, s.group, " ".join(s.flags), + " ".join(s.off_flags))).lower() + if not all(t in hay for t in tokens): + continue + # Rank a match on the name above one buried in the explanation. + name = f"{s.key} {s.key.replace('_', ' ')} {s.label}".lower() + rank = 0 if all(t in name for t in tokens) else 1 + scored.append((rank, s)) + scored.sort(key=lambda item: (item[0], item[1].key)) + return [s for _, s in scored] + + +# --------------------------------------------------------------------------- +# Values +# --------------------------------------------------------------------------- + +_TRUE = {"y", "yes", "on", "true", "1", "t"} +_FALSE = {"n", "no", "off", "false", "0", "f"} + +# Suffixes accepted on a frequency. Deliberately not the band-plan parser, +# which reads a bare small number as MHz -- here a bare number is always Hz, +# so a 3000 Hz resolution does not silently become 3 GHz. +_FREQ_MULT = {"": 1.0, "hz": 1.0, "k": 1e3, "khz": 1e3, + "m": 1e6, "mhz": 1e6, "g": 1e9, "ghz": 1e9} +_UNIT_WORDS = {"s": {"s", "sec", "secs", "second", "seconds"}, + "dB": {"db"}, "ppm": {"ppm"}, "Hz": set(_FREQ_MULT)} +# What the display prints for an empty or unlimited value, accepted back so +# that whatever the menu shows can be typed straight back in. +_UNLIMITED = {"no limit", "unlimited", "none", "off", "-"} +_NUMBER = re.compile(r"^([+-]?(?:[0-9]+\.?[0-9]*|\.[0-9]+))\s*([a-zA-Z]*)$") + + +def _numeric(setting: "Setting", raw: str) -> float: + """Read a number, tolerating the unit the display puts after it.""" + text = raw.strip().replace(",", "").replace("_", "") + if text.lower() in _UNLIMITED and (setting.minimum in (None, 0) + or setting.minimum <= 0): + return 0.0 + m = _NUMBER.match(text) + if not m: + raise SettingError(f"{raw.strip()!r} is not a number") + value = float(m.group(1)) + suffix = m.group(2).lower() + if setting.unit == "Hz": + if suffix not in _FREQ_MULT: + raise SettingError(f"unknown frequency unit {m.group(2)!r}") + return value * _FREQ_MULT[suffix] + if suffix and suffix not in _UNIT_WORDS.get(setting.unit, set()): + expected = setting.unit or "no unit" + raise SettingError(f"unexpected unit {m.group(2)!r}; expected {expected}") + return value + + +def parse_value(setting: Setting, text): + """Turn what the user typed into a value, or explain why it will not do.""" + if not isinstance(text, str): + text = str(text) + raw = text.strip() + kind = setting.kind + + try: + if kind == "bool": + low = raw.lower() + if low in _TRUE: + return True + if low in _FALSE: + return False + raise SettingError("answer yes or no") + + if kind == "int": + value = int(round(_numeric(setting, raw))) + elif kind == "float": + value = _numeric(setting, raw) + elif kind == "opt_float": + if raw == "" or raw.lower() in ("auto", "automatic", "none", "-"): + return None + value = _numeric(setting, raw) + elif kind == "choice": + low = raw.lower() + if low not in setting.choices: + raise SettingError("choose " + ", ".join(setting.choices)) + return low + elif kind in ("text", "path"): + if not raw: + raise SettingError("cannot be empty") + return raw + elif kind == "gain": + if raw.lower() in ("auto", "agc", ""): + return "auto" + value = float(raw) + if value < 0: + raise SettingError("gain cannot be negative") + return value + elif kind == "direct": + low = raw.lower() + if low == "auto": + return "auto" + if low not in ("0", "1", "2"): + raise SettingError("choose auto, 0, 1 or 2") + return int(low) + elif kind == "accept_list": + items = [p.strip().lower() for p in raw.replace(" ", ",").split(",") + if p.strip()] + if not items: + raise SettingError("name at least one kind of content") + bad = [i for i in items if i not in setting.choices] + if bad: + raise SettingError( + f"unknown: {', '.join(bad)} (choose from " + f"{', '.join(setting.choices)})") + return list(dict.fromkeys(items)) + elif kind == "freq_list": + from .ranges import RangeError, parse_frequency + if not raw or raw.lower() in ("(none)", "none", "-"): + return [] + out = [] + for part in raw.replace(";", ",").split(","): + part = part.strip() + if not part: + continue + try: + out.append(parse_frequency(part)) + except RangeError as exc: + raise SettingError(str(exc)) from None + return out + else: + raise SettingError(f"unsupported setting kind {kind!r}") + except SettingError: + raise + except ValueError: + raise SettingError(f"{raw!r} is not a number") from None + + if setting.minimum is not None and value < setting.minimum: + raise SettingError(f"must be at least {setting.minimum:g}" + + (f" {setting.unit}" if setting.unit else "")) + if setting.maximum is not None and value > setting.maximum: + raise SettingError(f"must be at most {setting.maximum:g}" + + (f" {setting.unit}" if setting.unit else "")) + return value + + +def format_value(setting: Setting, value) -> str: + """Render a value the way the user would type it.""" + kind = setting.kind + if kind == "bool": + return "yes" if value else "no" + if kind == "opt_float": + return "automatic" if value is None else f"{value:g} {setting.unit}".strip() + if kind == "accept_list": + return ", ".join(value) if value else "(nothing)" + if kind == "freq_list": + return ", ".join(fmt_hz(v) for v in value) if value else "(none)" + if kind == "gain": + return "auto" if isinstance(value, str) else f"{float(value):g} dB" + if kind == "direct": + return "auto" if value == "auto" else str(value) + if kind in ("text", "path", "choice"): + return str(value) + if kind == "int" and setting.unit == "Hz": + return fmt_hz(value) + if kind == "float" and setting.unit == "Hz": + return fmt_hz(value) + if kind in ("int", "float"): + if setting.key in ("record_seconds", "max_record_seconds", + "max_runtime_seconds", "max_cycles") and not value: + return "no limit" + return f"{value:g} {setting.unit}".strip() + return str(value) + + +# --------------------------------------------------------------------------- +# Command line +# --------------------------------------------------------------------------- + +_ARG_GROUP_TITLES = { + "Combining": "combining recordings by frequency", + "Transcription": "speech to text", + "Dwell and recording": "dwell behaviour", + "Detection": "detection", + "What counts as a signal": "what counts as a signal", + "Receiver": "receiver", + "Output": "output", + "Run control": "run control", +} + + +def add_arguments(parser) -> None: + """Add every registry setting to an argparse parser, grouped.""" + for group in GROUPS: + section = parser.add_argument_group(_ARG_GROUP_TITLES.get(group, group)) + for s in in_group(group): + if not s.flags and not s.off_flags: + continue + unit = f" ({s.unit})" if s.unit else "" + if s.kind == "bool": + if s.flags: + section.add_argument(*s.flags, dest=s.dest, + action="store_true", default=None, + help=s.help) + if s.off_flags: + section.add_argument(*s.off_flags, dest=s.dest, + action="store_false", default=None, + help=f"do not {s.help}") + elif s.kind == "freq_list": + section.add_argument(*s.flags, dest=s.dest, action="append", + default=None, + metavar=s.metavar or "VALUE", + help=s.help + "; repeatable") + else: + section.add_argument(*s.flags, dest=s.dest, default=None, + metavar=s.metavar or "VALUE", + help=s.help + unit) + + +def apply_args(cfg, args) -> list[str]: + """Apply command-line values onto a config. Returns the keys that changed.""" + changed = [] + for s in SETTINGS: + if not s.flags and not s.off_flags: + continue + raw = getattr(args, s.dest, None) + if raw is None: + continue + if s.kind == "bool": + value = bool(raw) + elif s.kind == "freq_list": + value = [] + for item in (raw if isinstance(raw, list) else [raw]): + value.extend(parse_value(s, item)) + else: + value = parse_value(s, raw) + setattr(cfg, s.key, value) + changed.append(s.key) + return changed + + +def coverage() -> tuple[list[str], list[str]]: + """Config fields with no setting, and settings with no config field. + + Used by the test suite to keep the table and the config from drifting. + """ + from .config import ScanConfig + exempt = {"ranges"} + config_keys = {f.name for f in fields(ScanConfig)} - exempt + setting_keys = {s.key for s in SETTINGS} + return sorted(config_keys - setting_keys), sorted(setting_keys - config_keys) diff --git a/bandsaunter/simulator.py b/bandsaunter/simulator.py new file mode 100755 index 0000000..9513501 --- /dev/null +++ b/bandsaunter/simulator.py @@ -0,0 +1,419 @@ +"""A synthetic RTL-SDR. + +Implements the same interface as :class:`~bandsaunter.device.RtlSdrDevice` but +generates IQ from a set of virtual transmitters. Used by the test suite and +by ``bandsaunter --simulate``, which lets the whole scanner be exercised without a +dongle attached. +""" + +from __future__ import annotations + +import math +import time +from dataclasses import dataclass, field + +import numpy as np +from scipy import ndimage as _ndi + +from .morse import encode_morse + +__all__ = ["SimulatedDevice", "VirtualTransmitter", "default_transmitters"] + + +_SPEECH_RATE = 16_000 +_SPEECH_CACHE: dict = {} + + +def _speech_loop(seed: int, pitch: float = 120.0, seconds: float = 8.0, + phrases: bool = True) -> np.ndarray: + """Render a loop of synthetic speech at :data:`_SPEECH_RATE` (cached). + + Glottal pulse train with pitch drift and jitter, shaped by three swept + formant resonators, gated into syllables with pauses between phrases -- + the structure that separates a voice from a tone or a hum. + """ + key = (seed, round(pitch, 1), seconds, phrases) + cached = _SPEECH_CACHE.get(key) + if cached is not None: + return cached + + from scipy import signal as sps + rng = np.random.default_rng(seed) + fs = float(_SPEECH_RATE) + n = int(seconds * fs) + t = np.arange(n) / fs + + f0 = pitch * (1.0 + 0.06 * np.sin(2 * np.pi * 0.7 * t) + + 0.03 * np.sin(2 * np.pi * 0.23 * t + 1.1)) + phase = np.cumsum(2.0 * np.pi * f0 / fs) + exc = np.zeros(n) + exc[np.flatnonzero(np.diff(np.floor(phase / (2 * np.pi))) > 0)] = 1.0 + exc -= exc.mean() + + out = np.zeros(n) + seg = int(fs * 0.04) + for base, bw, amp in ((650.0, 90.0, 1.0), (1200.0, 110.0, 0.55), + (2600.0, 160.0, 0.30)): + sweep = base * (1.0 + 0.22 * np.sin(2 * np.pi * 1.9 * t + + rng.uniform(0, 6.283))) + y = np.zeros(n) + for a in range(0, n - seg, seg): + fc = float(np.mean(sweep[a:a + seg])) + b, aa = sps.iirpeak(min(fc, fs / 2 * 0.95) / (fs / 2), + max(2.0, fc / bw)) + y[a:a + seg] = sps.lfilter(b, aa, exc[a:a + seg]) + out += amp * y + + # A little fricative noise on some syllables. + out += 0.05 * rng.standard_normal(n) * ( + np.abs(np.sin(2 * np.pi * 2.3 * t)) > 0.8) + out /= max(np.abs(out).max(), 1e-9) + + # Transmitter audio processing. Every real transmitter compresses hard + # before the modulator, which is why speech on the air sits near full + # modulation instead of at the -25 dB average that raw speech has. + # Without this the simulated signals are far too lightly modulated to + # resemble anything on a real band. + out = np.tanh(3.5 * out) + + # The talker's own syllables and pauses come after the compressor. + syl = 0.55 + 0.45 * np.sin(2 * np.pi * 4.0 * t) + out *= syl + if phrases: + # Pauses between phrases, for a transmitter that stays keyed. + phrase = (np.sin(2 * np.pi * 0.35 * t) > -0.5).astype(float) + k = max(1, int(0.02 * fs)) + out *= sps.lfilter(np.ones(k) / k, [1.0], phrase) + out /= max(np.abs(out).max(), 1e-9) + + _SPEECH_CACHE[key] = out + return out + + +@dataclass +class VirtualTransmitter: + """One synthetic signal on the air.""" + + frequency: float + mode: str = "nfm" # nfm wfm am usb cw fsk2 fsk4 psk carrier ook + power: float = 0.35 # linear amplitude + bandwidth: float = 12_500.0 + label: str = "" + # Scheduling: transmit for ``on_seconds`` out of every ``period_seconds``. + period_seconds: float = 0.0 # 0 = always on + on_seconds: float = 0.0 + phase_offset: float = 0.0 + # Mode-specific extras + ctcss: float = 0.0 + message: str = "CQ CQ DE W1AW K" + wpm: float = 18.0 + baud: float = 4800.0 + deviation: float = 2_500.0 + pitch_hz: float = 120.0 # synthetic talker's voice pitch + + _phase: float = field(default=0.0, init=False, repr=False) + _rng: np.random.Generator = field(default=None, init=False, repr=False) + + def __post_init__(self): + self._seed = abs(hash(self.label or self.frequency)) % 2**31 + self._rng = np.random.default_rng(self._seed) + if not self.label: + self.label = f"{self.frequency/1e6:.4f} MHz {self.mode}" + + def active(self, t: float) -> bool: + if self.period_seconds <= 0 or self.on_seconds <= 0: + return True + phase = (t + self.phase_offset) % self.period_seconds + return phase < self.on_seconds + + # -- waveform generation -------------------------------------------- + def generate(self, t0: float, n: int, fs: float) -> np.ndarray: + """Complex baseband centred on this transmitter's own frequency.""" + if not self.active(t0): + return np.zeros(n, dtype=np.complex64) + t = t0 + np.arange(n, dtype=np.float64) / fs + m = self.mode + + if m == "carrier": + out = np.exp(1j * self._advance(np.zeros(n), fs)) + elif m in ("nfm", "wfm"): + dev = self.deviation if m == "nfm" else 75_000.0 + audio = self._audio(t) + if self.ctcss: + audio = 0.85 * audio + 0.15 * np.sin(2 * np.pi * self.ctcss * t) + out = np.exp(1j * self._advance(dev * audio, fs)) + elif m == "am": + out = ((1.0 + 0.6 * self._audio(t)) * + np.exp(1j * self._advance(np.zeros(n), fs))) + elif m in ("usb", "lsb"): + audio = self._audio(t) + from scipy.signal import hilbert + an = hilbert(audio) + out = an if m == "usb" else np.conj(an) + elif m == "cw": + env = self._morse_envelope(t, fs) + out = env * np.exp(1j * self._advance(np.zeros(n), fs)) + elif m == "ook": + env = (self._symbols(t, fs, 2, self.baud) > 0).astype(np.float64) + out = env * np.exp(1j * self._advance(np.zeros(n), fs)) + elif m in ("fsk2", "fsk4"): + levels = 2 if m == "fsk2" else 4 + sym = self._symbols(t, fs, levels, self.baud) + lv = (sym - (levels - 1) / 2.0) / max(1.0, (levels - 1) / 2.0) + out = np.exp(1j * self._advance(lv * self.deviation, fs)) + elif m == "psk": + sym = self._symbols(t, fs, 4, self.baud) + out = np.exp(1j * (2 * np.pi * sym / 4.0)) + else: + out = (self._rng.standard_normal(n) + + 1j * self._rng.standard_normal(n)) * 0.3 + + return (self.power * out).astype(np.complex64) + + def _advance(self, inst_freq: np.ndarray, fs: float) -> np.ndarray: + """Integrate an instantaneous-frequency series, keeping phase continuous.""" + ph = self._phase + np.cumsum(2.0 * np.pi * inst_freq / fs) + self._phase = float(ph[-1] % (2.0 * np.pi)) if ph.size else self._phase + return ph + + def _audio(self, t: np.ndarray) -> np.ndarray: + """Speech on the RF time base. + + A synthetic talker (glottal pulses through moving formants, gated into + syllables with pauses) is rendered once at audio rate and cached, then + read out against absolute time. Sine tones would be far cheaper but + would not exercise the speech detector at all. + """ + # A transmitter that keys on and off is modelling push-to-talk, where + # the operator talks for as long as the key is down. Its speech loop + # therefore has no phrase pauses of its own -- the keying supplies + # them -- and each over starts at a fresh point in the loop, so a + # short over carries speech rather than whatever silence happened to + # line up with it. + ptt = self.period_seconds > 0 and self.on_seconds > 0 + loop = _speech_loop(self._seed, self.pitch_hz, phrases=not ptt) + if ptt: + shifted = t + self.phase_offset + into_over = shifted % self.period_seconds + over = np.floor(shifted / self.period_seconds) + start = (over * 2.7) % (loop.size / _SPEECH_RATE) + pos = ((start + into_over) * _SPEECH_RATE) % loop.size + else: + pos = (t * _SPEECH_RATE) % loop.size + # Linear interpolation, not nearest sample. Holding each audio sample + # across 128 RF samples is a zero-order hold, and its images put + # audible energy tens of kilohertz either side of the carrier -- which + # made a nominally 8 kHz AM signal measure 25 kHz wide. + i0 = pos.astype(np.int64) + frac = pos - i0 + i1 = (i0 + 1) % loop.size + return loop[i0] * (1.0 - frac) + loop[i1] * frac + + def _symbols(self, t: np.ndarray, fs: float, levels: int, + baud: float) -> np.ndarray: + """Deterministic pseudo-random symbols on an absolute-time grid.""" + idx = np.floor(t * baud).astype(np.int64) + # A cheap hash keeps the stream identical across block boundaries. + h = (idx * 2654435761) % 4294967296 + return (h % levels).astype(np.float64) + + def _morse_envelope(self, t: np.ndarray, fs: float) -> np.ndarray: + key = self._morse_key(self.message, self.wpm) + dot = 1.2 / self.wpm + total = len(key) * dot + pos = ((t % total) / dot).astype(np.int64) + env = key[np.clip(pos, 0, len(key) - 1)].astype(np.float64) + # Soften the edges the way real keying shaping does. Two box filters + # give a triangular window in O(n) -- a direct convolution with a + # 6 ms kernel at 2 MSPS is thousands of taps and dominates runtime. + k = max(3, int(0.003 * fs)) + if env.size > 2 * k: + env = _ndi.uniform_filter1d(env, k, mode="nearest") + env = _ndi.uniform_filter1d(env, k, mode="nearest") + return env + + @staticmethod + def _morse_key(msg: str, wpm: float) -> np.ndarray: + """One element per dot-unit: 1 = key down, 0 = key up.""" + units: list[int] = [0] * 2 + toks = encode_morse(msg).split(" ") + for i, tok in enumerate(toks): + if tok == "/": + units += [0] * 7 + continue + if i > 0 and toks[i - 1] != "/": + units += [0] * 3 + for j, el in enumerate(tok): + if j > 0: + units += [0] + units += [1] * (1 if el == "." else 3) + units += [0] * 2 + return np.array(units, dtype=np.float64) + + +def default_transmitters() -> list[VirtualTransmitter]: + """A demo band populated with one of each interesting signal type.""" + V = VirtualTransmitter + return [ + V(146_520_000, "nfm", 0.40, 12_500, "2 m simplex voice", ctcss=100.0, + period_seconds=14, on_seconds=6), + V(146_940_000, "nfm", 0.30, 12_500, "2 m repeater", ctcss=131.8, + period_seconds=19, on_seconds=5, phase_offset=7), + V(144_100_000, "cw", 0.30, 500, "2 m CW beacon", wpm=18, + message="VVV DE W1AW/B FN31"), + V(162_400_000, "nfm", 0.35, 12_500, "NOAA weather radio"), + V(121_500_000, "am", 0.30, 8_000, "airband AM", period_seconds=17, + on_seconds=5, phase_offset=3), + V(97_500_000, "wfm", 0.45, 180_000, "FM broadcast"), + V(460_025_000, "fsk4", 0.30, 12_500, "P25-style digital voice", + baud=4800, deviation=1_800, period_seconds=13, on_seconds=4), + V(929_612_500, "fsk2", 0.28, 12_500, "POCSAG pager", baud=1200, + deviation=2_400, period_seconds=11, on_seconds=2, phase_offset=5), + V(446_000_000, "carrier", 0.25, 1_000, "unmodulated carrier"), + V(433_920_000, "ook", 0.30, 40_000, "ISM remote", baud=2000, + period_seconds=9, on_seconds=1.2), + ] + + +class SimulatedDevice: + """Drop-in replacement for :class:`RtlSdrDevice` backed by synthetic IQ.""" + + def __init__(self, sample_rate: int = 2_048_000, + transmitters: list[VirtualTransmitter] | None = None, + noise_amplitude: float = 0.012, + seed: int = 0, realtime: bool = False, **_ignored): + self.sample_rate = int(sample_rate) + self.transmitters = (default_transmitters() if transmitters is None + else list(transmitters)) + self.noise_amplitude = float(noise_amplitude) + self.index = 0 + self.tuner = "simulated R820T2" + self.gain = "auto" + self.ppm = 0 + self.retunes = 0 + self.samples_read = 0 + self.realtime = realtime + self._rng = np.random.default_rng(seed) + self._center = 100_000_000 + self._t = 0.0 + self._start_wall = time.time() + self._open = False + + # -- lifecycle ------------------------------------------------------- + def open(self): + self._open = True + return self + + def close(self): + self._open = False + + def __enter__(self): + return self.open() + + def __exit__(self, *exc): + self.close() + return False + + @property + def is_open(self) -> bool: + return self._open + + # -- configuration ---------------------------------------------------- + @property + def available_gains(self): + return [0.0, 9.0, 14.4, 22.9, 28.0, 33.8, 40.2, 49.6] + + @property + def current_gain(self): + return self.gain + + def apply_gain(self, gain): + self.gain = gain + + def set_sample_rate(self, rate: int) -> int: + self.sample_rate = int(rate) + return self.sample_rate + + @property + def direct_sampling_mode(self) -> int: + return 2 if self._center < 24_000_000 else 0 + + def can_tune(self, hz: float) -> bool: + return 0 < hz <= 1_766_000_000 + + def tune(self, hz: float, settle: bool = True) -> int: + self._center = int(round(hz)) + self.retunes += 1 + return self._center + + @property + def center_freq(self) -> int: + return self._center + + def flush(self) -> None: + pass + + # -- streaming (the simulator has no USB to fall behind) ------------- + def start_stream(self, *a, **kw) -> None: + self._streaming = True + + def stop_stream(self) -> None: + self._streaming = False + + @property + def streaming(self) -> bool: + return getattr(self, "_streaming", False) + + @property + def dropped_samples(self) -> int: + return 0 + + def read_stream(self, count: int, timeout: float = 3.0) -> np.ndarray: + return self.read_samples(count) + + # -- capture ----------------------------------------------------------- + def read_samples(self, count: int, flush: bool = False) -> np.ndarray: + count = int(count) + fs = float(self.sample_rate) + # The simulated clock advances with the samples handed out, so signal + # schedules and keying line up with the recorded durations. + t0 = self._t + self._t += count / fs + + out = (self._rng.standard_normal(count) + + 1j * self._rng.standard_normal(count)).astype(np.complex64) + out *= np.complex64(self.noise_amplitude / math.sqrt(2.0)) + + half = fs / 2.0 + t = t0 + np.arange(count, dtype=np.float64) / fs + for tx in self.transmitters: + delta = tx.frequency - self._center + if abs(delta) > half * 0.98: + continue + base = tx.generate(t0, count, fs) + if not np.any(base): + continue + out += (base * np.exp(2j * np.pi * delta * t)).astype(np.complex64) + + if self.realtime: + target = self._start_wall + self._t + gap = target - time.time() + if gap > 0: + time.sleep(min(gap, 0.5)) + + self.samples_read += count + # Quantise to 8 bits, like the real RTL2832 ADC. + q = np.clip(np.stack([out.real, out.imag]), -1.0, 0.996) + q = np.round(q * 128.0) / 128.0 + return (q[0] + 1j * q[1]).astype(np.complex64) + + def read_seconds(self, seconds: float, flush: bool = False) -> np.ndarray: + return self.read_samples(int(self.sample_rate * seconds), flush=flush) + + def status(self) -> dict: + return {"index": self.index, "tuner": self.tuner, + "sample_rate": self.sample_rate, "center_freq": self._center, + "gain": self.gain, "ppm": self.ppm, + "direct_sampling": self.direct_sampling_mode, + "retunes": self.retunes, "samples_read": self.samples_read, + "simulated": True} diff --git a/bandsaunter/transcribe.py b/bandsaunter/transcribe.py new file mode 100644 index 0000000..a91ad76 --- /dev/null +++ b/bandsaunter/transcribe.py @@ -0,0 +1,411 @@ +"""Turning recorded speech into text. + +Speech recognition needs a trained model, so unlike the spoken announcements +there is no built-in fallback: an engine has to be installed. Several are +supported and whichever is present is used. Recognition is slow enough that +it runs on its own thread, so a scan never waits for it. +""" + +from __future__ import annotations + +import os +import queue +import shutil +import subprocess +import tempfile +import threading +import wave +from dataclasses import dataclass, field +from datetime import datetime +from pathlib import Path + +import numpy as np + +from ._quiet import suppress_stderr + +__all__ = ["Transcript", "transcribe", "available_engine", "ENGINES", + "TranscriptionWorker", "describe_engines"] + +# Best first. Whisper handles the noise and clipping of radio audio far +# better than the smaller recognisers, which were trained on clean speech. +ENGINES = ("faster-whisper", "whisper", "whisper-cli", "vosk", "pocketsphinx") + +# Vosk's bundled models are named by region. +_VOSK_LANGS = {"en": "en-us", "en-gb": "en-us", "": "en-us", + "pt": "pt", "zh": "cn", "cn": "cn"} + +# Set BANDSAUNTER_ENGINE_OUTPUT=1 to watch a model download or debug an engine. +_SHOW_ENGINE_OUTPUT = os.environ.get("BANDSAUNTER_ENGINE_OUTPUT", "") not in \ + ("", "0", "no", "false") + +_ENGINE_NOTES = { + "faster-whisper": "pip install faster-whisper (best on radio audio)", + "whisper": "pip install openai-whisper", + "whisper-cli": "whisper.cpp, no Python dependencies", + "vosk": "pip install vosk (small, weaker on noisy audio)", + "pocketsphinx": "apt install pocketsphinx (tiny, poor on radio audio)", +} + + +@dataclass +class Transcript: + text: str = "" + engine: str = "" + language: str = "" + duration: float = 0.0 + segments: list = field(default_factory=list) # (start, end, text) + note: str = "" + + def __bool__(self) -> bool: + return bool(self.text.strip()) + + +def describe_engines() -> list[tuple[str, bool, str]]: + """``(name, installed, how to get it)`` for every supported engine.""" + return [(name, _is_present(name), _ENGINE_NOTES.get(name, "")) + for name in ENGINES] + + +def _is_present(name: str) -> bool: + if name == "faster-whisper": + return _importable("faster_whisper") + if name == "whisper": + return _importable("whisper") + if name == "vosk": + return _importable("vosk") + if name == "pocketsphinx": + return _importable("pocketsphinx") + if name == "whisper-cli": + return any(shutil.which(exe) for exe in ("whisper-cli", "whisper-cpp", + "whisper.cpp")) + return False + + +def _importable(module: str) -> bool: + import importlib.util + try: + return importlib.util.find_spec(module) is not None + except (ImportError, ValueError): + return False + + +def available_engine() -> str | None: + for name in ENGINES: + if _is_present(name): + return name + return None + + +# --------------------------------------------------------------------------- +# Engines. Models are expensive to load, so each is kept once it is built. +# --------------------------------------------------------------------------- + +_MODELS: dict[tuple, object] = {} +_MODEL_LOCK = threading.Lock() + + +def _faster_whisper(audio, rate, model, language): + from faster_whisper import WhisperModel + key = ("faster-whisper", model or "base.en") + with _MODEL_LOCK: + engine = _MODELS.get(key) + if engine is None: + engine = WhisperModel(key[1], device="cpu", compute_type="int8") + _MODELS[key] = engine + segments, info = engine.transcribe( + audio, language=language or None, + vad_filter=True, beam_size=5) + out, pieces = [], [] + for seg in segments: + pieces.append((float(seg.start), float(seg.end), seg.text.strip())) + out.append(seg.text.strip()) + return Transcript(text=" ".join(p for p in out if p).strip(), + engine="faster-whisper", + language=getattr(info, "language", "") or "", + segments=pieces) + + +def _openai_whisper(audio, rate, model, language): + import whisper + key = ("whisper", model or "base.en") + with _MODEL_LOCK: + engine = _MODELS.get(key) + if engine is None: + engine = whisper.load_model(key[1]) + _MODELS[key] = engine + result = engine.transcribe(np.asarray(audio, dtype=np.float32), + language=language or None, fp16=False) + pieces = [(float(s["start"]), float(s["end"]), s["text"].strip()) + for s in result.get("segments", [])] + return Transcript(text=str(result.get("text", "")).strip(), + engine="whisper", language=result.get("language", ""), + segments=pieces) + + +def _whisper_cli(audio, rate, model, language): + exe = next((shutil.which(e) for e in ("whisper-cli", "whisper-cpp", + "whisper.cpp") if shutil.which(e)), + None) + if not exe: + return None + with tempfile.TemporaryDirectory() as tmp: + wav = Path(tmp) / "clip.wav" + _write_wav(wav, audio, 16000) + cmd = [exe, "-f", str(wav), "-otxt", "-of", str(Path(tmp) / "out"), + "-nt"] + if model: + cmd += ["-m", model] + if language: + cmd += ["-l", language] + try: + subprocess.run(cmd, check=True, capture_output=True, timeout=600) + text = (Path(tmp) / "out.txt").read_text().strip() + except (OSError, subprocess.SubprocessError): + return None + return Transcript(text=text, engine="whisper-cli", language=language) + + +def _vosk(audio, rate, model, language): + import json as _json + + from vosk import KaldiRecognizer, Model + # The model setting is shared with the whisper engines, where it holds a + # size such as "base.en". For vosk it is a path, so anything that is not + # an actual directory falls back to the language default rather than + # failing with a confusing "folder does not exist". + # Vosk names its bundled models by region, so the plain codes the rest of + # the program uses have to be mapped or it reports "lang en does not exist". + lang = _VOSK_LANGS.get((language or "en").lower(), language or "en-us") + path = Path(model).expanduser() if model else None + use_path = bool(path and path.is_dir()) + key = ("vosk", str(path) if use_path else f"lang:{lang}") + with _MODEL_LOCK: + engine = _MODELS.get(key) + if engine is None: + engine = Model(str(path)) if use_path else Model(lang=lang) + _MODELS[key] = engine + pcm = (np.clip(audio, -1, 1) * 32767).astype(" np.ndarray: + """Every one of these engines wants 16 kHz mono float. + + Applied once in :func:`transcribe` rather than inside each engine, so an + engine added later cannot forget to do it and be quietly handed audio at + the wrong rate. + """ + audio = np.asarray(audio, dtype=np.float32).ravel() + if int(rate) == 16000 or audio.size == 0: + return audio + from math import gcd + + from scipy.signal import resample_poly + g = gcd(int(rate), 16000) + return resample_poly(audio, 16000 // g, int(rate) // g).astype(np.float32) + + +def _write_wav(path: Path, audio: np.ndarray, rate: int) -> None: + with wave.open(str(path), "wb") as w: + w.setnchannels(1) + w.setsampwidth(2) + w.setframerate(rate) + w.writeframes((np.clip(audio, -1, 1) * 32767).astype(" Transcript | None: + """Recognise speech in a block of audio. + + Returns None when no engine is installed, so the caller can say so once + rather than failing on every capture. + """ + chosen = available_engine() if engine in ("", "auto") else engine + if not chosen: + return None + handler = _DISPATCH.get(chosen) + if handler is None: + return Transcript(note=f"unknown transcription engine {chosen!r}") + if not _is_present(chosen): + return Transcript(note=f"{chosen} is not installed") + original_seconds = np.asarray(audio).size / float(rate) + audio = _as_16k(audio, rate) + try: + # Engines announce model downloads and load progress on stderr, which + # would be drawn straight over the live display. + with suppress_stderr(not _SHOW_ENGINE_OUTPUT): + result = handler(audio, 16000, model, language) + except Exception as exc: # engines fail in many ways + return Transcript(note=f"{chosen} failed: {exc}") + if result is None: + return Transcript(note=f"{chosen} produced nothing") + result.duration = original_seconds + return result + + +# --------------------------------------------------------------------------- +# Running it off the scan loop +# --------------------------------------------------------------------------- + +@dataclass +class _Job: + audio: np.ndarray + rate: float + path: Path + when: datetime + frequency: float + append: bool + meta_path: Path | None = None + + +class TranscriptionWorker: + """Transcribes captures on its own thread. + + Recognition takes seconds per clip -- far longer than a capture -- so + doing it inline would mean missing whatever transmitted meanwhile. + """ + + def __init__(self, engine: str = "auto", model: str = "", + language: str = "en", max_queue: int = 32, + on_done=None, on_error=None): + self.engine = engine + self.model = model + self.language = language + self.on_done = on_done + self.on_error = on_error + self._queue: queue.Queue = queue.Queue(maxsize=max_queue) + self._thread: threading.Thread | None = None + self._stop = threading.Event() + self.written = 0 + self.dropped = 0 + self.failed = 0 + self.empty = 0 + + # -- lifecycle --------------------------------------------------------- + def start(self) -> None: + if self._thread is not None: + return + self._thread = threading.Thread(target=self._run, daemon=True, + name="transcribe") + self._thread.start() + + def close(self, timeout: float = 300.0) -> None: + """Finish the queue, then stop. Called when the scan ends.""" + if self._thread is None: + return + self._queue.put(None) + self._thread.join(timeout=timeout) + self._thread = None + + @property + def pending(self) -> int: + return self._queue.qsize() + + # -- work -------------------------------------------------------------- + def submit(self, audio: np.ndarray, rate: float, path: Path, + when: datetime, frequency: float, append: bool = False, + meta_path: Path | None = None) -> bool: + if audio is None or audio.size == 0: + return False + try: + self._queue.put_nowait(_Job(np.asarray(audio, dtype=np.float32), + float(rate), Path(path), when, + float(frequency), append, + Path(meta_path) if meta_path else None)) + return True + except queue.Full: + self.dropped += 1 + return False + + def _run(self) -> None: + while True: + job = self._queue.get() + if job is None: + break + try: + self._handle(job) + except Exception as exc: + self.failed += 1 + if self.on_error: + try: + self.on_error(exc) + except Exception: + pass + + def _handle(self, job: _Job) -> None: + result = transcribe(job.audio, job.rate, self.engine, self.model, + self.language) + if result is None: + self.failed += 1 + return + body = result.text.strip() + if not body: + # Nothing was said. Writing a file to announce that leaves a + # directory full of placeholders, so write nothing at all; the + # count is reported at the end of the scan instead. + self.empty += 1 + return + + if job.append: + stamp = job.when.strftime("%Y-%m-%d %H:%M:%S") + with open(job.path, "a") as fh: + fh.write(f"[{stamp}] {body}\n") + else: + with open(job.path, "w") as fh: + fh.write(f"{body}\n") + self.written += 1 + self._record_in_metadata(job, body) + if self.on_done: + try: + self.on_done(job.path, result) + except Exception: + pass + + @staticmethod + def _record_in_metadata(job: _Job, text: str) -> None: + """Note the transcript in the capture's sidecar, once it exists. + + Written here rather than when the capture was queued, so the metadata + never points at a transcript file that was never created. + """ + if job.meta_path is None or not job.meta_path.exists(): + return + try: + import json + data = json.loads(job.meta_path.read_text()) + hit = data.get("hit") + if isinstance(hit, dict): + hit["transcript_path"] = str(job.path) + hit["transcript"] = text + job.meta_path.write_text(json.dumps(data, indent=2, default=str)) + except (OSError, ValueError): + pass diff --git a/bandsaunter/tui.py b/bandsaunter/tui.py new file mode 100644 index 0000000..b1e8728 --- /dev/null +++ b/bandsaunter/tui.py @@ -0,0 +1,697 @@ +"""The in-application interface: configure everything without command-line flags. + +Every setting the command line accepts is reachable here, because both are +generated from the same table in :mod:`bandsaunter.settings`. Each one carries +its own help, so nothing has to be looked up elsewhere. +""" + +from __future__ import annotations + +from pathlib import Path + +from rich.console import Console +from rich.panel import Panel +from rich.prompt import Confirm, Prompt +from rich.table import Table +from rich.text import Text + +from . import bandplan, settings as st +from .bandplan import CATEGORIES, PRESETS, BandPreset, fmt_hz, in_category, search +from .config import (DEFAULT_CONFIG_DIR, DEFAULT_CONFIG_PATH, + DEFAULT_OUTPUT_DIR, ScanConfig, + delete_profile, list_profiles, load_config, save_config, + save_default) +from .ranges import RangeError, ScanRange, parse_frequency + +__all__ = ["run_tui", "show_ranges", "settings_menu", "help_screen", + "first_run_setup", "TUIAbort"] + +_BACK = ("", "b", "back", "q", "quit", "x") + + +def _rule(console: Console, text: str) -> None: + console.print() + console.rule(f"[bold]{text}[/bold]", style="blue") + + +class TUIAbort(Exception): + """Input ended or the user interrupted; unwind out of the menus.""" + + +def _ask(console: Console, prompt: str, default: str = "") -> str: + try: + return Prompt.ask(prompt, default=default, show_default=bool(default)) + except (EOFError, KeyboardInterrupt): + # Returning an empty string here would send the menu round again and, + # with no input left to read, round forever. + raise TUIAbort() from None + + +def _confirm(prompt: str, default: bool = False) -> bool: + """Yes/no that treats a closed input as "no" rather than crashing.""" + try: + return Confirm.ask(prompt, default=default) + except (EOFError, KeyboardInterrupt): + return False + + +# --------------------------------------------------------------------------- +# Ranges +# --------------------------------------------------------------------------- + +def show_ranges(console: Console, cfg: ScanConfig) -> None: + if not cfg.ranges: + console.print("[yellow]No frequency ranges selected yet.[/yellow] " + "[grey62]Add some with 1 or 2.[/grey62]") + return + t = Table(box=None, header_style="bold", pad_edge=False) + t.add_column("#", style="grey62", width=3, justify="right") + t.add_column("label") + t.add_column("from", justify="right") + t.add_column("to", justify="right") + t.add_column("span", justify="right", style="grey62") + t.add_column("mode", justify="center") + t.add_column("on", justify="center") + total = 0.0 + for i, r in enumerate(cfg.ranges, 1): + if r.enabled: + total += r.span + t.add_row(str(i), r.label, fmt_hz(r.start), fmt_hz(r.stop), + fmt_hz(r.span), r.mode, + "[green]yes[/green]" if r.enabled else "[red]no[/red]") + console.print(t) + console.print(f"[grey62]{len(cfg.ranges)} range(s), {fmt_hz(total)} of " + f"spectrum enabled[/grey62]") + + +def add_manual_ranges(console: Console, cfg: ScanConfig) -> None: + _rule(console, "add frequency ranges") + console.print( + "Enter a start and end frequency for each range. Units may be " + "written [cyan]144M[/cyan], [cyan]144 MHz[/cyan], " + "[cyan]144000k[/cyan] or plain Hz; a bare number under 10000 is read " + "as MHz.\nLeave the start blank when you are done. There is no limit " + "on how many ranges you add.\n") + added = 0 + while True: + start_s = _ask(console, f" [bold]start[/bold] of range " + f"{len(cfg.ranges) + 1}") + if not start_s.strip(): + break + try: + start = parse_frequency(start_s) + except RangeError as exc: + console.print(f" [red]{exc}[/red]") + continue + end_s = _ask(console, " [bold]end[/bold] of range") + try: + stop = parse_frequency(end_s) if end_s.strip() else start + except RangeError as exc: + console.print(f" [red]{exc}[/red]") + continue + + r = ScanRange(start, stop) + covering = bandplan.presets_covering(0.5 * (r.start + r.stop)) + if covering: + best = min(covering, key=lambda p: p.span) + r.label = f"{fmt_hz(r.start)}-{fmt_hz(r.stop)} ({best.name})" + console.print(f" [grey62]that falls in: {best.name} — default " + f"mode {best.mode}[/grey62]") + mode = _ask(console, " mode [grey62](auto/nfm/wfm/am/usb/lsb/cw/raw)" + "[/grey62]", "auto").lower() + r.mode = mode if mode in ("auto", "nfm", "wfm", "am", "usb", "lsb", + "cw", "raw") else "auto" + cfg.ranges.append(r) + added += 1 + console.print(f" [green]added[/green] {r.describe()}\n") + if added: + console.print(f"[green]{added} range(s) added.[/green]") + + +def choose_presets(console: Console, cfg: ScanConfig) -> None: + while True: + _rule(console, "US band plan") + t = Table(box=None, header_style="bold") + t.add_column("#", style="grey62", width=3, justify="right") + t.add_column("category") + t.add_column("presets", justify="right", style="grey62") + for i, cat in enumerate(CATEGORIES, 1): + t.add_row(str(i), cat, str(len(in_category(cat)))) + console.print(t) + console.print("[grey62]Enter a category number, a search term, or " + "blank to go back.[/grey62]") + answer = _ask(console, " category or search").strip() + if not answer: + return + if answer.isdigit() and 1 <= int(answer) <= len(CATEGORIES): + presets = in_category(CATEGORIES[int(answer) - 1]) + heading = CATEGORIES[int(answer) - 1] + else: + presets = search(answer) + heading = f"search: {answer!r}" + if not presets: + console.print(f" [yellow]nothing matched {answer!r}[/yellow]") + continue + _pick_from(console, cfg, presets, heading) + + +def _pick_from(console: Console, cfg: ScanConfig, + presets: list[BandPreset], heading: str) -> None: + _rule(console, heading) + t = Table(box=None, header_style="bold") + t.add_column("#", style="grey62", width=3, justify="right") + t.add_column("name") + t.add_column("range", justify="right") + t.add_column("mode", justify="center") + t.add_column("notes", style="grey62", overflow="fold", max_width=42) + for i, p in enumerate(presets, 1): + extent = (f"{len(p.expand())} ranges" if p.is_group + else f"{fmt_hz(p.start)} - {fmt_hz(p.stop)}") + t.add_row(str(i), p.name, extent, p.mode, p.note) + console.print(t) + console.print("[grey62]Numbers to add ([cyan]1,3,5[/cyan] or " + "[cyan]1-4[/cyan]), [cyan]all[/cyan], or blank to go " + "back.[/grey62]") + answer = _ask(console, " add").strip().lower() + if not answer: + return + chosen = list(presets) if answer == "all" else _expand(answer, presets) + existing = {r.preset_key for r in cfg.ranges if r.preset_key} + added = 0 + hf = False + for picked in chosen: + # A preset may stand for a set of others; add what it actually scans. + for p in picked.expand(): + if p.key in existing: + continue + cfg.ranges.append(ScanRange.from_preset(p)) + existing.add(p.key) + added += 1 + hf |= p.needs_direct_sampling + if hf: + console.print(" [yellow]some of these are below 24 MHz — they need " + "direct sampling and an HF antenna.[/yellow]") + console.print(f" [green]added {added} range(s)[/green]") + + +def _expand(answer: str, items: list) -> list: + out = [] + for tok in answer.replace(" ", "").split(","): + if not tok: + continue + if "-" in tok: + a, _, b = tok.partition("-") + if a.isdigit() and b.isdigit(): + out += [items[i - 1] for i in range(int(a), int(b) + 1) + if 1 <= i <= len(items)] + elif tok.isdigit() and 1 <= int(tok) <= len(items): + out.append(items[int(tok) - 1]) + return out + + +def edit_ranges(console: Console, cfg: ScanConfig) -> None: + while True: + _rule(console, "ranges") + show_ranges(console, cfg) + console.print( + "\n [cyan]a[/cyan] add by hand " + "[cyan]p[/cyan] add from the band plan\n" + " [cyan]r[/cyan] remove " + "[cyan]t[/cyan] toggle on/off\n" + " [cyan]m[/cyan] change mode " + "[cyan]c[/cyan] clear all\n" + " [cyan]b[/cyan] back\n") + choice = _ask(console, " choice", "b").strip().lower() + if choice in _BACK: + return + if choice == "a": + add_manual_ranges(console, cfg) + elif choice == "p": + choose_presets(console, cfg) + elif choice == "c": + if cfg.ranges and _confirm(" remove every range"): + cfg.ranges = [] + elif choice in ("r", "t", "m") and cfg.ranges: + which = _ask(console, " which numbers").strip() + picked = _expand(which, list(range(1, len(cfg.ranges) + 1))) + if not picked: + console.print(" [yellow]nothing selected[/yellow]") + continue + if choice == "r": + cfg.ranges = [r for i, r in enumerate(cfg.ranges, 1) + if i not in picked] + elif choice == "t": + for i in picked: + cfg.ranges[i - 1].enabled = not cfg.ranges[i - 1].enabled + else: + mode = _ask(console, " mode", "auto").strip().lower() + for i in picked: + cfg.ranges[i - 1].mode = mode or "auto" + + +# --------------------------------------------------------------------------- +# Settings +# --------------------------------------------------------------------------- + +def _setting_row(setting: st.Setting, cfg: ScanConfig, default: ScanConfig): + value = getattr(cfg, setting.key) + shown = st.format_value(setting, value) + changed = value != getattr(default, setting.key) + return Text(shown, style="bold cyan" if changed else "white"), changed + + +def _settings_table(console: Console, group: str, cfg: ScanConfig) -> list[st.Setting]: + items = st.in_group(group) + default = ScanConfig() + t = Table(box=None, header_style="bold", pad_edge=False) + t.add_column("#", style="grey62", width=3, justify="right") + t.add_column("setting", width=22) + t.add_column("value", width=18) + t.add_column("what it does", style="grey62", overflow="fold") + for i, s in enumerate(items, 1): + value, changed = _setting_row(s, cfg, default) + t.add_row(str(i), s.label + (" *" if changed else ""), value, s.help) + console.print(t) + console.print("[grey62]* differs from the built-in default[/grey62]") + return items + + +def setting_help(console: Console, setting: st.Setting, + cfg: ScanConfig) -> None: + default = ScanConfig() + body = [f"[bold]{setting.label}[/bold] [grey62]({setting.key})[/grey62]", + "", setting.help.capitalize() + "."] + if setting.detail: + body += ["", setting.detail] + body.append("") + body.append(f"[grey62]now:[/grey62] " + f"{st.format_value(setting, getattr(cfg, setting.key))}" + f" [grey62]default:[/grey62] " + f"{st.format_value(setting, getattr(default, setting.key))}") + rng = setting.describe_range() + if rng: + body.append(f"[grey62]accepts:[/grey62] {rng}") + if setting.flags: + flags = " ".join(setting.flags) + if setting.off_flags: + flags += " / " + " ".join(setting.off_flags) + body.append(f"[grey62]command line:[/grey62] {flags}") + console.print(Panel(Text.from_markup("\n".join(body)), + border_style="blue", padding=(0, 1))) + + +def edit_setting(console: Console, setting: st.Setting, cfg: ScanConfig) -> bool: + """Prompt for one value. Returns True if it changed.""" + current = getattr(cfg, setting.key) + setting_help(console, setting, cfg) + hint = "yes/no" if setting.kind == "bool" else ( + "/".join(setting.choices) if setting.choices else + (setting.example or setting.metavar or "value")) + while True: + raw = _ask(console, f" [bold]{setting.label}[/bold] [grey62]({hint})" + f"[/grey62]", st.format_value(setting, current) + if setting.kind not in ("freq_list", "accept_list") + else "") + if raw.strip() == "" or raw == st.format_value(setting, current): + return False + if raw.strip().lower() in ("d", "default"): + value = getattr(ScanConfig(), setting.key) + else: + try: + value = st.parse_value(setting, raw) + except st.SettingError as exc: + console.print(f" [red]{exc}[/red]") + continue + setattr(cfg, setting.key, value) + errs = [e for e in cfg.validate() if "frequency ranges" not in e] + if errs: + console.print(f" [red]{errs[0]}[/red]") + setattr(cfg, setting.key, current) + continue + console.print(f" [green]{setting.label} = " + f"{st.format_value(setting, value)}[/green]") + return True + + +def settings_menu(console: Console, cfg: ScanConfig) -> None: + while True: + _rule(console, "settings") + t = Table(box=None, header_style="bold") + t.add_column("#", style="grey62", width=3, justify="right") + t.add_column("group") + t.add_column("settings", justify="right", style="grey62") + for i, g in enumerate(st.GROUPS, 1): + t.add_row(str(i), g, str(len(st.in_group(g)))) + console.print(t) + console.print("[grey62]Enter a group number, a search term " + "(e.g. [cyan]hang[/cyan]), or blank to go back." + "[/grey62]") + answer = _ask(console, " group or search").strip() + if not answer: + return + if answer.isdigit() and 1 <= int(answer) <= len(st.GROUPS): + _group_menu(console, cfg, st.GROUPS[int(answer) - 1]) + else: + hits = st.search(answer) + if not hits: + console.print(f" [yellow]no setting matches " + f"{answer!r}[/yellow]") + continue + if len(hits) == 1: + edit_setting(console, hits[0], cfg) + else: + _list_menu(console, cfg, hits, f"matching {answer!r}") + + +def _list_menu(console: Console, cfg: ScanConfig, items: list[st.Setting], + heading: str) -> None: + while True: + _rule(console, heading) + default = ScanConfig() + t = Table(box=None, header_style="bold") + t.add_column("#", style="grey62", width=3, justify="right") + t.add_column("setting", width=22) + t.add_column("value", width=18) + t.add_column("group", style="grey62") + for i, s in enumerate(items, 1): + value, _ = _setting_row(s, cfg, default) + t.add_row(str(i), s.label, value, s.group) + console.print(t) + answer = _ask(console, " number to edit, or blank to go back").strip() + if not answer or not answer.isdigit(): + return + idx = int(answer) + if 1 <= idx <= len(items): + edit_setting(console, items[idx - 1], cfg) + + +def _group_menu(console: Console, cfg: ScanConfig, group: str) -> None: + while True: + _rule(console, group.lower()) + items = _settings_table(console, group, cfg) + console.print("[grey62]Number to change it, [cyan]?N[/cyan] for help " + "on one, [cyan]d[/cyan] to reset the group, blank to go " + "back.[/grey62]") + answer = _ask(console, " choice").strip().lower() + if answer in _BACK: + return + if answer == "d": + if _confirm(f" reset every setting in {group}"): + default = ScanConfig() + for s in items: + setattr(cfg, s.key, getattr(default, s.key)) + console.print(" [green]reset[/green]") + continue + want_help = answer.startswith("?") + token = answer.lstrip("?").strip() + if not token.isdigit(): + console.print(" [yellow]enter a number from the list[/yellow]") + continue + idx = int(token) + if not (1 <= idx <= len(items)): + console.print(" [yellow]no such number[/yellow]") + continue + if want_help: + setting_help(console, items[idx - 1], cfg) + else: + edit_setting(console, items[idx - 1], cfg) + + +# --------------------------------------------------------------------------- +# Profiles and saved settings +# --------------------------------------------------------------------------- + +def profiles_menu(console: Console, cfg: ScanConfig) -> ScanConfig: + while True: + _rule(console, "saved settings") + console.print(f"[grey62]Settings live in {DEFAULT_CONFIG_DIR}[/grey62]") + exists = DEFAULT_CONFIG_PATH.exists() + console.print(" default settings file: " + + (f"[green]{DEFAULT_CONFIG_PATH}[/green]" if exists + else "[yellow]not saved yet[/yellow]")) + profiles = list_profiles() + if profiles: + t = Table(box=None, header_style="bold") + t.add_column("#", style="grey62", width=3, justify="right") + t.add_column("profile") + t.add_column("ranges", justify="right") + t.add_column("record", justify="right") + t.add_column("hang", justify="right") + for i, p in enumerate(profiles, 1): + try: + other = load_config(str(p)) + t.add_row(str(i), p.stem, str(len(other.ranges)), + f"{other.record_seconds:g}s", + f"{other.hang_seconds:g}s") + except Exception: + t.add_row(str(i), p.stem, "[red]unreadable[/red]", "", "") + console.print(t) + else: + console.print(" [grey62]no named profiles yet[/grey62]") + + console.print( + "\n [cyan]s[/cyan] save as the default settings\n" + " [cyan]n[/cyan] save as a named profile\n" + " [cyan]l[/cyan] load a profile\n" + " [cyan]d[/cyan] delete a profile\n" + " [cyan]b[/cyan] back\n") + choice = _ask(console, " choice", "b").strip().lower() + if choice in _BACK: + return cfg + try: + if choice == "s": + path = save_default(cfg) + console.print(f" [green]saved — every run will start from " + f"{path}[/green]") + elif choice == "n": + name = _ask(console, " profile name", "myscan").strip() + if name: + console.print(f" [green]saved to " + f"{save_config(cfg, name)}[/green]") + elif choice == "l" and profiles: + name = _ask(console, " profile name or number").strip() + if name.isdigit() and 1 <= int(name) <= len(profiles): + name = profiles[int(name) - 1].stem + cfg = load_config(name) + console.print(f" [green]loaded {name}[/green]") + elif choice == "d" and profiles: + name = _ask(console, " profile name or number").strip() + if name.isdigit() and 1 <= int(name) <= len(profiles): + name = profiles[int(name) - 1].stem + if name and _confirm(f" delete {name}"): + console.print(" [green]deleted[/green]" + if delete_profile(name) + else " [yellow]no such profile[/yellow]") + except (OSError, FileNotFoundError, ValueError) as exc: + console.print(f" [red]{exc}[/red]") + + +# --------------------------------------------------------------------------- +# Help +# --------------------------------------------------------------------------- + +_TOPICS: dict[str, tuple[str, str]] = { + "1": ("Getting started", """ +Pick what to scan, then start. Ranges come from two places: type start and +end frequencies by hand, or choose from the built-in US band plan, which +carries each band's usual channel spacing and demodulator so you do not have +to set them. + +Everything on this menu can also be given on the command line, and every +command-line option can be set here. Settings you save become the starting +point for every later run."""), + "2": ("How the scan works", """ +The scanner sweeps each range in steps, listening at each tuner position for +the dwell time. Anything standing far enough above the noise counts as a +detection. + +It then drops onto that frequency, looks at the signal briefly to choose the +right demodulator, and records until either the record limit is reached or the +channel goes quiet for the hang time. Gaps shorter than the hang time are +recorded straight through, so a two-way exchange stays in one file."""), + "3": ("Why nothing is being recorded", """ +Most often the squelch threshold is too high, or the content check is +rejecting what it hears. + +Try 'Squelch threshold' lower (8 dB is sensitive, 15 dB is conservative), and +check the scan summary: it reports how many detections were discarded and +why. To see everything the squelch opens on, set 'Check for content' to no — +but expect static and interference to be recorded too."""), + "4": ("Why static is being recorded", """ +Turn 'Check for content' back on. With it on, a capture is kept only if it +carries speech, decodable Morse, or an identified digital keying scheme; +static, hum, bare carriers and interference are deleted. + +If real signals are being rejected, lower 'Minimum speech score'. Speech +detection needs roughly a second of audio, so very short overs may be +missed."""), + "5": ("Recording conversations", """ +Set 'Record for' to 0 so a long exchange is not cut off, and 'Wait for quiet' +longer than the pause between overs — five or six seconds suits most two-way +traffic. 'Absolute limit' still stops a capture running away. + +Silence, static and interference all count as quiet, so a burst of noise +during a pause will not park the receiver on a finished conversation."""), + "6": ("Files and where they go", """ +Recordings are written to the output directory, all in one flat folder, named +yyyy-mm-dd_hh.mm.ss_frequency_modulation.wav. Beside each one is a .json with +the identification and measurements, and with 'Save raw IQ' on, the raw +samples plus a SigMF sidecar. + +scan_log.csv lists every hit and opens in a spreadsheet."""), + "7": ("HF and direct sampling", """ +Below about 24 MHz the tuner cannot reach, so the signal is fed straight into +the digitiser. 'Direct sampling' set to auto switches this on and off as +needed; most dongles use the Q branch. + +There is no filtering or gain in front of the digitiser in this mode, so an HF +antenna and a quiet location matter more than usual."""), + "8": ("Keys during a scan", """ +q stop the scan +p pause and resume +s skip the signal being recorded and carry on sweeping +l lock out this frequency for the rest of the run ++/- raise or lower the squelch threshold on the fly"""), +} + + +def help_screen(console: Console) -> None: + while True: + _rule(console, "help") + t = Table(box=None, header_style="bold") + t.add_column("#", style="grey62", width=3, justify="right") + t.add_column("topic") + for k, (title, _) in _TOPICS.items(): + t.add_row(k, title) + console.print(t) + console.print("[grey62]Enter a topic number, a setting name to look " + "up, or blank to go back.[/grey62]") + answer = _ask(console, " topic").strip() + if not answer: + return + if answer in _TOPICS: + title, body = _TOPICS[answer] + console.print(Panel(Text(body.strip()), title=title, + border_style="blue", padding=(0, 1))) + else: + hits = st.search(answer) + if not hits: + console.print(f" [yellow]nothing matches {answer!r}[/yellow]") + for s in hits[:4]: + setting_help(console, s, ScanConfig()) + + +# --------------------------------------------------------------------------- +# First run +# --------------------------------------------------------------------------- + +def first_run_setup(console: Console, cfg: ScanConfig) -> bool: + """Ask where recordings should go, the first time the program is run. + + Returns True if settings were saved. Only the one question: everything + else has a working default and can be changed from the settings menu. + """ + console.print(Panel(Text.from_markup( + "[bold]Welcome to bandsaunter[/bold]\n\n" + "Recordings, transcripts and the scan log are all written to one " + "directory. Where would you like them?\n\n" + "[grey62]This is saved, so you are only asked once. Everything else " + "can be changed later from Settings, or with " + "[cyan]bandsaunter config[/cyan].[/grey62]"), + border_style="blue", padding=(0, 1))) + + while True: + answer = _ask(console, " recordings directory", + cfg.output_dir or DEFAULT_OUTPUT_DIR).strip() + if not answer: + answer = DEFAULT_OUTPUT_DIR + path = Path(answer).expanduser() + try: + path.mkdir(parents=True, exist_ok=True) + probe = path / ".bandsaunter-write-test" + probe.touch() + probe.unlink() + except OSError as exc: + console.print(f" [red]cannot use that directory: {exc}[/red]") + continue + cfg.output_dir = answer + break + + try: + saved = save_default(cfg) + console.print(f" [green]recordings will go to {path}[/green]") + console.print(f" [grey62]settings saved to {saved}[/grey62]\n") + return True + except OSError as exc: + console.print(f" [yellow]could not save settings: {exc}[/yellow]") + return False + + +# --------------------------------------------------------------------------- +# Main menu +# --------------------------------------------------------------------------- + +def _summary(cfg: ScanConfig) -> str: + record = ("no limit" if not cfg.record_seconds + else f"{cfg.record_seconds:g}s") + gate = ", ".join(cfg.accept) if cfg.require_signal else "everything" + return (f"record {record}, hang {cfg.hang_seconds:g}s, " + f"squelch +{cfg.threshold_db:g} dB, keep {gate}") + + +def run_tui(console: Console, cfg: ScanConfig | None = None, + source: Path | None = None) -> ScanConfig | None: + """The interactive front end. Returns a config to scan with, or None.""" + cfg = cfg or ScanConfig() + console.print(Panel(Text.from_markup( + "[bold]bandsaunter[/bold] — RTL-SDR signal scanner\n" + "[grey62]Scan any set of frequencies, record what turns up, and " + "identify it. Everything is configurable here; press " + "[cyan]h[/cyan] for help at any point.[/grey62]"), + border_style="blue")) + if source: + console.print(f"[grey62]settings loaded from {source}[/grey62]") + + try: + return _main_loop(console, cfg) + except TUIAbort: + console.print() + return None + + +def _main_loop(console: Console, cfg: ScanConfig) -> ScanConfig | None: + while True: + _rule(console, "main menu") + show_ranges(console, cfg) + console.print( + f"\n [cyan]1[/cyan] Frequency ranges " + f"[grey62]{len(cfg.ranges)} configured[/grey62]\n" + f" [cyan]2[/cyan] Band plan " + f"[grey62]{len(PRESETS)} US presets[/grey62]\n" + f" [cyan]3[/cyan] Settings " + f"[grey62]{_summary(cfg)}[/grey62]\n" + f" [cyan]4[/cyan] Saved settings and profiles\n" + f" [cyan]h[/cyan] Help\n" + f" [cyan]s[/cyan] [bold green]Start scanning[/bold green]\n" + f" [cyan]q[/cyan] Quit\n") + choice = _ask(console, " choice", "s").strip().lower() + + if choice == "1": + edit_ranges(console, cfg) + elif choice == "2": + choose_presets(console, cfg) + elif choice == "3": + settings_menu(console, cfg) + elif choice == "4": + cfg = profiles_menu(console, cfg) + elif choice in ("h", "?", "help"): + help_screen(console) + elif choice in ("s", "start", "go"): + errs = cfg.validate() + if errs: + for e in errs: + console.print(f" [red]{e}[/red]") + continue + return cfg + elif choice in ("q", "quit", "exit"): + return None diff --git a/bandsaunter/ui.py b/bandsaunter/ui.py new file mode 100755 index 0000000..8f3b05d --- /dev/null +++ b/bandsaunter/ui.py @@ -0,0 +1,373 @@ +"""Live terminal display for a running scan.""" + +from __future__ import annotations + +import select +import sys +import termios +import time +import tty +from collections import deque +from dataclasses import dataclass + +import numpy as np +from rich.console import Console, Group +from rich.panel import Panel +from rich.table import Table +from rich.text import Text + +from .bandplan import fmt_hz +from .recorder import HitRecord +from .scanner import Detection, Scanner + +__all__ = ["ScanDisplay", "KeyReader", "print_hit", "print_band_table"] + +_SPARK = " ▁▂▃▄▅▆▇█" + + +def _sparkline(values: np.ndarray, width: int = 60, + lo: float | None = None, hi: float | None = None) -> str: + """Compress a spectrum into one row of block characters.""" + v = np.asarray(values, dtype=np.float64) + if v.size == 0: + return " " * width + if v.size > width: + # Max-reduce rather than average: a narrow carrier must stay visible. + edges = np.linspace(0, v.size, width + 1).astype(int) + v = np.array([v[edges[i]:edges[i + 1]].max() if edges[i + 1] > edges[i] + else v[min(edges[i], v.size - 1)] for i in range(width)]) + lo = float(np.percentile(v, 5)) if lo is None else lo + hi = float(np.percentile(v, 99.5)) if hi is None else hi + if hi <= lo: + hi = lo + 1.0 + idx = np.clip((v - lo) / (hi - lo) * (len(_SPARK) - 1), 0, + len(_SPARK) - 1).astype(int) + return "".join(_SPARK[i] for i in idx) + + +class KeyReader: + """Non-blocking single-key input, restoring the terminal on exit.""" + + def __init__(self, enabled: bool = True): + self.enabled = enabled and sys.stdin.isatty() + self._old = None + + def __enter__(self): + if self.enabled: + try: + self._old = termios.tcgetattr(sys.stdin) + tty.setcbreak(sys.stdin.fileno()) + except (termios.error, ValueError): + self.enabled = False + return self + + def __exit__(self, *exc): + if self._old is not None: + try: + termios.tcsetattr(sys.stdin, termios.TCSADRAIN, self._old) + except (termios.error, ValueError): + pass + return False + + def get(self) -> str | None: + if not self.enabled: + return None + try: + r, _, _ = select.select([sys.stdin], [], [], 0) + if r: + return sys.stdin.read(1) + except (OSError, ValueError): + return None + return None + + +@dataclass +class _RecState: + frequency: float = 0.0 + mode: str = "" + elapsed: float = 0.0 + snr: float = 0.0 + present: bool = False + quiet_for: float = 0.0 + active: bool = False + + +class ScanDisplay: + """Renders scanner callbacks into a live dashboard. + + Attach with ``display.attach(scanner)``; the scanner then drives it. + """ + + def __init__(self, scanner: Scanner, console: Console | None = None, + max_hits: int = 12, show_spectrum: bool = True): + self.scanner = scanner + self.console = console or Console() + self.hits: deque[HitRecord] = deque(maxlen=max_hits) + self.messages: deque[str] = deque(maxlen=4) + self.show_spectrum = show_spectrum + + self._spark = "" + self._span = "" + self._peak = 0.0 + self._step_i = 0 + self._n_steps = 1 + self._rec = _RecState() + self._last_detection: Detection | None = None + self._dirty = True + + # -- callbacks -------------------------------------------------------- + def attach(self, scanner: Scanner | None = None) -> None: + s = scanner or self.scanner + cb = s.cb + cb.on_step = self.on_step + cb.on_detection = self.on_detection + cb.on_record_start = self.on_record_start + cb.on_record_tick = self.on_record_tick + cb.on_record_end = self.on_record_end + cb.on_status = self.on_status + cb.on_error = self.on_error + + def on_step(self, i, n, step, psd_db, freqs): + self._step_i, self._n_steps = i, n + if self.show_spectrum: + inband = (freqs >= step.low) & (freqs <= step.high) + if np.any(inband): + self._spark = _sparkline(psd_db[inband], width=self._spark_width()) + self._peak = float(psd_db[inband].max()) + self._span = f"{fmt_hz(step.low)} - {fmt_hz(step.high)}" + self._dirty = True + + def on_detection(self, det: Detection): + self._last_detection = det + self._dirty = True + + def on_record_start(self, det, rec): + self._rec = _RecState(frequency=rec.frequency, mode=rec.mode, + active=True) + self._dirty = True + + def on_record_tick(self, rec, elapsed, present, snr, quiet_for=0.0): + self._rec.elapsed = elapsed + self._rec.present = present + self._rec.snr = snr + self._rec.quiet_for = quiet_for + self._rec.mode = rec.mode + self._dirty = True + + def on_record_end(self, hit: HitRecord): + self._rec.active = False + if hit.kept: + self.hits.appendleft(hit) + self._dirty = True + + def on_status(self, msg: str): + self.messages.appendleft(msg) + self._dirty = True + + def on_error(self, exc: Exception): + self.messages.appendleft(f"[red]{type(exc).__name__}: {exc}[/red]") + self._dirty = True + + # -- rendering --------------------------------------------------------- + def _spark_width(self) -> int: + return max(20, min(120, self.console.width - 24)) + + def _header(self) -> Panel: + d = self.scanner.device + st = d.status() if d else {} + gain = st.get("gain", "?") + gain = f"{gain:.1f} dB" if isinstance(gain, (int, float)) else str(gain) + ds = st.get("direct_sampling", 0) + bits = [ + f"[bold]{st.get('tuner', '?')}[/bold]", + f"{st.get('sample_rate', 0)/1e6:.3f} MS/s", + f"gain {gain}", + f"{st.get('ppm', 0):+d} ppm", + ] + if ds: + bits.append("[yellow]direct sampling[/yellow]") + if st.get("simulated"): + bits.append("[magenta]SIMULATED[/magenta]") + return Panel(Text.from_markup(" ".join(bits)), + title="receiver", border_style="blue", padding=(0, 1)) + + def _sweep_panel(self, show_spectrum: bool = True) -> Panel: + s = self.scanner.stats + frac = (self._step_i + 1) / max(1, self._n_steps) + bar_w = max(10, min(40, self.console.width - 60)) + filled = int(frac * bar_w) + bar = "[green]" + "━" * filled + "[/green]" + \ + "[grey37]" + "━" * (bar_w - filled) + "[/grey37]" + lines = [ + Text.from_markup( + f"{bar} step {self._step_i + 1}/{self._n_steps} " + f"[bold]{self._span}[/bold]"), + ] + if show_spectrum and self.show_spectrum and self._spark: + lines.append(Text.from_markup( + f"[cyan]{self._spark}[/cyan] peak {self._peak:6.1f} dBFS")) + state = s.state + colour = {"recording": "red", "sweeping": "green", + "paused": "yellow"}.get(state, "white") + lines.append(Text.from_markup( + f"[{colour}]{state}[/{colour}] cycle {s.cycles + 1} " + f"hits {s.recordings} dropped {s.discarded} " + f"detections {s.detections} up {_dur(s.elapsed)}")) + return Panel(Group(*lines), title="sweep", border_style="blue", + padding=(0, 1)) + + def _record_panel(self) -> Panel | None: + r = self._rec + if not r.active: + return None + cfg = self.scanner.cfg + limit = cfg.record_seconds + bar_w = max(10, min(30, self.console.width - 70)) + frac = min(1.0, r.elapsed / limit) if limit else 0.0 + filled = int(frac * bar_w) + bar = ("[red]" + "█" * filled + "[/red]" + + "[grey37]" + "░" * (bar_w - filled) + "[/grey37]") if limit \ + else "[red]recording[/red]" + hang = cfg.hang_seconds + if r.present: + sq = "[green]SIGNAL[/green]" + else: + # Show the gap counting down, so it is obvious the recording is + # being held open across a pause rather than stuck. + sq = f"[yellow]gap {min(r.quiet_for, hang):4.1f}/{hang:g}s[/yellow]" + limit_s = f"/{limit:g}s" if limit else "" + return Panel( + Text.from_markup( + f"[bold red]REC[/bold red] {fmt_hz(r.frequency)} " + f"[{r.mode}] {bar} {r.elapsed:5.1f}{limit_s} " + f"{sq} SNR {r.snr:5.1f} dB"), + border_style="red", padding=(0, 1)) + + def _layout(self) -> tuple[bool, int, bool]: + """Decide what fits: ``(spectrum row, hit rows, footer)``. + + On a short terminal the optional parts are given up in order -- + spectrum, then the hit list, then the key hints -- so the receiver and + sweep panels always fit. A frame taller than the terminal cannot be + redrawn in place, and every refresh would leave another copy of it + behind, which is why the header ends up on screen several times over. + """ + height = self.console.size.height or 24 + budget = max(6, height - 1) + rec = 3 if self._rec.active else 0 + spectrum = bool(self.show_spectrum and self._spark) + + for want_spectrum in ((True, False) if spectrum else (False,)): + for want_footer in (True, False): + base = 3 + (4 if want_spectrum else 3) + rec + \ + (3 if want_footer else 0) + rows = budget - base - 4 # 4 = hits panel chrome + if rows >= 1: + return want_spectrum, rows, want_footer + if budget - base >= 0: + return want_spectrum, 0, want_footer + return False, 0, False + + def _hit_capacity(self) -> int: + """How many hit rows fit without pushing the display off the screen. + + A display taller than the terminal cannot be redrawn in place, so each + refresh scrolls another copy of it into the scrollback and the header + appears over and over. The number of hits grows as the scan runs, + which is why it starts fine and degrades. + """ + return self._layout()[1] + + def _hits_table(self) -> Panel: + t = Table(box=None, expand=True, pad_edge=False, show_edge=False) + t.add_column("time", style="grey62", width=8) + t.add_column("frequency", style="bold cyan", width=15, justify="right") + t.add_column("dur", width=6, justify="right") + t.add_column("SNR", width=7, justify="right") + t.add_column("identified as", ratio=1, overflow="ellipsis") + capacity = self._hit_capacity() + shown = list(self.hits)[:capacity] + for h in shown: + extra = "" + if h.morse_text: + extra = f' [yellow]"{h.morse_text.strip()[:32]}"[/yellow]' + elif h.ctcss_hz: + extra = f" [grey62]CTCSS {h.ctcss_hz:.1f}[/grey62]" + elif h.baud: + extra = f" [grey62]{h.baud:.0f} baud[/grey62]" + conf = h.confidence + colour = "green" if conf >= 0.7 else "yellow" if conf >= 0.45 else "grey62" + t.add_row( + time.strftime("%H:%M:%S", time.localtime(h.started_at)), + fmt_hz(h.frequency), + f"{h.duration:.1f}s", + f"{h.snr_db:.1f}", + Text.from_markup( + f"[{colour}]{h.classification or 'unclassified'}[/{colour}]{extra}"), + ) + if not self.hits: + t.add_row("", "", "", "", Text("no signals recorded yet", + style="grey42")) + hidden = len(self.hits) - len(shown) + title = "recorded signals" + if hidden > 0: + title += f" [grey62]({hidden} more above)[/grey62]" + return Panel(t, title=title, border_style="blue", padding=(0, 1)) + + def _footer(self) -> Panel: + keys = ("[bold]q[/bold] quit [bold]p[/bold] pause " + "[bold]s[/bold] skip [bold]l[/bold] lock out " + "[bold]+/-[/bold] threshold") + msg = self.messages[0] if self.messages else "" + return Panel(Text.from_markup(f"{keys} {msg}"), + border_style="grey37", padding=(0, 1)) + + def render(self): + spectrum, rows, footer = self._layout() + parts = [self._header(), self._sweep_panel(show_spectrum=spectrum)] + rec = self._record_panel() + if rec is not None: + parts.append(rec) + if rows > 0: + parts.append(self._hits_table()) + if footer: + parts.append(self._footer()) + return Group(*parts) + + +def _dur(seconds: float) -> str: + seconds = int(seconds) + h, rem = divmod(seconds, 3600) + m, s = divmod(rem, 60) + return f"{h}:{m:02d}:{s:02d}" if h else f"{m}:{s:02d}" + + +def print_hit(console: Console, hit: HitRecord) -> None: + """One-line-per-hit output for non-interactive runs.""" + if not hit.kept: + return + stamp = time.strftime("%H:%M:%S", time.localtime(hit.started_at)) + conf = f"{hit.confidence*100:.0f}%" + line = (f"[grey62]{stamp}[/grey62] [bold cyan]{fmt_hz(hit.frequency):>14}[/bold cyan]" + f" {hit.duration:5.1f}s SNR {hit.snr_db:5.1f} dB " + f"[green]{hit.classification or 'unclassified'}[/green] ({conf})") + console.print(line, highlight=False) + if hit.morse_text: + console.print(f'{"":>26}[yellow]Morse @ {hit.morse_wpm:.0f} WPM: ' + f'"{hit.morse_text.strip()}"[/yellow]', highlight=False) + if hit.reasons: + console.print(f'{"":>26}[grey54]{hit.reasons[0]}[/grey54]', highlight=False) + + +def print_band_table(console: Console, presets, title: str = "band plan") -> None: + t = Table(title=title, box=None, header_style="bold") + t.add_column("key", style="cyan") + t.add_column("name") + t.add_column("range", justify="right") + t.add_column("mode", justify="center") + t.add_column("notes", style="grey62", overflow="fold") + for p in presets: + extent = (f"{len(p.expand())} ranges, {fmt_hz(p.start)}-{fmt_hz(p.stop)}" + if p.is_group else f"{fmt_hz(p.start)} - {fmt_hz(p.stop)}") + t.add_row(p.key, p.name, extent, p.mode, p.note) + console.print(t) diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..c1b004b --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,41 @@ +[build-system] +requires = ["setuptools>=64"] +build-backend = "setuptools.build_meta" + +[project] +name = "bandsaunter" +version = "1.0.0" +description = "Signal scanner, recorder and identifier for RTL-SDR receivers" +readme = "README.md" +requires-python = ">=3.10" +license = {text = "MIT"} +dependencies = [ + "numpy>=1.22", + "scipy>=1.8", + "rich>=12.0", + "PyYAML>=5.4", +] + +[project.optional-dependencies] +plots = ["matplotlib>=3.5"] +dev = ["pytest>=7.0"] +# Speech recognition for --transcribe. Optional: transcription is off by +# default, and the scan says so plainly when no recogniser is installed. +transcribe = ["faster-whisper>=1.0"] +transcribe-small = ["vosk>=0.3"] + +# System packages, which pip cannot install. Recorded here so packaging does +# not have to rediscover them: +# Depends: librtlsdr0 -- the dongle is unreachable without it +# Recommends: espeak-ng -- clearer spoken timestamps; a built-in +# synthesiser is used when absent + +[project.scripts] +bandsaunter = "bandsaunter.cli:main" + +[tool.setuptools.packages.find] +include = ["bandsaunter*"] + +[tool.pytest.ini_options] +testpaths = ["tests"] +filterwarnings = ["ignore::DeprecationWarning"] diff --git a/tests/morse_gen.py b/tests/morse_gen.py new file mode 100644 index 0000000..c874951 --- /dev/null +++ b/tests/morse_gen.py @@ -0,0 +1,37 @@ +"""Generate correctly-timed Morse audio for testing (1/3/7 unit spacing).""" +import numpy as np +from bandsaunter.morse import encode_morse + + +def morse_keying(msg: str, wpm: float, fs: float) -> np.ndarray: + """Key-down/key-up waveform with ITU timing: element 1, letter 3, word 7.""" + dot = 1.2 / wpm + segs = [(0, 8)] # lead-in silence + letters = encode_morse(msg).split(" ") + for i, tok in enumerate(letters): + if tok == "/": + segs.append((0, 7)) # word gap + continue + if i > 0 and letters[i - 1] != "/": + segs.append((0, 3)) # letter gap + for j, el in enumerate(tok): + if j > 0: + segs.append((0, 1)) # element gap + segs.append((1, 1 if el == "." else 3)) + segs.append((0, 8)) + return np.concatenate([np.full(max(1, int(u * dot * fs)), float(s)) + for s, u in segs]) + + +def morse_audio(msg: str, wpm: float, fs: float = 16000.0, snr_db: float = 20.0, + tone: float = 700.0, seed: int = 0) -> np.ndarray: + rng = np.random.default_rng(seed) + env = morse_keying(msg, wpm, fs) + # Soften the keying edges the way a real transmitter's shaping does. + k = max(3, int(0.006 * fs)) + env = np.convolve(env, np.hanning(k), "same") + env /= max(env.max(), 1e-9) + t = np.arange(env.size) / fs + x = env * np.sin(2 * np.pi * tone * t) + n = np.sqrt(np.mean(x ** 2) / (10 ** (snr_db / 10.0))) + return x + n * rng.standard_normal(x.size) diff --git a/tests/signals.py b/tests/signals.py new file mode 100644 index 0000000..afca258 --- /dev/null +++ b/tests/signals.py @@ -0,0 +1,76 @@ +"""Synthetic test signals shared by the test modules.""" +import numpy as np +from scipy.signal import butter, hilbert, lfilter + +FS = 32000.0 + + +def _noise(x, snr_db, rng): + p = float(np.mean(np.abs(x) ** 2)) + n = np.sqrt(p / (2 * 10 ** (snr_db / 10.0))) + return (x + n * (rng.standard_normal(x.size) + + 1j * rng.standard_normal(x.size))).astype(np.complex64) + + +def voice(n, fs=FS, seed=0): + """Band-limited noise with a syllabic envelope -- a good speech stand-in.""" + rng = np.random.default_rng(seed) + b, a = butter(4, [300 / (fs / 2), 2700 / (fs / 2)], btype="band") + v = lfilter(b, a, rng.standard_normal(n)) + v /= max(np.abs(v).max(), 1e-9) + t = np.arange(n) / fs + return v * (0.4 + 0.6 * np.abs(np.sin(2 * np.pi * 1.7 * t))) + + +def make(kind, n=64000, fs=FS, snr_db=30.0, seed=3): + rng = np.random.default_rng(seed) + t = np.arange(n) / fs + v = voice(n, fs, seed) + + if kind == "nfm": + msg = v + 0.15 * np.sin(2 * np.pi * 100.0 * t) + x = np.exp(1j * np.cumsum(2 * np.pi * 2500 * msg / fs)) + elif kind == "wfm": + x = np.exp(1j * np.cumsum(2 * np.pi * 3000 * v / fs)) + elif kind == "am": + x = ((1 + 0.6 * v) * np.exp(2j * np.pi * 30 * t)) + elif kind == "usb": + x = 0.5 * hilbert(v) + elif kind == "lsb": + x = 0.5 * np.conj(hilbert(v)) + elif kind == "carrier": + x = np.exp(2j * np.pi * 137 * t) + elif kind == "cw": + dot = 0.08 + pat = [1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0] + key = np.zeros(n) + i = 0 + while i < n: + for b in pat: + m = int(dot * fs) + if i + m > n: + break + key[i:i + m] = b + i += m + env = np.convolve(key, np.hanning(int(0.005 * fs)), "same") + env /= max(env.max(), 1e-9) + x = env * np.exp(2j * np.pi * 300 * t) + elif kind.startswith("fsk"): + levels = int(kind[3]) + baud, dev = (1200.0, 2400.0) if levels == 2 else (4800.0, 1800.0) + sp = int(fs / baud) + sym = rng.integers(0, levels, n // sp + 1) + lv = (sym - (levels - 1) / 2) / max(1, (levels - 1) / 2) + f = np.resize(np.repeat(lv, sp) * dev, n) + x = np.exp(1j * np.cumsum(2 * np.pi * f / fs)) + elif kind.startswith("psk"): + m = int(kind[3]) + sp = int(fs / 4800.0) + sym = rng.integers(0, m, n // sp + 1) + x = np.exp(1j * np.resize(np.repeat(2 * np.pi * sym / m, sp), n)) + elif kind == "noise": + return (0.01 * (rng.standard_normal(n) + + 1j * rng.standard_normal(n))).astype(np.complex64) + else: + raise ValueError(kind) + return _noise(np.asarray(x, dtype=np.complex128), snr_db, rng) diff --git a/tests/speech.py b/tests/speech.py new file mode 100644 index 0000000..e647b11 --- /dev/null +++ b/tests/speech.py @@ -0,0 +1,52 @@ +"""Synthetic speech for testing the voice detector. + +A glottal pulse train with jitter, shaped by three formant resonators and +gated into syllables with pauses between phrases -- the structure a speech +detector is supposed to key on, without needing a recorded voice sample. +""" +import numpy as np +from scipy import signal as sps + + +def synth_speech(seconds=4.0, fs=16000.0, pitch=120.0, seed=0, + syllable_rate=4.0, pause_fraction=0.25): + rng = np.random.default_rng(seed) + n = int(seconds * fs) + t = np.arange(n) / fs + + # Glottal excitation: pulse train with natural pitch drift and jitter. + f0 = pitch * (1.0 + 0.06 * np.sin(2 * np.pi * 0.7 * t) + + 0.02 * rng.standard_normal(n).cumsum() / np.sqrt(n)) + phase = np.cumsum(2 * np.pi * f0 / fs) + exc = np.zeros(n) + pulses = np.flatnonzero(np.diff(np.floor(phase / (2 * np.pi))) > 0) + exc[pulses] = 1.0 + exc -= exc.mean() + + # Three formants, swept so the spectrum moves the way articulation does. + out = np.zeros(n) + for base, bw, amp in ((650.0, 90.0, 1.0), (1200.0, 110.0, 0.55), + (2600.0, 160.0, 0.30)): + sweep = base * (1.0 + 0.22 * np.sin(2 * np.pi * 1.9 * t + + rng.uniform(0, 6.283))) + # Piecewise-constant formant, cheap but spectrally honest. + seg = int(fs * 0.04) + y = np.zeros(n) + for s in range(0, n - seg, seg): + fc = float(np.mean(sweep[s:s + seg])) + b, a = sps.iirpeak(min(fc, fs / 2 * 0.95) / (fs / 2), + max(2.0, fc / bw)) + y[s:s + seg] = sps.lfilter(b, a, exc[s:s + seg]) + out += amp * y + + # Voiced/unvoiced: add fricative noise on some syllables. + out += 0.05 * rng.standard_normal(n) * (np.abs(np.sin(2 * np.pi * 2.3 * t)) > 0.8) + + # Syllabic gating plus phrase pauses. + syl = 0.5 + 0.5 * np.sin(2 * np.pi * syllable_rate * t) + phrase = (np.sin(2 * np.pi * 0.35 * t) > (-1 + 2 * pause_fraction)).astype(float) + phrase = sps.lfilter(np.ones(int(0.02 * fs)) / int(0.02 * fs), [1.0], phrase) + out *= syl * phrase + + out /= max(np.abs(out).max(), 1e-9) + return out * 0.7 diff --git a/tests/test_announce.py b/tests/test_announce.py new file mode 100644 index 0000000..0298791 --- /dev/null +++ b/tests/test_announce.py @@ -0,0 +1,321 @@ +"""Spoken announcements and the files they are appended to.""" +from datetime import datetime + +import numpy as np +import pytest + +from bandsaunter import announce +from bandsaunter.recorder import FrequencyLog, append_wav, read_wav +from bandsaunter.quality import voice_metrics + + +# --------------------------------------------------------------------------- +# Words +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("n,words", [ + (0, "zero"), (7, "seven"), (10, "ten"), (12, "twelve"), (19, "nineteen"), + (20, "twenty"), (21, "twenty one"), (45, "forty five"), (59, "fifty nine"), + (100, "one hundred"), (146, "one hundred forty six"), + (2026, "two thousand twenty six"), +]) +def test_number_words(n, words): + assert " ".join(announce.number_words(n)) == words + + +def test_timestamp_phrase_reads_naturally(): + phrase = announce.timestamp_phrase(datetime(2026, 8, 21, 14, 38, 5)) + assert "august" in phrase + assert "twenty one" in phrase + assert "twenty twenty six" in phrase # years are said in pairs + assert "fourteen thirty eight" in phrase + assert "oh five" in phrase # 05 seconds, not "five" + + +def test_midnight_and_single_digits(): + phrase = announce.timestamp_phrase(datetime(2026, 1, 3, 0, 5, 0)) + assert "january three" in phrase + assert "oh oh" in phrase # hour 00 and second 00 + + +def test_frequency_is_spoken_digit_by_digit(): + phrase = announce.timestamp_phrase(datetime(2026, 1, 1, 0, 0, 0), + frequency=146_520_000.0) + assert "one hundred forty six point one four six" not in phrase + assert "point five two zero" in phrase + assert "megahertz" in phrase + + +def test_every_word_in_the_vocabulary_can_be_spoken(): + """A word with no pronunciation would be silently dropped.""" + for word in announce.WORDS: + for phone in announce.WORDS[word].split(): + assert phone in announce.PHONEMES, f"{word} uses unknown {phone}" + + +def test_all_the_words_a_timestamp_needs_are_covered(): + for month in range(1, 13): + for day in (1, 9, 15, 21, 28): # 28 is safe in every month + for hour in (0, 5, 12, 23): + phrase = announce.timestamp_phrase( + datetime(2026, month, day, hour, 45, 9)) + for word in phrase.split(): + assert word in ("_", "__") or word in announce.WORDS, word + + +# --------------------------------------------------------------------------- +# Synthesis +# --------------------------------------------------------------------------- + +def _lpc_formants(x, fs, order=12): + x = x - x.mean() + x = np.append(x[0], x[1:] - 0.97 * x[:-1]) * np.hamming(x.size) + r = np.correlate(x, x, "full")[x.size - 1:x.size - 1 + order + 1] + if r[0] == 0: + return [] + a = np.zeros(order + 1) + a[0], e = 1.0, r[0] + for i in range(1, order + 1): + k = -(a[:i] @ r[i:0:-1]) / e + a[:i + 1] = (np.concatenate([a[:i], [0]]) + + k * np.concatenate([[0], a[i - 1::-1]])) + e *= (1 - k * k) + if e <= 0: + break + roots = [z for z in np.roots(a) if np.imag(z) > 0.01] + return [f for f in sorted(float(np.angle(z) * fs / (2 * np.pi)) + for z in roots) if 120 < f < 5000] + + +@pytest.mark.parametrize("vowel", ["iy", "ih", "eh", "ae", "aa", "ao", + "uw", "uh", "ah", "er"]) +def test_vowels_land_on_their_formant_targets(vowel): + """Intelligibility rests on the formants being where they are meant to be. + + Summing resonators in parallel instead of cascading them loses the first + formant entirely, and every vowel then sounds the same. + """ + spec = announce.PHONEMES[vowel] + announce.WORDS["_probe_"] = vowel + audio = announce.synthesize("_probe_", 16000) + seg = audio[int(audio.size * 0.35):][:800] + formants = _lpc_formants(seg, 16000) + assert len(formants) >= 2, f"{vowel}: found {formants}" + assert abs(formants[0] - spec.f1) < 130, f"{vowel} F1: {formants}" + assert abs(formants[1] - spec.f2) < 250, f"{vowel} F2: {formants}" + + +def test_speech_is_in_the_voice_band(): + audio = announce.speak_timestamp(datetime(2026, 8, 21, 14, 38, 5)) + n = 1 << 14 + spec = np.abs(np.fft.rfft(audio[:n] * np.hanning(n))) ** 2 + freqs = np.fft.rfftfreq(n, 1 / 16000) + band = (freqs >= 250) & (freqs < 3400) + assert spec[band].sum() / spec.sum() > 0.6 + + +def test_the_announcement_reads_as_speech(): + """The voice detector should hear the announcement as a voice.""" + audio = announce.speak_timestamp(datetime(2026, 8, 21, 14, 38, 5)) + m = voice_metrics(audio, 16000) + assert m.score > 0.6, m.describe() + assert 80 < m.pitch_hz < 200 + + +def test_synthesis_is_fast_enough_to_run_inline(): + import time + t0 = time.perf_counter() + audio = announce.speak_timestamp(datetime(2026, 8, 21, 14, 38, 5)) + elapsed = time.perf_counter() - t0 + assert elapsed < 0.25 * audio.size / 16000, f"{elapsed:.2f}s" + + +def test_unknown_words_do_not_crash(): + assert announce.synthesize("gibberishwordnotinvocabulary", 16000).size == 0 + + +# --------------------------------------------------------------------------- +# Appending +# --------------------------------------------------------------------------- + +def test_a_wav_stays_valid_after_every_append(tmp_path): + """A scan can be stopped at any moment; the file so far must still play.""" + path = tmp_path / "acc.wav" + for i in range(1, 5): + append_wav(path, np.full(1600, 0.1 * i, dtype=np.float32), 16000) + audio, rate = read_wav(path) + assert rate == 16000 + assert audio.size == 1600 * i + import wave + with wave.open(str(path)) as w: # the standard reader must agree + assert w.getnframes() == 1600 * i + + +def test_appending_a_different_rate_is_refused(tmp_path): + path = tmp_path / "acc.wav" + append_wav(path, np.zeros(160, dtype=np.float32), 16000) + with pytest.raises(ValueError, match="16000"): + append_wav(path, np.zeros(160, dtype=np.float32), 32000) + + +def test_frequency_log_groups_nearby_receptions(tmp_path): + log = FrequencyLog(tmp_path, tolerance_hz=6250.0, announce=False) + audio = np.full(1600, 0.2, dtype=np.float32) + log.add(146_520_040.0, audio, 16000) + log.add(146_519_800.0, audio, 16000) # 240 Hz away: same channel + log.add(147_100_000.0, audio, 16000) # far away: its own file + files = sorted(p.name for p in tmp_path.glob("*.wav")) + assert len(files) == 2, files + assert log.appended == 3 + + +def test_frequency_log_resamples_a_mismatched_capture(tmp_path): + log = FrequencyLog(tmp_path, announce=False) + log.add(146_520_000.0, np.full(16000, 0.2, dtype=np.float32), 16000) + log.add(146_520_000.0, np.full(32000, 0.2, dtype=np.float32), 32000) + audio, rate = read_wav(log.files[0]) + assert rate == 16000 + # one second at each rate, plus the gaps between them + assert 2.0 <= audio.size / rate <= 3.0 + + +def test_announcement_is_placed_before_each_recording(tmp_path): + quiet = FrequencyLog(tmp_path / "a", announce=False) + spoken = FrequencyLog(tmp_path / "b", announce=True) + body = np.full(16000, 0.2, dtype=np.float32) + quiet.add(146_520_000.0, body, 16000, when=datetime(2026, 8, 21, 14, 38, 5)) + spoken.add(146_520_000.0, body, 16000, when=datetime(2026, 8, 21, 14, 38, 5)) + short, _ = read_wav(quiet.files[0]) + long_, rate = read_wav(spoken.files[0]) + assert long_.size > short.size + 2 * rate, "no announcement was inserted" + # the recording is a steady 0.2; the announcement is not, and comes first + assert voice_metrics(long_[:int(3 * rate)], rate).score > 0.5 + + +def test_an_existing_file_from_an_earlier_run_is_continued(tmp_path): + first = FrequencyLog(tmp_path, announce=False) + first.add(146_520_000.0, np.full(1600, 0.2, dtype=np.float32), 16000) + before = read_wav(first.files[0])[0].size + + second = FrequencyLog(tmp_path, announce=False) + second.add(146_520_000.0, np.full(1600, 0.2, dtype=np.float32), 16000) + after = read_wav(second.files[0])[0].size + assert after > before, "a later run started a new file instead of appending" + assert len(list(tmp_path.glob("*.wav"))) == 1 + + +def test_the_date_is_only_spoken_when_it_changes(tmp_path): + """Repeating the date before every over takes longer than most overs last.""" + log = FrequencyLog(tmp_path, announce=True) + body = np.full(16000, 0.2, dtype=np.float32) + lengths = [] + previous = 0 + for when in (datetime(2026, 8, 21, 14, 38, 5), + datetime(2026, 8, 21, 14, 41, 0), + datetime(2026, 8, 22, 9, 2, 0)): + log.add(146_520_000.0, body, 16000, when=when) + size = read_wav(log.files[0])[0].size + lengths.append(size - previous) + previous = size + first, same_day, next_day = lengths + assert same_day < first * 0.7, "the date was repeated needlessly" + assert next_day > same_day * 1.5, "a new day should get the full date" + + +def test_time_only_announcement_is_shorter(): + when = datetime(2026, 8, 21, 14, 38, 5) + full = announce.speak_timestamp(when, 16000, with_date=True) + brief = announce.speak_timestamp(when, 16000, with_date=False) + assert brief.size < full.size + assert brief.size > 8000, "the time itself must still be spoken" + + +# --------------------------------------------------------------------------- +# Installed engines +# --------------------------------------------------------------------------- + +HAVE_ENGINE = announce.available_engine() is not None +needs_engine = pytest.mark.skipif(not HAVE_ENGINE, + reason="no text-to-speech program installed") + + +def test_engine_detection_does_not_throw(): + engine = announce.available_engine() + assert engine is None or engine in announce.ENGINES + + +@pytest.mark.parametrize("when", [ + datetime(2026, 8, 21, 14, 38, 5), + datetime(2026, 1, 3, 9, 5, 0), + datetime(2026, 12, 31, 23, 59, 59), +]) +def test_engine_text_avoids_the_forms_engines_misread(when): + """Punctuation is what gives an engine its phrasing, and the obvious + spellings are traps: a colon makes espeak read 14:38:05 as "fourteen + thirty, eight zero five", and an ISO date has its dashes read aloud.""" + text = announce.timestamp_text(when) + assert ":" not in text + assert "-" not in text + assert text.count(",") >= 2, text # date, year and time separated + assert "twenty twenty six" in text # not "two thousand and ..." + + +def test_engine_text_covers_the_time_only_case(): + text = announce.timestamp_text(datetime(2026, 8, 21, 9, 5, 0), + with_date=False) + assert "August" not in text + assert "09 05" in text + + +@needs_engine +def test_installed_engine_renders_speech(): + audio = announce.speak_timestamp(datetime(2026, 8, 21, 14, 38, 5), 16000, + engine="auto") + assert audio.size > 16000, "suspiciously short" + assert audio.dtype == np.float32 + m = voice_metrics(audio, 16000) + assert m.score > 0.6, m.describe() + + +@needs_engine +@pytest.mark.parametrize("rate", [8000, 16000, 32000, 48000]) +def test_installed_engine_is_resampled_to_the_asked_for_rate(rate): + """Engines render at their own rate; espeak-ng uses 22050 Hz.""" + audio = announce.speak_timestamp(datetime(2026, 8, 21, 14, 38, 5), rate, + engine="auto") + seconds = audio.size / rate + assert 2.0 < seconds < 12.0, f"{seconds:.2f}s at {rate} Hz" + + +@needs_engine +def test_both_engines_produce_the_same_level(): + """Switching engines must not change how loud announcements are.""" + when = datetime(2026, 8, 21, 14, 38, 5) + external = announce.speak_timestamp(when, 16000, engine="auto") + builtin = announce.speak_timestamp(when, 16000, engine="builtin") + assert abs(float(np.abs(external).max()) + - float(np.abs(builtin).max())) < 0.05 + + +@needs_engine +def test_dropping_the_date_shortens_the_announcement(): + when = datetime(2026, 8, 21, 14, 38, 5) + full = announce.speak_timestamp(when, 16000, engine="auto") + brief = announce.speak_timestamp(when, 16000, engine="auto", + with_date=False) + assert brief.size < full.size * 0.75 + + +def test_an_engine_that_is_not_installed_falls_back_to_the_builtin(): + """The feature must keep working with nothing else on the machine.""" + audio = announce.speak_timestamp(datetime(2026, 8, 21, 14, 38, 5), 16000, + engine="definitely-not-installed") + assert audio.size > 0 + assert voice_metrics(audio, 16000).score > 0.6 + + +def test_a_failing_engine_falls_back_rather_than_crashing(monkeypatch): + monkeypatch.setattr(announce, "_external", lambda *a, **k: None) + audio = announce.speak_timestamp(datetime(2026, 8, 21, 14, 38, 5), 16000, + engine="auto") + assert audio.size > 0 diff --git a/tests/test_classify.py b/tests/test_classify.py new file mode 100644 index 0000000..1c1a37f --- /dev/null +++ b/tests/test_classify.py @@ -0,0 +1,50 @@ +import pytest + +from bandsaunter.classify import classify, extract_features +from signals import FS, make + +CASES = [ + ("cw", "cw", 14.05e6), ("nfm", "nfm", 146.52e6), ("am", "am", 121.5e6), + ("usb", "ssb", 14.2e6), ("lsb", "ssb", 7.2e6), + ("fsk2", "fsk", 929.5e6), ("fsk4", "fsk", 460.0e6), + ("psk4", "psk", 450.0e6), ("psk2", "psk", 437.0e6), + ("carrier", "carrier", 446.0e6), ("noise", "unknown", 300e6), +] + + +@pytest.mark.parametrize("kind,family,freq", CASES) +@pytest.mark.parametrize("seed", [3, 11, 42]) +def test_modulation_family(kind, family, freq, seed): + c = classify(make(kind, seed=seed), FS, freq_hz=freq, snr_db=28) + assert c.family == family, f"{kind} -> {c.label!r} (family {c.family})" + + +def test_known_systems_are_named_from_frequency(): + assert "airband" in classify(make("am"), FS, freq_hz=121.5e6, + snr_db=28).label.lower() + assert "broadcast" in classify(make("wfm"), FS, freq_hz=97.5e6, + snr_db=28).label.lower() + assert "weather" in classify(make("nfm"), FS, freq_hz=162.475e6, + snr_db=28).label.lower() + + +def test_ctcss_tone_recovered(): + c = classify(make("nfm"), FS, freq_hz=146.52e6, snr_db=28) + assert c.features.ctcss_hz == pytest.approx(100.0, abs=0.5) + + +def test_low_snr_reduces_confidence(): + strong = classify(make("nfm", snr_db=30), FS, freq_hz=146.52e6, snr_db=30) + weak = classify(make("nfm", snr_db=5), FS, freq_hz=146.52e6, snr_db=5) + assert weak.confidence < strong.confidence + + +def test_fsk_level_count(): + assert extract_features(make("fsk2"), FS, snr_db=28).freq_modes == 2 + assert extract_features(make("fsk4"), FS, snr_db=28).freq_modes == 4 + + +def test_symbol_rate_estimate(): + f = extract_features(make("fsk4"), FS, snr_db=28) + # The cyclostationary line lands on the baud rate or a low harmonic of it. + assert f.baud > 1000 diff --git a/tests/test_dsp.py b/tests/test_dsp.py new file mode 100644 index 0000000..494c336 --- /dev/null +++ b/tests/test_dsp.py @@ -0,0 +1,114 @@ +import numpy as np +import pytest + +from bandsaunter import dsp + + +def test_decimator_is_stateful_across_blocks(): + """Block-by-block filtering must equal one-shot filtering exactly.""" + rng = np.random.default_rng(0) + x = (rng.standard_normal(20000) + 1j * rng.standard_normal(20000)).astype(np.complex64) + d1 = dsp.FIRDecimator(8) + blocked = np.concatenate([d1(x[:7000]), d1(x[7000:13000]), d1(x[13000:])]) + d2 = dsp.FIRDecimator(8) + assert np.allclose(blocked, d2(x), atol=1e-9) + + +def test_decimation_chain_factors(): + assert dsp.design_decimation(128) == [8, 8, 2] + assert dsp.design_decimation(1) == [] + chain = dsp.DecimationChain(64) + out = chain(np.ones(6400, dtype=np.complex64)) + assert out.size == 100 + + +def test_mixer_keeps_phase_continuous(): + fs = 48000.0 + m = dsp.Mixer(1000.0, fs) + a = m(np.ones(1000, dtype=np.complex64)) + b = m(np.ones(1000, dtype=np.complex64)) + joined = np.concatenate([a, b]) + one_shot = dsp.frequency_shift(np.ones(2000, dtype=np.complex64), 1000.0, fs) + assert np.allclose(joined, one_shot, atol=1e-4) + + +def test_noise_floor_curve_ignores_signals(): + """A carrier must not raise the floor it is being measured against.""" + psd = np.full(1024, -80.0) + psd[500:504] = -20.0 + floor = dsp.noise_floor_curve(psd) + assert floor[502] == pytest.approx(-80.0, abs=1.0) + assert (psd - floor)[502] > 55.0 + + +def test_peak_hold_beats_averaging_on_bursts(): + fs = 2_048_000 + x = np.zeros(102400, dtype=np.complex64) + t = np.arange(102400) / fs + x[:8000] = 0.5 * np.exp(2j * np.pi * 100_000 * t[:8000]) + rng = np.random.default_rng(1) + x += 0.01 * (rng.standard_normal(102400) + 1j * rng.standard_normal(102400)) + _, p_avg = dsp.welch_psd(x, 1024, combine="mean") + _, p_max = dsp.welch_psd(x, 1024, combine="max") + avg = (dsp.db(p_avg) - dsp.noise_floor_curve(dsp.db(p_avg))).max() + peak = (dsp.db(p_max) - dsp.noise_floor_curve(dsp.db(p_max))).max() + assert peak > avg + + +def test_occupied_bandwidth_and_flatness(): + psd = np.zeros(1024) + psd[500:524] = 1.0 + bw, offset = dsp.occupied_bandwidth(psd, 100.0) + assert bw == pytest.approx(2400.0, rel=0.15) + assert dsp.spectral_flatness(np.ones(256)) == pytest.approx(1.0) + tone = np.full(256, 1e-9) + tone[128] = 1.0 + assert dsp.spectral_flatness(tone) < 0.01 + + +def test_decimator_matches_a_direct_filter_reference(): + """The polyphase form must equal filtering then discarding samples. + + It replaced an lfilter that computed every output and threw most away; + that was slow enough to stop captures keeping up with real time, which + shows up as recordings that play too fast. + """ + from scipy import signal as sps + rng = np.random.default_rng(4) + x = (rng.standard_normal(30000) + 1j * rng.standard_normal(30000)).astype(np.complex64) + for factor in (2, 4, 8): + d = dsp.FIRDecimator(factor) + got = d(x) + ref = sps.lfilter(d.taps.astype(np.float64), [1.0], + x.astype(np.complex128))[::factor] + assert np.allclose(got, ref[:got.size], atol=1e-4), f"factor {factor}" + + +def test_quarter_rate_mixer_matches_an_explicit_reference(): + """The trig-free shortcut must be exact, and stay phase-continuous.""" + rng = np.random.default_rng(5) + n = 9000 + x = (rng.standard_normal(n) + 1j * rng.standard_normal(n)).astype(np.complex64) + fs = 2_048_000.0 + mixer = dsp.Mixer(fs / 4, fs) + assert mixer._is_quarter_rate + got = np.concatenate([mixer(x[:3000]), mixer(x[3000:5000]), mixer(x[5000:])]) + ref = x * np.exp(-2j * np.pi * (fs / 4) * np.arange(n) / fs) + assert np.allclose(got, ref, atol=1e-5) + + +def test_decimation_is_fast_enough_for_realtime(): + """A 50 ms block must decimate in well under 50 ms of CPU.""" + import time + sr, n = 2_048_000, 102_400 + x = (np.random.default_rng(6).standard_normal(n) + + 1j * np.random.default_rng(7).standard_normal(n)).astype(np.complex64) + chain = dsp.DecimationChain(64) + chain(x) + t0 = time.perf_counter() + for _ in range(5): + chain(x) + per_block = (time.perf_counter() - t0) / 5 + budget = n / sr + assert per_block < 0.35 * budget, \ + f"{per_block*1000:.1f} ms per {budget*1000:.0f} ms block" diff --git a/tests/test_morse.py b/tests/test_morse.py new file mode 100644 index 0000000..fb5d5d0 --- /dev/null +++ b/tests/test_morse.py @@ -0,0 +1,50 @@ +import numpy as np +import pytest + +from morse_gen import morse_audio +from bandsaunter.morse import decode_morse, encode_morse + +FS = 16000 + +MESSAGES = [ + ("CQ CQ DE W1AW K", 18, 25), ("SOS SOS", 12, 20), + ("TEST DE N0CALL 599 TU", 25, 18), ("HELLO WORLD 73", 35, 15), + ("PARIS PARIS", 8, 25), ("VVV DE K1ABC", 22, 12), + ("QRZ? DE VE3XYZ/M", 20, 20), ("R 5NN TU 73 GL", 30, 22), + ("MAYDAY MAYDAY", 15, 8), ("THE QUICK BROWN FOX 1234567890", 28, 20), + ("CQ DX DE W1AW", 18, 6), +] + + +@pytest.mark.parametrize("msg,wpm,snr", MESSAGES) +def test_decodes_exactly(msg, wpm, snr): + r = decode_morse(morse_audio(msg, wpm, FS, snr), FS) + assert r.text.strip() == msg + assert r.is_morse + + +@pytest.mark.parametrize("wpm", [8, 15, 22, 30, 40]) +def test_speed_estimate_is_accurate(wpm): + r = decode_morse(morse_audio("CQ DE W1AW TEST", wpm, FS, 20), FS) + assert r.wpm == pytest.approx(wpm, rel=0.10) + + +def test_tone_frequency_found(): + r = decode_morse(morse_audio("CQ TEST", 20, FS, 20, tone=1100.0), FS) + assert r.tone_hz == pytest.approx(1100.0, abs=25) + + +def test_noise_is_not_morse(): + rng = np.random.default_rng(0) + r = decode_morse(0.1 * rng.standard_normal(FS * 3), FS) + assert not r.is_morse + assert r.text == "" + + +def test_silence_is_rejected(): + assert not decode_morse(np.zeros(FS * 2), FS).is_morse + + +def test_encode_round_trip(): + assert encode_morse("SOS") == "... --- ..." + assert encode_morse("A B") == ".- / -..." diff --git a/tests/test_quality.py b/tests/test_quality.py new file mode 100644 index 0000000..e7f94d8 --- /dev/null +++ b/tests/test_quality.py @@ -0,0 +1,217 @@ +"""Tests for the content gate: what counts as a signal worth recording.""" +import numpy as np +import pytest +from scipy.signal import butter, lfilter + +from bandsaunter.classify import classify +from bandsaunter.demod import make_demodulator +from bandsaunter.morse import decode_morse +from bandsaunter.quality import (RAYLEIGH_CV, assess, digital_structure, + noise_likeness, voice_metrics) +from bandsaunter import dsp +from bandsaunter.simulator import SimulatedDevice, VirtualTransmitter as V +from speech import synth_speech +from signals import FS, make + +AUDIO_FS = 16000 + + +def _tone(hz, seconds=5.0, fs=AUDIO_FS): + t = np.arange(int(seconds * fs)) / fs + return np.sin(2 * np.pi * hz * t) + + +# --------------------------------------------------------------------------- +# Voice detection +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("pitch,seed", [(110, 0), (150, 1), (210, 2)]) +def test_speech_scores_high(pitch, seed): + m = voice_metrics(synth_speech(5.0, AUDIO_FS, pitch, seed), AUDIO_FS) + assert m.score > 0.6, m.describe() + assert m.pitch_hz == pytest.approx(pitch, rel=0.2) + assert m.voiced_fraction > 0.2 + + +@pytest.mark.parametrize("name,signal", [ + ("white noise", np.random.default_rng(0).standard_normal(AUDIO_FS * 5)), + ("steady tone", _tone(1000.0)), + ("mains-style hum", sum(_tone(120.0 * k) / k for k in range(1, 6))), + ("silence", np.zeros(AUDIO_FS * 3)), +]) +def test_non_speech_scores_low(name, signal): + assert voice_metrics(signal, AUDIO_FS).score < 0.42, name + + +def test_static_scores_below_speech(): + b, a = butter(4, [300 / (AUDIO_FS / 2), 3000 / (AUDIO_FS / 2)], btype="band") + static = lfilter(b, a, np.random.default_rng(1).standard_normal(AUDIO_FS * 5)) + speech = synth_speech(5.0, AUDIO_FS, 120, 0) + assert voice_metrics(static, AUDIO_FS).score < \ + voice_metrics(speech, AUDIO_FS).score - 0.3 + + +def test_pitch_variation_separates_voice_from_a_buzz(): + """A talker's pitch wanders; a hum sits on one frequency.""" + assert voice_metrics(synth_speech(5.0, AUDIO_FS, 120, 0), + AUDIO_FS).pitch_variation > 0.02 + hum = sum(_tone(120.0 * k) / k for k in range(1, 6)) + assert voice_metrics(hum, AUDIO_FS).pitch_variation < 0.01 + + +# --------------------------------------------------------------------------- +# Noise and structure scoring +# --------------------------------------------------------------------------- + +def test_noise_likeness_high_for_noise_low_for_signals(): + from bandsaunter.classify import extract_features + noise = noise_likeness(extract_features(make("noise"), FS, snr_db=0)) + assert noise > 0.7 + for kind in ("nfm", "fsk4", "cw", "am"): + assert noise_likeness(extract_features(make(kind), FS, snr_db=28)) < noise + + +def test_rayleigh_constant(): + """Complex Gaussian noise has this exact envelope coefficient of variation.""" + rng = np.random.default_rng(0) + x = (rng.standard_normal(200000) + 1j * rng.standard_normal(200000)) + env = np.abs(x) + assert env.std() / env.mean() == pytest.approx(RAYLEIGH_CV, rel=0.02) + + +def test_symbol_rate_alone_is_not_digital(): + """A baud estimate with no identified keying proves nothing.""" + from bandsaunter.classify import extract_features + f = extract_features(make("noise"), FS, snr_db=0) + score, bits = digital_structure(f) + assert score == 0.0 and bits == [] + + +def test_fsk_and_ook_are_digital(): + from bandsaunter.classify import extract_features + for kind in ("fsk2", "fsk4"): + score, bits = digital_structure(extract_features(make(kind), FS, snr_db=28)) + assert score > 0.3, f"{kind}: {bits}" + + +# --------------------------------------------------------------------------- +# End-to-end verdicts through the real RF chain +# --------------------------------------------------------------------------- + +def _through_radio(mode, freq_hz, demod_mode, bw=12_500, seconds=3.0, **kw): + """Run a simulated transmitter through tune, mix, decimate and demodulate.""" + sr = 2_048_000 + tx = [] if mode is None else [V(146_520_000, mode, 0.4, bw, "tx", **kw)] + dev = SimulatedDevice(transmitters=tx).open() + dev.tune(146_520_000 - sr * 0.25) + mixer = dsp.Mixer(sr * 0.25, sr) + demod = make_demodulator(demod_mode, sr, bw, 16_000) + audio, iq = [], [] + for _ in range(int(seconds / 0.05)): + a, i = demod.step(mixer(dev.read_samples(102400))) + audio.append(a) + iq.append(i) + audio = np.concatenate(audio) + iq = np.concatenate(iq) + cls = classify(iq, demod.if_rate, freq_hz=freq_hz, snr_db=28) + morse = None + if cls.family in ("cw", "ook", "carrier"): + cw = make_demodulator("cw", int(demod.if_rate), 800.0, 16_000) + morse = decode_morse(cw.process(iq), cw.audio_rate) + return assess(cls, audio, demod.audio_rate, morse=morse, freq_hz=freq_hz) + + +@pytest.mark.parametrize("mode,freq,dmode,bw,kw,want", [ + ("nfm", 146.52e6, "nfm", 12_500, {"ctcss": 100.0}, "voice"), + ("nfm", 146.52e6, "nfm", 12_500, {}, "voice"), + ("am", 121.5e6, "am", 8_000, {}, "voice"), + ("usb", 14.2e6, "usb", 3_000, {}, "voice"), + ("wfm", 97.5e6, "wfm", 180_000, {"deviation": 75_000}, "voice"), + ("carrier", 446.0e6, "nfm", 12_500, {}, "carrier"), + ("fsk4", 460.0e6, "nfm", 12_500, {"baud": 4800, "deviation": 1800}, "digital"), + ("fsk2", 929.6e6, "nfm", 12_500, {"baud": 1200, "deviation": 2400}, "digital"), + ("ook", 433.92e6, "nfm", 40_000, {"baud": 2000}, "digital"), + ("cw", 14.05e6, "cw", 800, {"wpm": 18}, "cw"), + (None, 300.0e6, "nfm", 12_500, {}, "noise"), +]) +def test_content_categories(mode, freq, dmode, bw, kw, want): + v = _through_radio(mode, freq, dmode, bw, **kw) + assert v.category == want, f"{mode} -> {v.category}: {v.reason}" + + +def test_accept_list_controls_what_is_kept(): + v = _through_radio("carrier", 446.0e6, "nfm") + assert v.category == "carrier" and not v.accept + v2 = _through_radio("nfm", 146.52e6, "nfm") + assert v2.category == "voice" and v2.accept + + +def test_broadcast_leniency_does_not_apply_outside_the_fm_band(): + """A wideband hump at 250 MHz is not a broadcast station.""" + v = _through_radio("wfm", 250.0e6, "wfm", 180_000, deviation=75_000) + assert v.category != "voice" or v.voice.score >= 0.45 + + +# --------------------------------------------------------------------------- +# Regressions: each of these let static through at some point. +# --------------------------------------------------------------------------- + +def test_voicing_is_necessary_not_merely_weighted(): + """Noise can satisfy dynamics, syllable rate and band -- but not pitch. + + Weighting voicing alongside the other three let hiss score over the line + on their strength alone, which is how static ended up recorded as speech. + """ + rng = np.random.default_rng(7) + n = AUDIO_FS * 5 + t = np.arange(n) / AUDIO_FS + b, a = butter(4, [300 / (AUDIO_FS / 2), 3000 / (AUDIO_FS / 2)], btype="band") + hiss = lfilter(b, a, rng.standard_normal(n)) + # Give it speech-like dynamics and a 4 Hz syllabic envelope. + hiss *= (0.15 + 0.85 * np.abs(np.sin(2 * np.pi * 4.0 * t))) * \ + (np.sin(2 * np.pi * 0.3 * t) > -0.4) + m = voice_metrics(hiss, AUDIO_FS) + assert m.dynamic_range_db > 10, "test signal should have big dynamics" + assert m.syllabic > 0.2, "test signal should have syllabic modulation" + assert m.voiced_fraction < 0.15, "but no pitch track" + assert m.score < 0.45, f"shaped hiss scored {m.score:.2f} as voice" + + +def test_pitch_pinned_at_the_search_limit_is_not_a_pitch(): + """A maximum on the edge of the search range is the search running out.""" + rng = np.random.default_rng(3) + m = voice_metrics(rng.standard_normal(AUDIO_FS * 4), AUDIO_FS) + assert not (0 < m.pitch_hz < 75) and m.pitch_hz < 395 or m.pitch_hz == 0.0 + + +def test_symbol_rate_must_hold_across_the_capture(): + """Real data keeps one symbol rate; an estimator on noise does not.""" + from bandsaunter.classify import extract_features + for kind in ("fsk2", "fsk4", "psk2", "psk4"): + f = extract_features(make(kind), FS, snr_db=28) + assert f.baud_stability > 0.8, f"{kind}: {f.baud_stability}" + for kind in ("noise", "nfm", "carrier"): + f = extract_features(make(kind), FS, snr_db=28) + assert f.baud_stability < 0.8, f"{kind}: {f.baud_stability}" + + +def test_keying_needs_a_regular_grid(): + """On/off contrast alone is not keying: a fading carrier produces plenty.""" + from bandsaunter.classify import extract_features + keyed = extract_features(make("cw"), FS, snr_db=28) + noise = extract_features(make("noise"), FS, snr_db=0) + assert keyed.keying_regularity > noise.keying_regularity + + +def test_a_fading_carrier_is_not_digital(): + """A carrier drifting across the squelch has contrast but no symbol grid.""" + from bandsaunter.classify import extract_features + rng = np.random.default_rng(5) + n = 64000 + t = np.arange(n) / FS + # Slow random fading, nothing quantised about it. + fade = np.interp(t, np.linspace(0, t[-1], 40), rng.random(40) ** 2) + x = (fade * np.exp(2j * np.pi * 300 * t)).astype(np.complex64) + x += 0.02 * (rng.standard_normal(n) + 1j * rng.standard_normal(n)) + score, bits = digital_structure(extract_features(x, FS, snr_db=20)) + assert score <= 0.3, f"fading carrier scored as digital: {bits}" diff --git a/tests/test_ranges.py b/tests/test_ranges.py new file mode 100644 index 0000000..bece1ed --- /dev/null +++ b/tests/test_ranges.py @@ -0,0 +1,213 @@ +import pytest + +from bandsaunter.bandplan import PRESETS, by_key, fmt_hz +from bandsaunter.ranges import (RangeError, ScanRange, build_plan, + parse_frequency, parse_range, parse_range_list) + + +@pytest.mark.parametrize("text,expected", [ + ("146.52M", 146_520_000), ("146520000", 146_520_000), + ("433.92 MHz", 433_920_000), ("14074k", 14_074_000), + ("1.09G", 1_090_000_000), ("162.4", 162_400_000), ("500Hz", 500), +]) +def test_parse_frequency(text, expected): + assert parse_frequency(text) == pytest.approx(expected) + + +def test_parse_range_forms(): + assert parse_range("144M-148M").start == 144e6 + # the unit on the right end carries over to the left + r = parse_range("144-148M") + assert (r.start, r.stop) == (144e6, 148e6) + r = parse_range("144M-148M/25k@nfm") + assert r.step == 25_000 and r.mode == "nfm" + assert parse_range("gmrs").preset_key == "gmrs" + + +def test_single_frequency_gets_width(): + r = parse_range("146.52M") + assert r.stop > r.start + + +def test_parse_range_list_unlimited(): + rs = parse_range_list("144M-148M, 420-450M, gmrs, 162.4-162.55M, 88-108M") + assert len(rs) == 5 + + +def test_bad_input_raises(): + with pytest.raises(RangeError): + parse_range("banana") + + +def test_plan_parks_lo_clear_of_the_covered_span(): + """The DC spike must never land inside the frequencies being searched.""" + steps = build_plan(parse_range_list("144.05M-144.15M"), 2_048_000, + dc_guard=8000.0) + assert len(steps) == 1 + s = steps[0] + assert s.center < s.low + assert s.low - s.center == pytest.approx(8000.0) + + +def test_plan_covers_whole_range_contiguously(): + ranges = parse_range_list("144M-148M") + steps = build_plan(ranges, 2_048_000) + assert steps[0].low == pytest.approx(144e6) + assert steps[-1].high == pytest.approx(148e6) + for a, b in zip(steps, steps[1:]): + assert a.high == pytest.approx(b.low) + + +def test_band_plan_is_well_formed(): + keys = [p.key for p in PRESETS] + assert len(keys) == len(set(keys)), "duplicate preset keys" + for p in PRESETS: + assert p.stop > p.start, f"{p.key} has an empty span" + assert p.mode in ("nfm", "wfm", "am", "usb", "lsb", "cw", "raw", + "auto") + # "auto" defers to the segments beneath it, so it must not also + # declare a bandwidth that would be used in preference to theirs. + if p.mode == "auto": + assert p.bandwidth == 0, f"{p.key} both defers and dictates" + assert by_key(p.key) is p + + +def test_fmt_hz(): + assert fmt_hz(146_520_000) == "146.52 MHz" + assert fmt_hz(1_090_000_000) == "1.09 GHz" + assert fmt_hz(500) == "500 Hz" + + +# --------------------------------------------------------------------------- +# Presets that stand for a set of others +# --------------------------------------------------------------------------- + +def test_all_cw_expands_to_every_cw_segment(): + from bandsaunter.bandplan import PRESETS, by_key + group = by_key("all-cw") + assert group is not None and group.is_group + members = group.expand() + # every CW preset in the plan should be in it, and nothing else + cw_keys = {p.key for p in PRESETS if p.mode == "cw" and not p.is_group} + assert {p.key for p in members} == cw_keys, "the group and the plan disagree" + assert all(p.mode == "cw" for p in members) + assert len(members) >= 10 + + +def test_a_group_becomes_one_range_per_member(): + ranges = parse_range_list("all-cw") + assert len(ranges) >= 10 + assert all(r.mode == "cw" for r in ranges) + # separate ranges, not one span from the lowest to the highest + assert all(r.span < 1e6 for r in ranges), [r.span for r in ranges] + assert sum(r.span for r in ranges) < 5e6 + + +def test_a_group_declares_the_extent_of_its_members(): + """The displayed span must not drift from what the group actually covers.""" + from bandsaunter.bandplan import PRESETS + for group in [p for p in PRESETS if p.is_group]: + members = group.expand() + assert group.start == pytest.approx(min(m.start for m in members)) + assert group.stop == pytest.approx(max(m.stop for m in members)) + + +def test_group_members_all_exist(): + from bandsaunter.bandplan import PRESETS, by_key + for group in [p for p in PRESETS if p.is_group]: + for key in group.members: + assert by_key(key) is not None, f"{group.key} names missing {key}" + assert len(group.expand()) == len(group.members) + + +def test_a_group_is_refused_where_only_one_range_fits(): + with pytest.raises(RangeError, match="separate ranges"): + parse_range("all-cw") + + +def test_groups_do_not_label_detections(): + """A group spans a huge range; it must not be used to name a signal.""" + from bandsaunter.bandplan import presets_covering + assert all(not p.is_group for p in presets_covering(14_050_000)) + + +def test_a_group_mixes_with_ordinary_ranges(): + ranges = parse_range_list("all-cw, 144M-148M, gmrs") + labels = [r.label for r in ranges] + assert "2 m CW" in labels + assert "GMRS / FRS" in labels + assert any("144 MHz-148 MHz" in x for x in labels) + + +def test_the_cw_segments_are_inside_their_bands(): + """A CW segment that strays outside its amateur band would be wrong.""" + from bandsaunter.bandplan import by_key + bands = {"160m-cw": "160m", "80m-cw": "80m", "40m-cw": "40m", + "20m-cw": "20m", "17m-cw": "17m", "15m-cw": "15m", + "12m-cw": "12m", "10m-cw": "10m", "6m-cw": "6m", "2m-cw": "2m"} + for cw_key, band_key in bands.items(): + cw, band = by_key(cw_key), by_key(band_key) + assert cw.start >= band.start, cw_key + assert cw.stop <= band.stop, cw_key + + +# --------------------------------------------------------------------------- +# Whole amateur bands +# --------------------------------------------------------------------------- + +COMPLETE_BANDS = ["160m", "80m", "60m", "40m", "30m", "20m", "17m", "15m", + "12m", "10m", "6m", "2m", "1.25m", "70cm", "33cm"] + + +@pytest.mark.parametrize("band", COMPLETE_BANDS) +def test_every_amateur_band_has_a_complete_entry(band): + from bandsaunter.bandplan import by_key + whole = by_key(f"{band}-complete") + assert whole is not None, f"no {band}-complete" + assert whole.category == "Amateur Radio" + assert whole.mode == "auto" + # It must actually cover the band it names. + plain = by_key(band) + assert whole.start <= plain.start and whole.stop >= plain.stop + + +@pytest.mark.parametrize("band", COMPLETE_BANDS) +def test_a_complete_band_resolves_a_real_mode_everywhere(band): + """"auto" must never survive to the demodulator.""" + ranges = parse_range_list(f"{band}-complete") + assert len(ranges) == 1 + r = ranges[0] + step = r.span / 200 + freq = r.start + while freq < r.stop: + mode = r.resolved_mode(freq) + assert mode in ("nfm", "wfm", "am", "usb", "lsb", "cw", "raw"), \ + f"{band} at {freq}: {mode}" + assert r.resolved_bandwidth(freq) > 0, f"{band} at {freq}" + freq += step + + +def test_2m_complete_covers_cw_then_ssb_then_fm(): + r = parse_range_list("2m-complete")[0] + assert r.resolved_mode(144_050_000) == "cw" + assert r.resolved_mode(144_200_000) == "usb" + assert r.resolved_mode(146_520_000) == "nfm" + + +def test_70cm_complete_is_not_hijacked_by_the_ism_band(): + """433 MHz is shared; inside an amateur sweep the amateur reading wins.""" + r = parse_range_list("70cm-complete")[0] + assert r.resolved_mode(433_920_000) == "nfm" + assert r.resolved_mode(432_050_000) == "cw" + assert r.resolved_mode(432_200_000) == "usb" + + +def test_a_plain_range_still_gets_the_ism_reading(): + assert ScanRange(433.9e6, 433.95e6, mode="auto").resolved_mode(433.92e6) \ + == "raw" + + +def test_complete_bands_are_offered_where_they_would_be_looked_for(): + from bandsaunter.bandplan import in_category + keys = {p.key for p in in_category("Amateur Radio")} + assert {f"{b}-complete" for b in COMPLETE_BANDS} <= keys diff --git a/tests/test_scanner.py b/tests/test_scanner.py new file mode 100644 index 0000000..d5a9945 --- /dev/null +++ b/tests/test_scanner.py @@ -0,0 +1,510 @@ +"""End-to-end scanner tests driven by the built-in simulator.""" +import json +import re +import wave +from pathlib import Path + +import numpy as np +import pytest + +from bandsaunter.config import ScanConfig +from bandsaunter.ranges import parse_range_list +from bandsaunter.scanner import Scanner, ScannerCallbacks +from bandsaunter.simulator import SimulatedDevice, VirtualTransmitter as V + + +def run_scan(tmp_path, transmitters, ranges, **over): + dev = SimulatedDevice(transmitters=transmitters).open() + cfg = ScanConfig(ranges=parse_range_list(ranges), output_dir=str(tmp_path), + record_seconds=3.0, hang_seconds=1.0, + min_record_seconds=0.3, threshold_db=12, + dwell_seconds=0.05, max_cycles=2, revisit_seconds=0.2) + cfg.accept = list(over.pop("accept", cfg.accept)) + for k, v in over.items(): + setattr(cfg, k, v) + hits = [] + s = Scanner(cfg, device=dev, + callbacks=ScannerCallbacks(on_record_end=hits.append)) + s.prepare() + s.run() + return s, [h for h in hits if h.kept] + + +def test_finds_and_identifies_narrowband_fm(tmp_path): + tx = [V(146_520_000, "nfm", 0.4, 12_500, "voice", ctcss=100.0)] + s, hits = run_scan(tmp_path, tx, "146.4M-146.6M") + assert hits, "nothing recorded" + h = hits[0] + assert h.frequency == pytest.approx(146_520_000, abs=8_000) + assert h.category == "voice" + assert h.family == "nfm", h.classification + assert "voice" in h.classification.lower() + assert h.ctcss_hz == pytest.approx(100.0, abs=0.5) + + +def test_record_seconds_is_honoured_exactly(tmp_path): + tx = [V(146_520_000, "nfm", 0.4, 12_500, "voice")] + s, hits = run_scan(tmp_path, tx, "146.4M-146.6M", record_seconds=2.0) + assert hits + for h in hits: + assert h.duration == pytest.approx(2.0, abs=0.15) + assert "record limit" in h.stop_reason + + +def test_hang_seconds_ends_a_finished_transmission(tmp_path): + """A transmitter that stops must release the scanner after the hang time.""" + tx = [V(146_520_000, "nfm", 0.4, 12_500, "burst", + period_seconds=6.0, on_seconds=1.0)] + s, hits = run_scan(tmp_path, tx, "146.4M-146.6M", + record_seconds=20.0, hang_seconds=0.8, max_cycles=3) + assert hits, "nothing recorded" + ended_on_silence = [h for h in hits if "quiet" in h.stop_reason] + assert ended_on_silence, [h.stop_reason for h in hits] + for h in ended_on_silence: + assert h.duration < 20.0 + + +def test_min_record_seconds_discards_blips(tmp_path): + """A transmission shorter than the minimum must leave nothing behind.""" + tx = [V(146_520_000, "nfm", 0.5, 12_500, "blip", + period_seconds=3.0, on_seconds=0.35)] + dev = SimulatedDevice(transmitters=tx).open() + cfg = ScanConfig(ranges=parse_range_list("146.4M-146.6M"), + output_dir=str(tmp_path), record_seconds=20.0, + min_record_seconds=6.0, hang_seconds=0.5, + threshold_db=12, dwell_seconds=0.05, max_cycles=4, + revisit_seconds=0.1) + all_hits = [] + s = Scanner(cfg, device=dev, + callbacks=ScannerCallbacks(on_record_end=all_hits.append)) + s.prepare() + s.run() + assert all_hits, "the blip was never detected" + assert not any(h.kept for h in all_hits) + assert s.stats.discarded >= 1 + # Nothing should have been left behind on disk. + assert not list(tmp_path.glob("*/*/audio.wav")) + + +def test_invalid_config_is_rejected_before_touching_the_radio(tmp_path): + cfg = ScanConfig(ranges=parse_range_list("146.4M-146.6M"), + output_dir=str(tmp_path), + record_seconds=5.0, min_record_seconds=9.0) + s = Scanner(cfg, device=SimulatedDevice().open()) + with pytest.raises(ValueError, match="min_record_seconds"): + s.prepare() + + +def test_decodes_morse_from_a_cw_beacon(tmp_path): + tx = [V(144_100_000, "cw", 0.35, 500, "beacon", wpm=18, + message="VVV DE W1AW")] + # Several sweeps, because a 50 ms dwell can easily land in a key-up gap. + s, hits = run_scan(tmp_path, tx, "144.05M-144.15M", + record_seconds=10.0, hang_seconds=4.0, max_cycles=8, + revisit_seconds=0.1) + assert hits + h = hits[0] + assert h.mode == "cw", "should have chosen the CW demodulator" + assert "W1AW" in h.morse_text or "VVV" in h.morse_text, h.morse_text + assert h.morse_wpm == pytest.approx(18, rel=0.15) + + +def test_lockout_is_respected(tmp_path): + tx = [V(146_520_000, "nfm", 0.4, 12_500, "voice")] + dev = SimulatedDevice(transmitters=tx).open() + cfg = ScanConfig(ranges=parse_range_list("146.4M-146.6M"), + output_dir=str(tmp_path), record_seconds=2.5, + threshold_db=12, dwell_seconds=0.05, max_cycles=2, + lockout=[146_520_000.0]) + s = Scanner(cfg, device=dev) + s.prepare() + s.run() + assert s.stats.recordings == 0 + + +def test_writes_audio_iq_and_metadata(tmp_path): + tx = [V(146_520_000, "nfm", 0.4, 12_500, "voice")] + s, hits = run_scan(tmp_path, tx, "146.4M-146.6M", + record_seconds=2.5, save_iq=True, max_cycles=1) + assert hits + h = hits[0] + with wave.open(h.audio_path) as w: + assert w.getnframes() / w.getframerate() == pytest.approx(2.5, abs=0.2) + assert w.getnchannels() == 1 + iq = np.fromfile(h.iq_path, dtype=np.complex64) + assert iq.size > 0 + meta = json.loads(open(h.meta_path).read()) + assert meta["frequency_hz"] == pytest.approx(h.frequency) + assert meta["hit"]["classification"] + sigmf = json.loads(open(h.iq_path.replace(".cf32", ".sigmf-meta")).read()) + assert sigmf["global"]["core:datatype"] == "cf32_le" + # the run log + assert (tmp_path / "scan_log.jsonl").exists() + lines = (tmp_path / "scan_log.csv").read_text().strip().split("\n") + assert len(lines) >= 2 + + +def test_multiple_signals_in_one_step_all_get_visited(tmp_path): + tx = [V(146_500_000, "nfm", 0.40, 12_500, "a"), + V(146_800_000, "nfm", 0.30, 12_500, "b"), + V(147_100_000, "carrier", 0.25, 1_000, "c")] + s, hits = run_scan(tmp_path, tx, "146.4M-147.2M", record_seconds=2.5, + max_cycles=2, accept=["voice", "cw", "digital", "carrier"]) + found = {round(h.frequency / 1e5) for h in hits} + assert len(found) >= 3, f"only found {found}" + + +def test_unreachable_frequencies_are_skipped_not_fatal(tmp_path): + tx = [V(146_520_000, "nfm", 0.4, 12_500, "voice")] + # 2.5 GHz is beyond the tuner; the scan must still run the reachable part. + s, hits = run_scan(tmp_path, tx, "146.4M-146.6M, 2500M-2600M", + record_seconds=2.5, max_cycles=1) + assert hits + + +# --------------------------------------------------------------------------- +# The content gate: only voice, CW and digital should reach the disk. +# --------------------------------------------------------------------------- + +def test_static_is_never_recorded(tmp_path): + """An empty band must produce no files, whatever the squelch does.""" + dev = SimulatedDevice(transmitters=[], noise_amplitude=0.05).open() + cfg = ScanConfig(ranges=parse_range_list("146M-148M"), + output_dir=str(tmp_path), record_seconds=4.0, + hang_seconds=1.0, threshold_db=2.0, # deliberately wide open + dwell_seconds=0.05, max_cycles=2, revisit_seconds=0.1) + hits = [] + s = Scanner(cfg, device=dev, + callbacks=ScannerCallbacks(on_record_end=hits.append)) + s.prepare() + s.run() + assert not any(h.kept for h in hits), "static was recorded" + assert not list(tmp_path.glob("*/*/audio.wav")) + + +def test_bare_carrier_is_rejected_by_default(tmp_path): + tx = [V(146_520_000, "carrier", 0.4, 1_000, "dead carrier")] + s, hits = run_scan(tmp_path, tx, "146.4M-146.6M", record_seconds=3.0) + assert not hits, "an unmodulated carrier is not a signal worth keeping" + assert not list(tmp_path.glob("*/*/audio.wav")) + + +def test_bare_carrier_kept_when_asked_for(tmp_path): + tx = [V(146_520_000, "carrier", 0.4, 1_000, "dead carrier")] + s, hits = run_scan(tmp_path, tx, "146.4M-146.6M", record_seconds=3.0, + accept=["voice", "cw", "digital", "carrier"]) + assert hits and hits[0].category == "carrier" + + +@pytest.mark.parametrize("mode,bw,category", [ + ("nfm", 12_500, "voice"), + ("am", 8_000, "voice"), + ("fsk4", 12_500, "digital"), + ("fsk2", 12_500, "digital"), +]) +def test_real_signals_are_categorised_and_kept(tmp_path, mode, bw, category): + tx = [V(146_520_000, mode, 0.4, bw, "tx", baud=4800 if mode == "fsk4" else 1200, + deviation=1800 if mode.startswith("fsk") else 2500)] + s, hits = run_scan(tmp_path, tx, "146.4M-146.6M", record_seconds=4.0) + assert hits, f"{mode} was rejected" + assert hits[0].category == category, \ + f"{mode} -> {hits[0].category}: {hits[0].content_reason}" + + +def test_keep_everything_disables_the_gate(tmp_path): + """--keep-everything restores plain power-threshold behaviour.""" + dev = SimulatedDevice(transmitters=[], noise_amplitude=0.05).open() + cfg = ScanConfig(ranges=parse_range_list("146M-148M"), + output_dir=str(tmp_path), record_seconds=1.0, + hang_seconds=0.5, threshold_db=1.0, min_record_seconds=0.2, + dwell_seconds=0.05, max_cycles=2, revisit_seconds=0.1, + require_signal=False, detector_bias_db=0.0) + hits = [] + s = Scanner(cfg, device=dev, + callbacks=ScannerCallbacks(on_record_end=hits.append)) + s.prepare() + s.run() + assert s.stats.detections > 0, "threshold was not wide open enough" + + +def test_filenames_are_flat_dated_and_named_for_the_modulation(tmp_path): + tx = [V(146_520_000, "nfm", 0.4, 12_500, "voice")] + s, hits = run_scan(tmp_path, tx, "146.4M-146.6M", record_seconds=3.0) + assert hits + wavs = list(tmp_path.glob("*.wav")) + assert wavs, "no wav written at the top level" + assert not list(tmp_path.glob("*/*.wav")), "files should not be in subfolders" + name = wavs[0].name + assert re.match(r"^\d{4}\.\d{6}MHz--\d{4}-\d{2}-\d{2}_" + r"\d{2}_\d{2}_\d{2}-[a-z0-9-]+\.wav$", name), name + assert name.endswith("-nfm.wav"), name + assert name.startswith("0146.5"), name + # every artefact of one capture shares a stem + stem = wavs[0].stem + assert (tmp_path / f"{stem}.json").exists() + + +def test_recorded_length_matches_the_requested_record_time(tmp_path): + """The WAV must hold as many seconds as the scanner says it captured. + + A mismatch here is what makes a recording play at the wrong speed. + """ + tx = [V(146_520_000, "nfm", 0.4, 12_500, "voice")] + s, hits = run_scan(tmp_path, tx, "146.4M-146.6M", record_seconds=3.0) + assert hits + h = hits[0] + with wave.open(h.audio_path) as w: + seconds = w.getnframes() / w.getframerate() + assert seconds == pytest.approx(3.0, abs=0.15) + assert seconds == pytest.approx(h.duration, abs=0.02) + + +# --------------------------------------------------------------------------- +# Two-way conversations: natural pauses must not end the capture. +# --------------------------------------------------------------------------- + +def _two_way(freq=146_520_000): + """Two stations alternating on one frequency with a 1.5 s gap between overs. + + A is up for [0, 2.5), B for [4, 6.5), repeating every 8 s. + """ + return [ + V(freq, "nfm", 0.40, 12_500, "station A", + period_seconds=8.0, on_seconds=2.5, phase_offset=0.0), + V(freq, "nfm", 0.35, 12_500, "station B", pitch_hz=190.0, + period_seconds=8.0, on_seconds=2.5, phase_offset=4.0), + ] + + +def test_a_pause_between_overs_does_not_end_the_capture(tmp_path): + """With hang longer than the gap, both overs land in one recording.""" + s, hits = run_scan(tmp_path, _two_way(), "146.4M-146.6M", + record_seconds=0.0, hang_seconds=3.0, + max_record_seconds=14.0, max_cycles=1) + assert hits, "nothing recorded" + h = hits[0] + # Must span the 1.5 s gap and reach the second station's over. + assert h.duration > 6.0, \ + f"capture ended after {h.duration:.1f}s, so it cut at the pause" + assert len(list(tmp_path.glob("*.wav"))) == 1, \ + "the exchange was split into separate files" + + +def test_a_short_hang_stops_at_the_first_pause(tmp_path): + """With hang shorter than the gap, the capture ends when the over does.""" + s, hits = run_scan(tmp_path, _two_way(), "146.4M-146.6M", + record_seconds=0.0, hang_seconds=0.8, + max_record_seconds=14.0, max_cycles=1) + assert hits + h = hits[0] + assert h.duration < 5.0, f"ran {h.duration:.1f}s past a {0.8}s hang" + assert "quiet" in h.stop_reason or "no signal" in h.stop_reason + + +def test_an_established_conversation_is_never_abandoned_as_noise(tmp_path): + """Once speech is heard, a quiet spell must not trigger the content abort.""" + s, hits = run_scan(tmp_path, _two_way(), "146.4M-146.6M", + record_seconds=0.0, hang_seconds=3.0, + verify_max_seconds=2.0, # would fire during the gap + max_record_seconds=14.0, max_cycles=1) + assert hits, "an established conversation was thrown away" + assert hits[0].category == "voice" + + +def test_max_record_seconds_bounds_an_unlimited_capture(tmp_path): + """--record 0 must still not run forever.""" + tx = [V(146_520_000, "nfm", 0.4, 12_500, "continuous")] + s, hits = run_scan(tmp_path, tx, "146.4M-146.6M", record_seconds=0.0, + hang_seconds=5.0, max_record_seconds=5.0, max_cycles=1) + assert hits + assert hits[0].duration == pytest.approx(5.0, abs=0.3) + assert "safety limit" in hits[0].stop_reason + + +def test_interference_during_a_gap_does_not_hold_the_capture_open(tmp_path): + """Static breaking squelch mid-pause must still count as quiet. + + Otherwise a burst of noise after the conversation ends keeps the receiver + parked on a dead channel indefinitely. + """ + tx = [V(146_520_000, "nfm", 0.40, 12_500, "voice", + period_seconds=40.0, on_seconds=3.0, phase_offset=0.0), + V(146_520_000, "carrier", 0.35, 1_000, "interference", + period_seconds=40.0, on_seconds=8.0, phase_offset=36.0)] + s, hits = run_scan(tmp_path, tx, "146.4M-146.6M", record_seconds=0.0, + hang_seconds=2.0, max_record_seconds=20.0, max_cycles=1) + assert hits + h = hits[0] + # The carrier holds squelch open until 12 s; the capture must not wait it + # out. Some lag is unavoidable -- deciding a signal carries nothing takes + # a couple of seconds of evidence -- but it must be well short of that. + assert h.duration < 10.0, f"ran {h.duration:.1f}s on an empty carrier" + assert h.duration > 3.0, "cut the real speech short" + assert "silence, static or interference" in h.stop_reason + + +def test_being_cut_off_mid_transmission_is_reported(tmp_path): + """The record limit silently truncating a live signal must be visible. + + Otherwise the file just ends and there is nothing to say the hang time was + never the thing that stopped it. + """ + tx = [V(146_520_000, "nfm", 0.4, 12_500, "continuous")] + notes = [] + dev = SimulatedDevice(transmitters=tx).open() + cfg = ScanConfig(ranges=parse_range_list("146.4M-146.6M"), + output_dir=str(tmp_path), record_seconds=4.0, + hang_seconds=3.0, threshold_db=12, dwell_seconds=0.05, + max_cycles=1, revisit_seconds=0.2) + hits = [] + s = Scanner(cfg, device=dev, + callbacks=ScannerCallbacks(on_record_end=hits.append, + on_status=notes.append)) + s.prepare() + s.run() + assert hits and "record limit" in hits[0].stop_reason + assert s.stats.truncated == 1 + assert any("--record 0" in n for n in notes), notes + + +def test_no_warning_when_the_signal_really_ended(tmp_path): + tx = [V(146_520_000, "nfm", 0.4, 12_500, "burst", + period_seconds=12.0, on_seconds=2.0)] + s, hits = run_scan(tmp_path, tx, "146.4M-146.6M", record_seconds=20.0, + hang_seconds=1.0, max_cycles=3, revisit_seconds=0.1) + assert s.stats.truncated == 0 + + +def test_record_zero_follows_a_transmission_past_the_old_limit(tmp_path): + tx = [V(146_520_000, "nfm", 0.4, 12_500, "continuous")] + s, hits = run_scan(tmp_path, tx, "146.4M-146.6M", record_seconds=0.0, + hang_seconds=3.0, max_record_seconds=9.0, max_cycles=1) + assert hits + assert hits[0].duration > 8.0, "stopped early despite no record limit" + assert s.stats.truncated == 0 + + +# --------------------------------------------------------------------------- +# One file per frequency +# --------------------------------------------------------------------------- + +def _two_channels(): + return [V(146_520_000, "nfm", 0.40, 12_500, "A", + period_seconds=6.0, on_seconds=2.0, phase_offset=0.0), + V(147_100_000, "nfm", 0.35, 12_500, "B", pitch_hz=190.0, + period_seconds=6.0, on_seconds=2.0, phase_offset=3.0)] + + +def test_combining_gives_one_file_per_frequency(tmp_path): + s, hits = run_scan(tmp_path, _two_channels(), "146.4M-147.2M", + record_seconds=3.0, hang_seconds=1.0, max_cycles=4, + revisit_seconds=0.1, combine_by_frequency=True) + assert len(hits) >= 3, "not enough transmissions to test combining" + wavs = sorted(p.name for p in tmp_path.glob("*.wav")) + assert len(wavs) == 2, wavs + assert all(name.endswith("MHz.wav") for name in wavs), wavs + # every capture went into one of them + assert all(h.combined_path for h in hits) + + +def test_combining_removes_the_separate_files_by_default(tmp_path): + s, hits = run_scan(tmp_path, _two_channels(), "146.4M-147.2M", + record_seconds=3.0, max_cycles=3, revisit_seconds=0.1, + combine_by_frequency=True) + assert hits + dated = [p for p in tmp_path.glob("*.wav") if "--" in p.name] + assert not dated, f"per-transmission files left behind: {dated}" + # the metadata for each capture is still there + assert len(list(tmp_path.glob("*.json"))) >= len(hits) + + +def test_combining_can_keep_the_separate_files_too(tmp_path): + s, hits = run_scan(tmp_path, _two_channels(), "146.4M-147.2M", + record_seconds=3.0, max_cycles=3, revisit_seconds=0.1, + combine_by_frequency=True, combine_keep_individual=True) + assert hits + dated = [p for p in tmp_path.glob("*.wav") if "--" in p.name] + assert len(dated) == len(hits) + + +def test_a_combined_file_grows_with_each_reception(tmp_path): + from bandsaunter.recorder import read_wav + # Continuously active, so each sweep produces another capture and the + # combined file has to grow. + tx = [V(146_520_000, "nfm", 0.4, 12_500, "A")] + sizes = [] + + def note(hit): + if hit.kept and hit.combined_path: + sizes.append(read_wav(Path(hit.combined_path))[0].size) + + dev = SimulatedDevice(transmitters=tx).open() + cfg = ScanConfig(ranges=parse_range_list("146.4M-146.6M"), + output_dir=str(tmp_path), record_seconds=2.0, + hang_seconds=1.0, threshold_db=12, dwell_seconds=0.05, + max_cycles=3, revisit_seconds=0.05, + combine_by_frequency=True, announce_timestamps=False) + s = Scanner(cfg, device=dev, callbacks=ScannerCallbacks(on_record_end=note)) + s.prepare() + s.run() + assert len(sizes) >= 2, sizes + assert sizes == sorted(sizes) and sizes[-1] > sizes[0], sizes + + +def test_combining_off_keeps_the_old_layout(tmp_path): + s, hits = run_scan(tmp_path, _two_channels(), "146.4M-147.2M", + record_seconds=3.0, max_cycles=2, revisit_seconds=0.1) + assert hits + assert not [p for p in tmp_path.glob("*.wav") if p.name.endswith("MHz.wav")] + assert all(not h.combined_path for h in hits) + + +def test_a_complete_band_demodulates_each_segment_correctly(tmp_path): + """One sweep of 2 m must handle its CW, SSB and FM segments each properly. + + That is the whole point of a complete-band entry: the band is not one + mode, so scanning it as one would demodulate two thirds of it wrongly. + """ + tx = [V(144_050_000, "cw", 0.35, 500, "beacon", wpm=18, + message="VVV DE W1AW"), + V(144_200_000, "usb", 0.35, 3_000, "ssb"), + V(146_520_000, "nfm", 0.40, 12_500, "fm", ctcss=100.0)] + s, hits = run_scan(tmp_path, tx, "2m-complete", record_seconds=6.0, + hang_seconds=2.0, max_cycles=2, revisit_seconds=0.2) + def segment(freq): + if freq < 144.1e6: + return "cw" + return "ssb" if freq < 144.3e6 else "fm" + + seen = {segment(h.frequency): (h.mode, h.category) for h in hits} + assert seen.get("cw") == ("cw", "cw"), seen + assert seen.get("ssb") == ("usb", "voice"), seen + assert seen.get("fm") == ("nfm", "voice"), seen + + +def test_frequencies_are_padded_so_names_sort_by_frequency(tmp_path): + """Unpadded, a directory listing puts 1090 MHz before 146 MHz.""" + from bandsaunter.recorder import build_stem, safe_freq_name + import datetime + assert safe_freq_name(146_520_000) == "0146.520000MHz" + assert safe_freq_name(1_090_000_000) == "1090.000000MHz" + assert safe_freq_name(1_800_000) == "0001.800000MHz" + + when = datetime.datetime(2026, 8, 21, 19, 35, 48).timestamp() + names = [build_stem(when, hz, "nfm") for hz in + (1_090_000_000, 146_520_000, 98_295_546, 1_800_000, 14_058_000)] + # sorted as text must equal sorted by frequency + by_text = sorted(names) + by_freq = [build_stem(when, hz, "nfm") for hz in + sorted((1_090_000_000, 146_520_000, 98_295_546, 1_800_000, + 14_058_000))] + assert by_text == by_freq + + +def test_combined_files_are_padded_too(tmp_path): + from bandsaunter.recorder import FrequencyLog + log = FrequencyLog(tmp_path, announce=False) + log.add(1_090_000_000.0, np.full(1600, 0.2, dtype=np.float32), 16000) + log.add(146_520_000.0, np.full(1600, 0.2, dtype=np.float32), 16000) + names = sorted(p.name for p in tmp_path.glob("*.wav")) + assert names == ["0146.520000MHz.wav", "1090.000000MHz.wav"] diff --git a/tests/test_settings.py b/tests/test_settings.py new file mode 100644 index 0000000..cef11d2 --- /dev/null +++ b/tests/test_settings.py @@ -0,0 +1,288 @@ +"""The settings table, and the two front ends generated from it.""" +import argparse + +import pytest + +from bandsaunter import settings as st +from bandsaunter.cli import build_parser +from bandsaunter.config import ScanConfig + + +# --------------------------------------------------------------------------- +# Parity: every setting reachable from both the command line and the menu. +# --------------------------------------------------------------------------- + +def test_every_config_field_has_a_setting(): + """A field with no entry would be invisible in the in-app menu.""" + missing, extra = st.coverage() + assert not missing, f"config fields with no setting: {missing}" + assert not extra, f"settings with no config field: {extra}" + + +def test_every_setting_has_a_command_line_flag(): + """Anything settable in the app must also be settable as an argument.""" + without = [s.key for s in st.SETTINGS if not s.flags and not s.off_flags] + assert not without, f"settings with no flag: {without}" + + +def test_every_flag_reaches_the_parser(): + parser = build_parser() + args = parser.parse_args(["scan"]) + for s in st.SETTINGS: + assert hasattr(args, s.dest), f"{s.key} never reached argparse" + assert getattr(args, s.dest) is None, \ + f"{s.key} defaults to a value, so it would override saved settings" + + +def test_flags_are_unique(): + seen = {} + for s in st.SETTINGS: + for flag in s.flags + s.off_flags: + assert flag not in seen, f"{flag} used by {seen.get(flag)} and {s.key}" + seen[flag] = s.key + + +def test_every_setting_is_documented(): + for s in st.SETTINGS: + assert s.help, s.key + assert not s.help.endswith("."), f"{s.key} help should not end in a full stop" + assert s.label, s.key + assert s.group in st.GROUPS, s.key + if s.kind in ("choice", "accept_list"): + assert s.choices, f"{s.key} is a choice with no choices" + + +# --------------------------------------------------------------------------- +# Values +# --------------------------------------------------------------------------- + +def test_defaults_round_trip_through_parse_and_format(): + """Whatever the menu shows must be something the menu accepts back.""" + cfg = ScanConfig() + for s in st.SETTINGS: + value = getattr(cfg, s.key) + shown = st.format_value(s, value) + if s.key in ("record_seconds", "max_record_seconds", + "max_runtime_seconds", "max_cycles") and not value: + continue # shown as "no limit", typed back as 0 + if s.kind == "freq_list" and not value: + continue # shown as "(none)" + back = st.parse_value(s, shown) + assert back == value, f"{s.key}: {value!r} -> {shown!r} -> {back!r}" + + +@pytest.mark.parametrize("key,text,expected", [ + ("hang_seconds", "5", 5.0), + ("record_seconds", "0", 0.0), + ("gain", "auto", "auto"), + ("gain", "28.0", 28.0), + ("direct_sampling", "auto", "auto"), + ("direct_sampling", "2", 2), + ("detector", "PEAK", "peak"), + ("save_iq", "yes", True), + ("save_iq", "n", False), + ("accept", "voice, cw", ["voice", "cw"]), + ("accept", "voice cw digital", ["voice", "cw", "digital"]), + ("lockout", "162.55M, 146.52M", [162_550_000.0, 146_520_000.0]), + ("detector_bias_db", "", None), + ("detector_bias_db", "6", 6.0), + ("sample_rate", "2048000", 2_048_000), + ("threshold_db", "12", 12.0), +]) +def test_parse(key, text, expected): + assert st.parse_value(st.by_key(key), text) == expected + + +@pytest.mark.parametrize("key,text", [ + ("hang_seconds", "-1"), # below the minimum + ("hang_seconds", "banana"), + ("detector", "sideways"), + ("accept", "voice,banana"), + ("min_voice_score", "1.5"), # above the maximum + ("usable_fraction", "0.01"), + ("save_iq", "maybe"), + ("lockout", "not-a-frequency"), + ("output_dir", ""), +]) +def test_bad_values_are_refused_with_a_reason(key, text): + with pytest.raises(st.SettingError) as exc: + st.parse_value(st.by_key(key), text) + assert str(exc.value), "an error with no explanation is no use" + + +def test_search_finds_settings_by_words_from_anywhere(): + assert [s.key for s in st.search("voice score")] == ["min_voice_score"] + assert st.search("hang")[0].key == "hang_seconds" + assert st.search("--hang")[0].key == "hang_seconds" + assert st.search("")== [] + assert st.search("zzzznothing") == [] + + +def test_name_matches_rank_above_description_matches(): + hits = [s.key for s in st.search("record")] + assert hits.index("record_seconds") < hits.index("audio_rate") + + +# --------------------------------------------------------------------------- +# Applying the command line +# --------------------------------------------------------------------------- + +def test_apply_args_sets_only_what_was_given(): + parser = build_parser() + args = parser.parse_args(["scan", "--hang", "6", "--record", "0"]) + cfg = ScanConfig() + changed = st.apply_args(cfg, args) + assert cfg.hang_seconds == 6.0 + assert cfg.record_seconds == 0.0 + assert set(changed) == {"hang_seconds", "record_seconds"} + assert cfg.threshold_db == ScanConfig().threshold_db + + +def test_boolean_flags_work_both_ways(): + parser = build_parser() + cfg = ScanConfig() + st.apply_args(cfg, parser.parse_args(["scan", "--no-audio", "--iq"])) + assert cfg.save_audio is False and cfg.save_iq is True + cfg = ScanConfig() + st.apply_args(cfg, parser.parse_args(["scan", "--keep-everything"])) + assert cfg.require_signal is False + + +def test_repeated_lockout_flags_accumulate(): + parser = build_parser() + args = parser.parse_args(["scan", "--lockout", "162.55M", + "--lockout", "146.52M"]) + cfg = ScanConfig() + st.apply_args(cfg, args) + assert cfg.lockout == [162_550_000.0, 146_520_000.0] + + +def test_help_text_renders_for_every_entry(): + """--help must format without an error from any entry in the table.""" + top = build_parser().format_help() + assert "scan" in top and "config" in top + scan = build_parser().parse_known_args(["scan"]) + assert scan is not None + # The scan sub-parser has to render too, since that is where the table goes. + sub = [a for a in build_parser()._actions + if isinstance(a, argparse._SubParsersAction)][0] + text = sub.choices["scan"].format_help() + for setting in st.SETTINGS: + for flag in setting.flags + setting.off_flags: + assert flag in text, f"{flag} missing from --help" + + +# --------------------------------------------------------------------------- +# The saved settings file +# --------------------------------------------------------------------------- + +def test_saved_settings_round_trip(tmp_path): + from bandsaunter.config import load_default, save_default + from bandsaunter.ranges import parse_range_list + cfg = ScanConfig(ranges=parse_range_list("144M-148M, gmrs")) + cfg.hang_seconds = 6.0 + cfg.record_seconds = 0.0 + cfg.accept = ["voice", "cw"] + cfg.lockout = [162_550_000.0] + cfg.gain = 28.0 + save_default(cfg, tmp_path) + + back, path = load_default(tmp_path) + assert path == tmp_path / "config.yaml" + assert back.hang_seconds == 6.0 + assert back.record_seconds == 0.0 + assert back.accept == ["voice", "cw"] + assert back.lockout == [162_550_000.0] + assert back.gain == 28.0 + assert [r.label for r in back.ranges] == [r.label for r in cfg.ranges] + + +def test_every_setting_survives_a_save_and_load(tmp_path): + """No setting may be lost or mangled by the config file.""" + from bandsaunter.config import load_default, save_default + cfg = ScanConfig() + marks = {} + for s in st.SETTINGS: + current = getattr(cfg, s.key) + if s.kind == "bool": + value = not current + elif s.kind == "choice": + value = [c for c in s.choices if c != current][0] + elif s.kind == "accept_list": + value = ["cw"] + elif s.kind == "freq_list": + value = [146_520_000.0] + elif s.kind in ("text", "path"): + value = "somewhere" + elif s.kind == "gain": + value = 22.9 + elif s.kind == "direct": + value = 2 + elif s.kind == "opt_float": + value = 7.0 + elif s.kind == "int": + value = int(max(s.minimum or 1, 3)) + else: + lo = s.minimum if s.minimum is not None else 1.0 + hi = s.maximum if s.maximum is not None else lo + 5.0 + value = min(hi, lo + 0.5) + setattr(cfg, s.key, value) + marks[s.key] = value + save_default(cfg, tmp_path) + back, _ = load_default(tmp_path) + for key, value in marks.items(): + assert getattr(back, key) == value, key + + +def test_a_corrupt_config_file_does_not_crash(tmp_path): + from bandsaunter.config import load_default + (tmp_path / "config.yaml").write_text("this: [is: not: valid: yaml") + cfg, path = load_default(tmp_path) + assert cfg.hang_seconds == ScanConfig().hang_seconds + assert path is None + + +def test_unknown_keys_in_a_config_file_are_ignored(tmp_path): + """An old file from a newer or older version must still load.""" + from bandsaunter.config import load_default + (tmp_path / "config.yaml").write_text( + "hang_seconds: 9\nsomething_removed: 5\n") + cfg, _ = load_default(tmp_path) + assert cfg.hang_seconds == 9.0 + + +# --------------------------------------------------------------------------- +# Layering: saved settings, then a profile, then flags +# --------------------------------------------------------------------------- + +def test_flags_override_saved_settings(tmp_path, monkeypatch): + from bandsaunter import cli + from bandsaunter.config import save_default + saved = ScanConfig() + saved.hang_seconds = 6.0 + saved.threshold_db = 12.0 + save_default(saved, tmp_path) + monkeypatch.setattr("bandsaunter.cli.load_default", + lambda: __import__("bandsaunter.config", fromlist=["x"]) + .load_default(tmp_path)) + + args = cli.build_parser().parse_args(["scan", "-b", "2m", "--hang", "1.5"]) + cfg, source = cli._build_config(args) + assert cfg.hang_seconds == 1.5, "the flag must win" + assert cfg.threshold_db == 12.0, "the saved value must survive" + assert source is not None + + +def test_no_config_ignores_the_saved_file(tmp_path, monkeypatch): + from bandsaunter import cli + from bandsaunter.config import save_default + saved = ScanConfig() + saved.hang_seconds = 6.0 + save_default(saved, tmp_path) + monkeypatch.setattr("bandsaunter.cli.load_default", + lambda: __import__("bandsaunter.config", fromlist=["x"]) + .load_default(tmp_path)) + args = cli.build_parser().parse_args(["scan", "-b", "2m", "--no-config"]) + cfg, source = cli._build_config(args) + assert cfg.hang_seconds == ScanConfig().hang_seconds + assert source is None diff --git a/tests/test_transcribe.py b/tests/test_transcribe.py new file mode 100644 index 0000000..8edbb2f --- /dev/null +++ b/tests/test_transcribe.py @@ -0,0 +1,378 @@ +"""Speech to text: the engine plumbing, and how captures reach it.""" +import time +from datetime import datetime +from pathlib import Path + +import numpy as np +import pytest + +from bandsaunter import transcribe as tr +from bandsaunter.config import ScanConfig +from bandsaunter.ranges import parse_range_list +from bandsaunter.scanner import Scanner, ScannerCallbacks +from bandsaunter.simulator import SimulatedDevice, VirtualTransmitter as V + + +@pytest.fixture +def fake_engine(monkeypatch): + """A recogniser that reports what it was given, so the plumbing is testable + on a machine with none installed.""" + seen = [] + + def engine(audio, rate, model, language): + seen.append({"samples": audio.size, "rate": rate, "model": model, + "language": language}) + return tr.Transcript(text="this is the transcribed text", + engine="fake", language=language or "en") + + monkeypatch.setitem(tr._DISPATCH, "fake", engine) + monkeypatch.setattr(tr, "ENGINES", ("fake",) + tr.ENGINES) + monkeypatch.setattr(tr, "_is_present", lambda name: name == "fake") + return seen + + +def _speech(seconds=3.0, rate=16000): + import sys + sys.path.insert(0, str(Path(__file__).parent)) + from speech import synth_speech + return synth_speech(seconds, rate, 120, 0) + + +# --------------------------------------------------------------------------- +# Engines +# --------------------------------------------------------------------------- + +def test_no_engine_reports_rather_than_failing(monkeypatch): + """Every capture failing noisily would be worse than saying so once.""" + monkeypatch.setattr(tr, "_is_present", lambda name: False) + assert tr.available_engine() is None + assert tr.transcribe(np.zeros(1000, np.float32), 16000) is None + + +def test_engine_listing_is_complete(): + listed = {name for name, _, _ in tr.describe_engines()} + assert listed == set(tr.ENGINES) + for _, _, how in tr.describe_engines(): + assert how, "every engine should say how to get it" + + +def test_a_failing_engine_is_reported_not_raised(monkeypatch): + def boom(audio, rate, model, language): + raise RuntimeError("model file is corrupt") + monkeypatch.setitem(tr._DISPATCH, "fake", boom) + monkeypatch.setattr(tr, "_is_present", lambda name: name == "fake") + result = tr.transcribe(np.zeros(16000, np.float32), 16000, engine="fake") + assert result is not None and not result + assert "corrupt" in result.note + + +def test_audio_is_resampled_to_what_the_engines_expect(fake_engine): + for rate in (8000, 16000, 32000, 48000): + tr.transcribe(np.zeros(int(rate * 2), np.float32), rate, engine="fake") + assert [s["samples"] for s in fake_engine] == [32000] * 4 + + +def test_the_model_and_language_reach_the_engine(fake_engine): + tr.transcribe(np.zeros(16000, np.float32), 16000, engine="fake", + model="small.en", language="fr") + assert fake_engine[-1]["model"] == "small.en" + assert fake_engine[-1]["language"] == "fr" + + +# --------------------------------------------------------------------------- +# The worker +# --------------------------------------------------------------------------- + +def _drain(worker, timeout=20.0): + worker.close(timeout=timeout) + + +def test_worker_writes_a_transcript_beside_the_recording(tmp_path, fake_engine): + worker = tr.TranscriptionWorker(engine="fake") + worker.start() + out = tmp_path / "0146.520000MHz--2026-08-21_12_00_00-nfm_transcription.txt" + worker.submit(_speech(), 16000, out, datetime(2026, 8, 21, 12, 0, 0), + 146.52e6) + _drain(worker) + assert out.exists() + assert "transcribed text" in out.read_text() + assert worker.written == 1 + + +def test_worker_appends_with_a_timestamp_when_combining(tmp_path, fake_engine): + worker = tr.TranscriptionWorker(engine="fake") + worker.start() + out = tmp_path / "0146.520000MHz_transcription.txt" + for minute in (0, 5, 9): + worker.submit(_speech(), 16000, out, + datetime(2026, 8, 21, 12, minute, 0), 146.52e6, + append=True) + _drain(worker) + lines = out.read_text().strip().split("\n") + assert len(lines) == 3 + assert lines[0].startswith("[2026-08-21 12:00:00] ") + assert lines[2].startswith("[2026-08-21 12:09:00] ") + + +def test_nothing_recognised_writes_no_file_at_all(tmp_path, monkeypatch): + """A directory of placeholder files is worse than no file.""" + monkeypatch.setitem(tr._DISPATCH, "fake", + lambda a, r, m, l: tr.Transcript(text="", engine="fake")) + monkeypatch.setattr(tr, "_is_present", lambda name: name == "fake") + worker = tr.TranscriptionWorker(engine="fake") + worker.start() + out = tmp_path / "quiet_transcription.txt" + worker.submit(np.zeros(16000, np.float32), 16000, out, datetime.now(), 1e6) + _drain(worker) + assert not out.exists() + assert not list(tmp_path.iterdir()) + assert worker.empty == 1 and worker.written == 0 + + +def test_whitespace_only_speech_writes_no_file(tmp_path, monkeypatch): + monkeypatch.setitem(tr._DISPATCH, "fake", + lambda a, r, m, l: tr.Transcript(text=" \n ", + engine="fake")) + monkeypatch.setattr(tr, "_is_present", lambda name: name == "fake") + worker = tr.TranscriptionWorker(engine="fake") + worker.start() + out = tmp_path / "blank_transcription.txt" + worker.submit(np.zeros(16000, np.float32), 16000, out, datetime.now(), 1e6) + _drain(worker) + assert not out.exists() and worker.empty == 1 + + +def test_an_empty_result_adds_no_line_when_combining(tmp_path, monkeypatch): + """Combined transcripts must not fill up with empty timestamps.""" + texts = iter(["something was said", "", "and something else"]) + monkeypatch.setitem(tr._DISPATCH, "fake", + lambda a, r, m, l: tr.Transcript(text=next(texts), + engine="fake")) + monkeypatch.setattr(tr, "_is_present", lambda name: name == "fake") + worker = tr.TranscriptionWorker(engine="fake") + worker.start() + out = tmp_path / "0146.520000MHz_transcription.txt" + for minute in (0, 5, 9): + worker.submit(np.zeros(16000, np.float32), 16000, out, + datetime(2026, 8, 21, 12, minute, 0), 1e6, append=True) + _drain(worker) + lines = out.read_text().strip().split("\n") + assert len(lines) == 2, lines + assert "12:00:00" in lines[0] and "12:09:00" in lines[1] + + +def test_worker_does_not_hold_up_the_caller(tmp_path, monkeypatch): + """Recognition takes seconds; a scan must not wait for it.""" + def slow(audio, rate, model, language): + time.sleep(1.0) + return tr.Transcript(text="eventually", engine="fake") + monkeypatch.setitem(tr._DISPATCH, "fake", slow) + monkeypatch.setattr(tr, "_is_present", lambda name: name == "fake") + worker = tr.TranscriptionWorker(engine="fake") + worker.start() + started = time.perf_counter() + for i in range(3): + worker.submit(_speech(), 16000, tmp_path / f"{i}.txt", datetime.now(), + 1e6) + assert time.perf_counter() - started < 0.5, "submitting blocked" + _drain(worker) + assert worker.written == 3 + + +def test_a_full_queue_is_counted_not_blocked(tmp_path, monkeypatch): + def slow(audio, rate, model, language): + time.sleep(0.4) + return tr.Transcript(text="x", engine="fake") + monkeypatch.setitem(tr._DISPATCH, "fake", slow) + monkeypatch.setattr(tr, "_is_present", lambda name: name == "fake") + worker = tr.TranscriptionWorker(engine="fake", max_queue=2) + worker.start() + accepted = sum(worker.submit(_speech(0.5), 16000, tmp_path / f"{i}.txt", + datetime.now(), 1e6) for i in range(12)) + assert accepted < 12 and worker.dropped > 0 + _drain(worker) + + +def test_empty_audio_is_not_submitted(tmp_path, fake_engine): + worker = tr.TranscriptionWorker(engine="fake") + worker.start() + assert not worker.submit(np.zeros(0, np.float32), 16000, + tmp_path / "x.txt", datetime.now(), 1e6) + _drain(worker) + assert worker.written == 0 + + +# --------------------------------------------------------------------------- +# Through a scan +# --------------------------------------------------------------------------- + +def _scan(tmp_path, transmitters, **over): + cfg = ScanConfig(ranges=parse_range_list("146.4M-146.6M"), + output_dir=str(tmp_path), record_seconds=4.0, + hang_seconds=1.0, threshold_db=12, dwell_seconds=0.05, + max_cycles=1, revisit_seconds=0.2, transcribe=True, + transcribe_engine="fake") + for k, v in over.items(): + setattr(cfg, k, v) + hits = [] + scanner = Scanner(cfg, device=SimulatedDevice(transmitters=transmitters).open(), + callbacks=ScannerCallbacks(on_record_end=hits.append)) + scanner.prepare() + scanner.run() + return scanner, [h for h in hits if h.kept] + + +def test_a_voice_capture_is_transcribed(tmp_path, fake_engine, monkeypatch): + monkeypatch.setattr("bandsaunter.scanner.available_engine", lambda: "fake") + scanner, hits = _scan(tmp_path, [V(146_520_000, "nfm", 0.4, 12_500, "v")]) + assert hits and hits[0].category == "voice" + written = list(tmp_path.glob("*_transcription.txt")) + assert written, "no transcript written" + assert written[0].stem.startswith(hits[0].filename) + assert "transcribed text" in written[0].read_text() + + +def test_morse_and_data_are_not_transcribed(tmp_path, fake_engine, monkeypatch): + """Running a recogniser over CW or a data burst wastes seconds per capture.""" + monkeypatch.setattr("bandsaunter.scanner.available_engine", lambda: "fake") + scanner, hits = _scan(tmp_path, [ + V(146_520_000, "fsk4", 0.4, 12_500, "data", baud=4800, deviation=1800)]) + assert hits and hits[0].category == "digital" + assert not list(tmp_path.glob("*_transcription.txt")) + + +def test_short_captures_are_skipped(tmp_path, fake_engine, monkeypatch): + monkeypatch.setattr("bandsaunter.scanner.available_engine", lambda: "fake") + scanner, hits = _scan(tmp_path, [V(146_520_000, "nfm", 0.4, 12_500, "v")], + transcribe_min_seconds=60.0) + assert hits + assert not list(tmp_path.glob("*_transcription.txt")) + + +def test_the_transcript_is_recorded_in_the_metadata(tmp_path, fake_engine, + monkeypatch): + import json + monkeypatch.setattr("bandsaunter.scanner.available_engine", lambda: "fake") + scanner, hits = _scan(tmp_path, [V(146_520_000, "nfm", 0.4, 12_500, "v")]) + meta = json.loads(Path(hits[0].meta_path).read_text()) + assert meta["hit"]["transcript_path"].endswith("_transcription.txt") + assert "transcribed text" in meta["hit"]["transcript"] + + +def test_the_metadata_never_names_a_transcript_that_was_not_written( + tmp_path, monkeypatch): + """Recording a path for a file that never appears would be a lie.""" + import json + monkeypatch.setitem(tr._DISPATCH, "fake", + lambda a, r, m, l: tr.Transcript(text="", engine="fake")) + monkeypatch.setattr(tr, "_is_present", lambda name: name == "fake") + monkeypatch.setattr("bandsaunter.scanner.available_engine", lambda: "fake") + scanner, hits = _scan(tmp_path, [V(146_520_000, "nfm", 0.4, 12_500, "v")]) + assert hits + assert not list(tmp_path.glob("*_transcription.txt")) + meta = json.loads(Path(hits[0].meta_path).read_text()) + assert not meta["hit"].get("transcript_path") + + +def test_combined_recordings_get_one_transcript_per_frequency(tmp_path, + fake_engine, + monkeypatch): + monkeypatch.setattr("bandsaunter.scanner.available_engine", lambda: "fake") + scanner, hits = _scan( + tmp_path, [V(146_520_000, "nfm", 0.4, 12_500, "v")], + combine_by_frequency=True, announce_timestamps=False, + record_seconds=2.0, max_cycles=3, revisit_seconds=0.05) + assert len(hits) >= 2 + written = list(tmp_path.glob("*_transcription.txt")) + assert len(written) == 1, written + lines = written[0].read_text().strip().split("\n") + assert len(lines) == len(hits) + assert all(line.startswith("[") for line in lines) + + +def test_transcription_off_writes_nothing(tmp_path, fake_engine, monkeypatch): + monkeypatch.setattr("bandsaunter.scanner.available_engine", lambda: "fake") + scanner, hits = _scan(tmp_path, [V(146_520_000, "nfm", 0.4, 12_500, "v")], + transcribe=False) + assert hits + assert not list(tmp_path.glob("*_transcription.txt")) + + +def test_missing_engine_says_so_once(tmp_path, monkeypatch): + monkeypatch.setattr("bandsaunter.scanner.available_engine", lambda: None) + notes = [] + cfg = ScanConfig(ranges=parse_range_list("146.4M-146.6M"), + output_dir=str(tmp_path), record_seconds=2.0, + threshold_db=12, max_cycles=1, transcribe=True) + scanner = Scanner(cfg, + device=SimulatedDevice(transmitters=[ + V(146_520_000, "nfm", 0.4, 12_500, "v")]).open(), + callbacks=ScannerCallbacks(on_status=notes.append)) + scanner.prepare() + scanner.run() + assert any("no speech recogniser" in n for n in notes), notes + assert scanner.transcriber is None + + +# --------------------------------------------------------------------------- +# Installed engines, when there are any +# --------------------------------------------------------------------------- + +INSTALLED = tr.available_engine() +needs_engine = pytest.mark.skipif(INSTALLED is None, + reason="no speech recogniser installed") + + +@needs_engine +def test_an_installed_engine_recognises_synthesised_speech(): + """Round trip: say something, then read it back off the audio.""" + from bandsaunter.announce import say + audio = say("one two three four five six seven eight nine", 16000) + result = tr.transcribe(audio, 16000, engine="auto", + model="base.en", language="en") + assert result is not None + assert not result.note, result.note + # Engines are free to write numbers as digits, and whisper does. + text = result.text.lower() + spelled = ("one", "two", "three", "four", "five", "six", "seven", + "eight", "nine") + hits = sum((word in text) or (str(i) in text) + for i, word in enumerate(spelled, 1)) + assert hits >= 4, f"only {hits} of nine numbers recognised: {result.text!r}" + + +@needs_engine +@pytest.mark.parametrize("rate", [8000, 16000, 32000]) +def test_an_installed_engine_copes_with_any_rate(rate): + from bandsaunter.announce import say + audio = say("testing one two three", rate) + result = tr.transcribe(audio, rate, engine="auto", model="base.en", + language="en") + assert result is not None and not result.note, result.note + + +@needs_engine +def test_silence_produces_no_transcript_rather_than_invention(): + result = tr.transcribe(np.zeros(16000 * 3, np.float32), 16000, + engine="auto", model="base.en", language="en") + assert result is not None + assert not result.text.strip(), f"invented {result.text!r} from silence" + + +@pytest.mark.skipif(not tr._is_present("vosk"), reason="vosk not installed") +def test_vosk_falls_back_when_given_a_whisper_model_name(): + """The model setting is shared with whisper, whose names are not paths.""" + from bandsaunter.announce import say + result = tr.transcribe(say("one two three", 16000), 16000, engine="vosk", + model="base.en", language="en") + assert result is not None + assert not result.note, result.note + + +@pytest.mark.skipif(not tr._is_present("vosk"), reason="vosk not installed") +def test_vosk_accepts_the_plain_language_code(): + """Vosk names its models by region and rejects a bare "en".""" + from bandsaunter.announce import say + result = tr.transcribe(say("one two three", 16000), 16000, engine="vosk", + language="en") + assert result is not None and not result.note, result.note diff --git a/tests/test_tui.py b/tests/test_tui.py new file mode 100644 index 0000000..5e81c73 --- /dev/null +++ b/tests/test_tui.py @@ -0,0 +1,206 @@ +"""The in-application menus, driven by scripted answers.""" +import pytest +from rich.console import Console + +from bandsaunter import settings as st, tui +from bandsaunter.config import ScanConfig + + +@pytest.fixture +def console(): + return Console(width=100, file=open("/dev/null", "w"), force_terminal=False) + + +def drive(monkeypatch, answers): + """Feed the menus a fixed list of answers, then stop.""" + script = list(answers) + + def fake_ask(console, prompt, default=""): + if not script: + raise _Done() + return script.pop(0) + + monkeypatch.setattr(tui, "_ask", fake_ask) + monkeypatch.setattr(tui.Confirm, "ask", lambda *a, **k: True) + return script + + +class _Done(Exception): + """Raised to break out when the script runs out.""" + + +def run(monkeypatch, console, func, answers, *args): + drive(monkeypatch, answers) + try: + return func(console, *args) + except _Done: + return None + + +def test_settings_menu_edits_a_value(monkeypatch, console): + cfg = ScanConfig() + run(monkeypatch, console, tui.settings_menu, ["1", "2", "6", "", ""], cfg) + assert cfg.hang_seconds == 6.0 + + +def test_settings_menu_finds_a_setting_by_search(monkeypatch, console): + cfg = ScanConfig() + run(monkeypatch, console, tui.settings_menu, ["voice score", "0.3", ""], cfg) + assert cfg.min_voice_score == 0.3 + + +def test_a_bad_value_is_refused_and_the_old_one_kept(monkeypatch, console): + cfg = ScanConfig() + before = cfg.hang_seconds + run(monkeypatch, console, tui.settings_menu, + ["1", "2", "banana", "", "", ""], cfg) + assert cfg.hang_seconds == before + + +def test_a_value_failing_validation_is_rolled_back(monkeypatch, console): + """min_record longer than record would mean nothing is ever kept.""" + cfg = ScanConfig() + cfg.record_seconds = 5.0 + run(monkeypatch, console, tui.settings_menu, + ["1", "4", "9", "", "", ""], cfg) + assert cfg.min_record_seconds != 9.0 + assert not [e for e in cfg.validate() if "ranges" not in e] + + +def test_every_group_can_be_opened_and_left(monkeypatch, console): + cfg = ScanConfig() + for i in range(1, len(st.GROUPS) + 1): + run(monkeypatch, console, tui.settings_menu, [str(i), "", ""], cfg) + + +def test_every_setting_renders_its_help(console): + """Built-in help must exist and render for every single setting.""" + cfg = ScanConfig() + for s in st.SETTINGS: + tui.setting_help(console, s, cfg) + + +def test_reset_a_group_restores_defaults(monkeypatch, console): + cfg = ScanConfig() + cfg.hang_seconds = 99.0 + cfg.record_seconds = 99.0 + run(monkeypatch, console, tui.settings_menu, ["1", "d", "", ""], cfg) + assert cfg.hang_seconds == ScanConfig().hang_seconds + assert cfg.record_seconds == ScanConfig().record_seconds + + +def test_help_screen_shows_every_topic(monkeypatch, console): + for topic in tui._TOPICS: + run(monkeypatch, console, tui.help_screen, [topic, ""]) + + +def test_help_screen_looks_up_settings(monkeypatch, console): + run(monkeypatch, console, tui.help_screen, ["hang", ""]) + + +def test_ranges_can_be_added_from_the_band_plan(monkeypatch, console): + cfg = ScanConfig() + run(monkeypatch, console, tui.choose_presets, ["gmrs", "1", ""], cfg) + assert cfg.ranges, "no range was added" + + +def test_ranges_can_be_typed_in(monkeypatch, console): + cfg = ScanConfig() + run(monkeypatch, console, tui.add_manual_ranges, + ["144M", "148M", "nfm", ""], cfg) + assert len(cfg.ranges) == 1 + assert cfg.ranges[0].start == 144e6 and cfg.ranges[0].stop == 148e6 + assert cfg.ranges[0].mode == "nfm" + + +def test_a_bad_frequency_is_refused(monkeypatch, console): + cfg = ScanConfig() + run(monkeypatch, console, tui.add_manual_ranges, ["banana", ""], cfg) + assert not cfg.ranges + + +def test_main_menu_can_start_a_scan(monkeypatch, console): + cfg = ScanConfig() + from bandsaunter.ranges import parse_range_list + cfg.ranges = parse_range_list("144M-148M") + out = run(monkeypatch, console, tui.run_tui, ["s"], cfg) + assert out is cfg + + +def test_main_menu_refuses_to_start_without_ranges(monkeypatch, console): + cfg = ScanConfig() + out = run(monkeypatch, console, tui.run_tui, ["s", "q"], cfg) + assert out is None, "started a scan with nothing to scan" + + +def test_main_menu_quits(monkeypatch, console): + assert run(monkeypatch, console, tui.run_tui, ["q"], ScanConfig()) is None + + +def test_closed_input_unwinds_instead_of_looping(monkeypatch, console): + """With no input left, the menus must give up rather than spin forever.""" + def eof(*a, **k): + raise EOFError + monkeypatch.setattr(tui.Prompt, "ask", eof) + assert tui.run_tui(console, ScanConfig()) is None + with pytest.raises(tui.TUIAbort): + tui.settings_menu(console, ScanConfig()) + + +def test_interrupt_unwinds_too(monkeypatch, console): + def interrupt(*a, **k): + raise KeyboardInterrupt + monkeypatch.setattr(tui.Prompt, "ask", interrupt) + assert tui.run_tui(console, ScanConfig()) is None + + +# --------------------------------------------------------------------------- +# First run +# --------------------------------------------------------------------------- + +def test_first_run_is_detected_and_then_not(tmp_path): + from bandsaunter.config import ScanConfig, is_first_run, save_default + assert is_first_run(tmp_path) + save_default(ScanConfig(), tmp_path) + assert not is_first_run(tmp_path) + + +def test_first_run_asks_where_to_save_and_remembers(monkeypatch, console, + tmp_path): + from bandsaunter import tui + from bandsaunter.config import load_default + wanted = tmp_path / "my recordings" + monkeypatch.setattr(tui, "_ask", lambda c, p, d="": str(wanted)) + monkeypatch.setattr(tui, "save_default", + lambda cfg: __import__("bandsaunter.config", + fromlist=["x"]).save_default( + cfg, tmp_path)) + cfg = ScanConfig() + assert tui.first_run_setup(console, cfg) + assert cfg.output_dir == str(wanted) + assert wanted.is_dir(), "the directory should be created" + back, _ = load_default(tmp_path) + assert back.output_dir == str(wanted) + + +def test_first_run_accepts_the_suggested_default(monkeypatch, console, + tmp_path): + from bandsaunter import tui + monkeypatch.setattr(tui, "_ask", lambda c, p, d="": "") + monkeypatch.setattr(tui, "save_default", lambda cfg: tmp_path / "x.yaml") + monkeypatch.setattr(tui, "DEFAULT_OUTPUT_DIR", str(tmp_path / "default")) + cfg = ScanConfig() + tui.first_run_setup(console, cfg) + assert cfg.output_dir == str(tmp_path / "default") + + +def test_first_run_rejects_a_directory_it_cannot_write(monkeypatch, console, + tmp_path): + """Better to say so now than to fail on the first recording.""" + from bandsaunter import tui + answers = iter(["/proc/nonsense/cannot-create", str(tmp_path / "ok")]) + monkeypatch.setattr(tui, "_ask", lambda c, p, d="": next(answers)) + monkeypatch.setattr(tui, "save_default", lambda cfg: tmp_path / "x.yaml") + cfg = ScanConfig() + tui.first_run_setup(console, cfg) + assert cfg.output_dir == str(tmp_path / "ok") diff --git a/tests/test_ui.py b/tests/test_ui.py new file mode 100644 index 0000000..3a2b5f6 --- /dev/null +++ b/tests/test_ui.py @@ -0,0 +1,147 @@ +"""The live display, and keeping the driver from writing over it.""" +import os +import subprocess +import sys +import tempfile +import time + +import numpy as np +import pytest +from rich.console import Console + +from bandsaunter.config import ScanConfig +from bandsaunter.ranges import parse_range_list +from bandsaunter.recorder import HitRecord +from bandsaunter.scanner import Scanner +from bandsaunter.simulator import SimulatedDevice +from bandsaunter.ui import ScanDisplay, _sparkline + + +def _display(height: int, hits: int, recording: bool) -> ScanDisplay: + console = Console(width=100, height=height, + file=open(os.devnull, "w")) + cfg = ScanConfig(ranges=parse_range_list("144M-148M"), + output_dir=tempfile.mkdtemp()) + scanner = Scanner(cfg, device=SimulatedDevice().open()) + scanner.prepare() + display = ScanDisplay(scanner, console=console) + display.attach() + step = scanner.plan[0] + display.on_step(0, 6, step, np.full(1024, -70.0), + np.linspace(step.low, step.high, 1024)) + for i in range(hits): + display.hits.append(HitRecord( + frequency=146e6 + i * 1e5, started_at=time.time(), duration=5.0, + snr_db=20.0, classification="FM broadcast station (stereo)")) + display._rec.active = recording + display._rec.frequency = 146.52e6 + display._rec.mode = "nfm" + return display + + +def _rendered_height(display: ScanDisplay) -> int: + probe = Console(width=100, height=200, file=open(os.devnull, "w"), + record=True) + probe.print(display.render()) + return len(probe.export_text().rstrip("\n").split("\n")) + + +@pytest.mark.parametrize("height", [16, 20, 24, 30, 40, 60]) +@pytest.mark.parametrize("hits,recording", [(0, False), (3, False), + (12, False), (12, True)]) +def test_the_display_never_outgrows_the_terminal(height, hits, recording): + """A frame taller than the terminal cannot be redrawn in place. + + Every refresh then scrolls another copy into the scrollback, which is why + the header ends up on screen several times over. + """ + display = _display(height, hits, recording) + assert _rendered_height(display) <= height, \ + f"{hits} hits, rec={recording}: overflowed a {height}-line terminal" + + +def test_older_hits_are_dropped_not_the_layout(): + """When space runs short the list shortens; the panels stay.""" + display = _display(20, 12, True) + probe = Console(width=100, height=200, file=open(os.devnull, "w"), + record=True) + probe.print(display.render()) + text = probe.export_text() + assert "receiver" in text and "sweep" in text + assert "more above" in text, "no sign that hits were trimmed" + + +def test_a_tall_terminal_shows_every_hit(): + display = _display(60, 12, False) + probe = Console(width=100, height=200, file=open(os.devnull, "w"), + record=True) + probe.print(display.render()) + assert "more above" not in probe.export_text() + + +def test_sparkline_keeps_narrow_carriers_visible(): + values = np.full(200, -70.0) + values[100] = -20.0 + assert "█" in _sparkline(values, 60) + + +# --------------------------------------------------------------------------- +# The driver writes its own messages straight to file descriptor 2 +# --------------------------------------------------------------------------- + +def test_quiet_driver_suppresses_writes_to_fd_2(): + """librtlsdr prints from C, so Python-level redirection cannot catch it.""" + code = ( + "import os, sys; sys.path.insert(0, %r)\n" + "from bandsaunter.device import quiet_driver\n" + "with quiet_driver():\n" + " os.write(2, b'CHATTER\\n')\n" + "os.write(2, b'AFTERWARDS\\n')\n" + ) % os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + out = subprocess.run([sys.executable, "-c", code], capture_output=True, + text=True, timeout=30) + assert "CHATTER" not in out.stderr + assert "AFTERWARDS" in out.stderr, "stderr was not restored" + + +def test_quiet_driver_restores_on_an_exception(): + code = ( + "import os, sys; sys.path.insert(0, %r)\n" + "from bandsaunter.device import quiet_driver\n" + "try:\n" + " with quiet_driver():\n" + " raise ValueError('boom')\n" + "except ValueError:\n" + " pass\n" + "os.write(2, b'RESTORED\\n')\n" + ) % os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + out = subprocess.run([sys.executable, "-c", code], capture_output=True, + text=True, timeout=30) + assert "RESTORED" in out.stderr + + +def test_driver_messages_can_be_turned_back_on_for_debugging(): + code = ( + "import os, sys; sys.path.insert(0, %r)\n" + "from bandsaunter.device import quiet_driver\n" + "with quiet_driver():\n" + " os.write(2, b'CHATTER\\n')\n" + ) % os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + env = dict(os.environ, BANDSAUNTER_DRIVER_MESSAGES="1") + out = subprocess.run([sys.executable, "-c", code], capture_output=True, + text=True, timeout=30, env=env) + assert "CHATTER" in out.stderr + + +def test_python_errors_still_reach_stderr(): + """Silencing the driver must not swallow a traceback.""" + code = ( + "import sys; sys.path.insert(0, %r)\n" + "from bandsaunter.device import quiet_driver\n" + "with quiet_driver():\n" + " pass\n" + "raise RuntimeError('visible')\n" + ) % os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + out = subprocess.run([sys.executable, "-c", code], capture_output=True, + text=True, timeout=30) + assert "visible" in out.stderr