dustcouncil/bandsaunter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Listen to FT8: fifteen seconds of everybody at once

Browse source 
A new section, alongside the aircraft, the weather sensors and APRS.

FT8 is the odd one out among the things this program listens to, and the
reason is worth stating because it shapes everything below.  Every station on
the band transmits in the same quarter-minute slots, on the same dial
frequency, fifty hertz wide each, stacked across three kilohertz of audio.
One receiver parked on one frequency therefore hears the whole band's worth of
stations at once -- and hears most of them well below the noise, because half
of what is sent is error-correcting code.  That is the entire trick: a rate of
about one half buys a mode that decodes twenty-odd decibels under what an
operator can hear.  A receiver that took the loudest tone of each symbol and
hoped would decode almost nothing, which is why the tone detector reports how
confident it is bit by bit rather than what it thinks it heard.

Written from first principles except for two tables.  The checksum, the
belief propagation over the sparse graph, the Costas sync search, the
waterfall, the soft-bit metric, and the seventy-seven bits that hold two
callsigns and a grid square are all here.  The generator and the parity-check
matrix are not: they cannot be derived, being the code itself rather than
consequences of anything, so they are taken from ft8_lib under its MIT licence
with the attribution it asks for, and said so in the readme, the manual and
the file.  No decoding logic came with them.  That the two agree -- and they
are not derivable from one another, the generator's parity half running to
fifty-odd bits a row against the sparse matrix's six or seven -- is a test
rather than an assumption.

Tested against the air, not against itself.  Eleven off-air recordings with
published decodes: ninety-seven of a hundred and fifty messages, no false
decodes, timing within a hundredth of a second, frequency within a hertz,
signal reports within half a decibel on average.  The third not decoded are
the weakest in each slot; a mature decoder subtracts what it has decoded and
looks again in the remainder, and does ordered-statistics decoding where
belief propagation fails, and neither is built here.  What is here decodes
nothing that other receivers did not also hear, which is the property that
matters in a log.  Ten whole codewords lifted off the air are in the tests as
a permanent fixture, so the recordings can go missing and the regression
cannot.

Three things that looked like bugs and were not, and three that were.  The
half-second timing discrepancy was the convention: a transmission is 12.64
seconds in a slot of fifteen and everybody starts half a second in, so
lateness is reported against that.  Synthetic signals at known offsets proved
the clock self-consistent before anything was changed.  The signal reports
were twenty-one decibels optimistic because those recordings have a receiver
passband above three kilohertz, putting a whole-band median twelve to sixteen
decibels below the real noise floor -- so noise is now measured beside the
signal, and in the tone that was actually sent rather than the loudest of
eight, the largest of eight noisy numbers being well above their mean even
with no signal at all.  And the test transmitter was thirteen decibels
pessimistic, scaling its noise into a fifty-hertz reference instead of the
sampled bandwidth, which made the decoder look deaf when it was the test
signal that had been quietly attenuated.

The real bug the simulator caught was a one-block timestamp error: samples
were dated a block earlier than they were taken, which slid every slot slice a
second late and cut the first half-second -- three symbols, part of the
opening Costas array -- off every transmission on the band.  One decode a slot
became six.

Reachable both ways, as everything here is.  Twenty-one options, every one of
them a command-line flag and a line in the menu, both built from one table so
they cannot disagree -- and a test that says so, since an option in no group
would be settable from the command line and invisible in the menu.  The band
list says which channels a plain dongle can reach and which need an
upconverter, because almost all the activity is on shortwave and finding that
out by listening to silence for ten minutes is the wrong way to learn it.  The
default is two metres, which a plain dongle can hear.

--grid turns decodes into geography: every CQ says where it is, so each gets a
distance and a bearing and the furthest is named.  --adif writes the log in
the form every amateur logging program imports, marked as heard rather than
worked, because nothing here transmits and an ADIF that let a logging program
treat these as contacts would put claims into somebody's log that they cannot
make.

One bug shipped and found by being used rather than by being tested: the
line that opens the receiver called a function this program has never had.
Every test reached it through the simulator, which takes the other branch, so
the one line that matters to somebody with an aerial was the one line never
run.  There is now a check that every name these modules import actually
exists -- it names the missing one rather than failing somewhere downstream --
and two that say a receiver which cannot be opened is reported rather than
raised, and that nothing claims to be listening before there is one.  It had
been announcing the frequency first, so a dongle that would not open read as
listening that had gone wrong.

Ninety-six new tests.  Full suite 2781 passed.  Built as 2026-09-21_04.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016PsWPTweCT6pwxKngvVxcg
This commit is contained in:
The Dust Council 2026-09-21 16:09:57 -07:00
parent e203b3e581
commit 93120b80a6
14 changed files with 4247 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

105
README.md
View file

@ -2630,6 +2630,93 @@ dongles ship with; the stock telescopic one extended properly does well.
`--packets` shows each frame as it arrives, which is the thing to watch while `--packets` shows each frame as it arrives, which is the thing to watch while
moving an aerial about. moving an aerial about.
## FT8
Fifteen seconds of everybody at once.
```
bandsaunter ft8 --band 20m --grid IO91
```
Every station on the band transmits in the same quarter-minute slots, on the
same dial frequency, fifty hertz wide each, stacked across three kilohertz of
audio. So one receiver parked on one frequency hears the whole band's worth
of stations simultaneously — and hears most of them **well below the noise**.
```
FT8 20m — 14.074 MHz 12 slots 187 decodes 63 stations 15.6 a slot 3:00 from IO91
last slot — 16 decoded
snr dt hz message
-19 -0.5 309 G4CUS SP4FCA RRR
-6 0.1 528 VK3EVE SQ3MZM RR73
18 0.1 1110 9A9TT IK4LZH -10
6 0.0 2535 CQ IZ3XJM JN55
heard so far — 63 stations
station grid away n best hz last
IZ3XJM JN55 1180 km 4 6 2535 2s
IK4LZH JN54 1140 km 7 18 1110 2s
```
Half of what is transmitted is error-correcting code, and that is the whole
trick: it is what buys a mode that decodes twenty-odd decibels under what an
operator can hear. A receiver that took the loudest tone of each symbol and
hoped would decode almost nothing.
### What it needs
**The clock has to be right**, to a second or two. The slots are
quarter-minutes of UTC and every station on earth agrees about which one it
is. A receiver a second out still decodes; one a slot out hears every
transmission split across two captures and decodes none of them. This is the
one failure that looks exactly like a dead band, so the display and the
report both say so when nothing arrives.
**Almost all the activity is on shortwave**, which a plain dongle cannot
reach. The default here is therefore the two-metre channel at 144.174 MHz,
which it can. All thirteen channels are in the list and the shortwave ones
work perfectly well through an upconverter or a receiver in direct-sampling
mode — the menu says which is which rather than leaving you to find out by
listening to silence.
### What comes out
`--grid IO91` is what turns decodes into geography: every station calling CQ
says where it is, so with your own square filled in the report gives each one
a distance and a bearing, and names the furthest heard. On shortwave that is
the entire interest of the thing.
Three tables afterwards: **stations heard** with grid, distance, count and
best and worst report; **calling CQ**, which is who is available; and **who
was working whom**, which is the band as a social event rather than a list.
`--adif` writes the log again in the form every amateur logging program
imports — marked as *heard*, not worked. Nothing here transmits, so nothing
here is a contact, and an ADIF that let a logging program treat these as
worked would put claims into somebody's log they cannot make.
### How well it works
Checked against eleven off-air recordings with published decodes, which is
the only honest way to test a decoder — an encoder tested against its own
decoder agrees with it about anything they are both wrong about, a lesson
this project learned expensively on 433 MHz.
| | |
|---|---|
| messages decoded | 97 of 150 (65%), **no false decodes** |
| timing | +0.00 s, spread 0.04 |
| frequency | +0.1 Hz, spread 1.1 |
| signal report | −0.4 dB, spread 4.8 |
| weakest decoded | −24 dB on this program's own scale |
The 35% not decoded are the weakest signals in each slot. A mature decoder
subtracts each signal it decodes and looks again in what is left, and does
ordered-statistics decoding when belief propagation fails; neither is built
here. What *is* here decodes nothing that other receivers did not also hear,
which is the property that matters in a log.
## Meters on 900 MHz ## Meters on 900 MHz
A scan of 902–928 MHz that turns up a burst gets it named rather than reported A scan of 902–928 MHz that turns up a burst gets it named rather than reported
@ -3439,3 +3526,21 @@ by the Free Software Foundation. It is distributed in the hope that it will be
useful, but with no warranty whatsoever — not even the implied warranty of useful, but with no warranty whatsoever — not even the implied warranty of
merchantability or fitness for a particular purpose. The full text is in merchantability or fitness for a particular purpose. The full text is in
[LICENSE](LICENSE), and at <https://www.gnu.org/licenses/>. [LICENSE](LICENSE), and at <https://www.gnu.org/licenses/>.
### Borrowed material
One file is not original work. `bandsaunter/ft8tables.py` holds the two fixed
tables that *define* the FT8 error-correcting code — the generator and the
sparse parity-check matrix — taken from
[ft8_lib](https://github.com/kgoba/ft8_lib), MIT licensed, copyright © 2018
Kārlis Goba, which took them in turn from WSJT-X. They are reproduced under
the MIT terms, which permit it, and that file carries the attribution they
ask for.
They are there because they cannot be derived. Everything else about FT8 in
this program is worked out from first principles — the tones, the sync, the
parity arithmetic, the belief propagation, the way a callsign is squeezed
into twenty-eight bits — but those two tables are not derived from anything.
They are the code itself, chosen once by its designers and published, and a
receiver that guessed at them would be speaking a different protocol. No
decoding logic was taken.

2
bandsaunter/__init__.py
View file

@ -9,7 +9,7 @@ and transcribing speech.
# 2026-08-21_02 is the second build made on the 21st. The revision is padded # 2026-08-21_02 is the second build made on the 21st. The revision is padded
# to two digits so versions sort as text. # to two digits so versions sort as text.
VERSION_DATE = "2026-09-21" VERSION_DATE = "2026-09-21"
VERSION_REVISION = 3 VERSION_REVISION = 4
__version__ = f"{VERSION_DATE}_{VERSION_REVISION:02d}" __version__ = f"{VERSION_DATE}_{VERSION_REVISION:02d}"

107
bandsaunter/cli.py
View file

@ -26,6 +26,7 @@ from .device import RtlSdrError, list_devices, set_driver_messages
from .librtlsdr import load_error from .librtlsdr import load_error
from . import settings as st from . import settings as st
from .ax25 import APRS_CHANNELS as _APRS_CHANNELS from .ax25 import APRS_CHANNELS as _APRS_CHANNELS
from .ft8 import BANDS as _FT8_BANDS
from .ranges import (RangeError, ScanRange, build_plan, parse_range_list) from .ranges import (RangeError, ScanRange, build_plan, parse_range_list)
from .scanner import Scanner, ScannerCallbacks from .scanner import Scanner, ScannerCallbacks
from .tui import TUIAbort, first_run_setup, run_tui, settings_menu from .tui import TUIAbort, first_run_setup, run_tui, settings_menu
@ -639,6 +640,74 @@ examples:
a.add_argument("--freq", type=float, default=0.0, a.add_argument("--freq", type=float, default=0.0,
help="centre frequency in Hz, for band-aware naming") help="centre frequency in Hz, for band-aware naming")
a.add_argument("--morse", action="store_true", help="force a CW decode") a.add_argument("--morse", action="store_true", help="force a CW decode")
# -- ft8 ------------------------------------------------------------------
f8 = sub.add_parser("ft8",
help="listen to FT8: fifteen-second slots, forty "
"stations at once, most of them under the noise")
f8.add_argument("--device", type=int, default=None, help="which receiver")
f8.add_argument("--gain", default=None, help="tuner gain in dB, or auto")
f8.add_argument("--rate", type=float, default=None,
help="sample rate in Hz; 96 kS/s is the least that holds "
"three kilohertz of audio")
f8.add_argument("--band", default=None,
choices=[key for key, _hz, _n in _FT8_BANDS],
help="which FT8 channel to listen on; sets the frequency")
f8.add_argument("--frequency", "--freq", dest="frequency", type=float,
default=None, metavar="HZ",
help="the dial frequency, for an upconverter or a "
"channel the band list does not have")
f8.add_argument("--simulate", dest="simulate", action="store_true",
default=None, help="invent a band instead of using a "
"receiver")
f8.add_argument("--no-simulate", dest="simulate", action="store_false",
help="use the receiver (the default)")
f8.add_argument("--seconds", type=float, default=None,
help="stop after this long (default: until interrupted)")
f8.add_argument("--slots", type=int, default=None, metavar="N",
help="stop after this many fifteen-second slots")
f8.add_argument("--log", dest="log", action="store_true", default=None,
help="write a log of every decode (the default)")
f8.add_argument("--no-log", dest="log", action="store_false",
help="do not write a log")
f8.add_argument("--decodes-seen", dest="decodes_seen",
action="store_true", default=None,
help="print a line per decode instead of a live table")
f8.add_argument("--no-decodes-seen", dest="decodes_seen",
action="store_false", help="show the live table")
f8.add_argument("--hold", type=float, default=None, metavar="SECONDS",
help="how long a station stays on the display")
f8.add_argument("--lowest", type=float, default=None, metavar="HZ",
help="the lowest audio frequency to search")
f8.add_argument("--highest", type=float, default=None, metavar="HZ",
help="the highest audio frequency to search")
f8.add_argument("--most", type=int, default=None, metavar="N",
help="how many candidate transmissions to try per slot")
f8.add_argument("--rounds", type=int, default=None, metavar="N",
help="how many error-correction passes before giving up")
f8.add_argument("--grid", default=None, metavar="SQUARE",
help="your own grid square, so distances can be worked "
"out (four characters, like IO91)")
f8.add_argument("--units", default=None, choices=("metric", "imperial"),
help="which units to show distances in")
f8.add_argument("--calls-only", dest="calls_only", action="store_true",
default=None, help="leave out free text and telemetry")
f8.add_argument("--no-calls-only", dest="calls_only",
action="store_false", help="show everything decoded")
f8.add_argument("--report", dest="report", action="store_true",
default=None, help="print the tables at the end")
f8.add_argument("--no-report", dest="report", action="store_false",
help="do not print the tables")
f8.add_argument("--csv", dest="csv", action="store_true", default=None,
help="also write a spreadsheet of every decode")
f8.add_argument("--no-csv", dest="csv", action="store_false",
help="do not write a spreadsheet")
f8.add_argument("--adif", dest="adif", action="store_true", default=None,
help="also write an ADIF of stations heard, for a "
"logging program")
f8.add_argument("--no-adif", dest="adif", action="store_false",
help="do not write an ADIF")
return p return p
@ -2151,6 +2220,7 @@ def main(argv=None) -> int:
"flights": cmd_flights, "weather": cmd_weather, "flights": cmd_flights, "weather": cmd_weather,
"readings": cmd_readings, "sensors": cmd_sensors, "readings": cmd_readings, "sensors": cmd_sensors,
"aprs": cmd_aprs, "packets": cmd_packets, "aprs": cmd_aprs, "packets": cmd_packets,
"ft8": cmd_ft8,
} }
try: try:
return handlers[args.command](args) return handlers[args.command](args)
@ -2161,5 +2231,42 @@ def main(argv=None) -> int:
return 0 return 0
def cmd_ft8(args) -> int:
"""Park on an FT8 channel and decode every slot.
Fifteen seconds of everybody at once: one dial frequency carries the
whole band's worth of stations, fifty hertz apart across three
kilohertz of audio, and most of them arrive below the noise.
"""
from . import ft8
cfg, _ = load_default()
options = ft8.load_options()
for flag, key in (("device", "device"), ("gain", "gain"),
("rate", "rate"), ("band", "band"),
("frequency", "frequency"), ("simulate", "simulate"),
("seconds", "seconds"), ("slots", "slots"),
("log", "log"), ("decodes_seen", "decodes_seen"),
("hold", "hold"), ("lowest", "lowest"),
("highest", "highest"), ("most", "most"),
("rounds", "rounds"), ("grid", "grid"),
("units", "units"), ("calls_only", "calls_only"),
("report", "report"), ("csv", "csv"),
("adif", "adif")):
value = getattr(args, flag, None)
if value is not None:
setattr(options, key, value)
# The band picks the frequency unless the frequency was given outright,
# which is the one order that lets both flags mean what they say.
if getattr(args, "band", None) and getattr(args, "frequency", None) is None:
ft8.use_band(options, options.band)
errs = options.validate()
if errs:
for e in errs:
console.print(f"[red]{e}[/red]")
return 2
ft8.listen(console, options, cfg.output_dir)
return 0
if __name__ == "__main__": if __name__ == "__main__":
sys.exit(main()) sys.exit(main())

919
bandsaunter/ft8.py Normal file
View file

@ -0,0 +1,919 @@
"""Listening to FT8, from either front end.
Fifteen seconds of everybody at once. Every station on the band transmits
in the same quarter-minute slots, on the same dial frequency, fifty hertz
wide each, stacked across three kilohertz of audio -- so one receiver
parked on one frequency hears the whole band's worth of stations at the
same time, and hears most of them well below the noise.
That is why this is a section rather than something the scanner stops on.
A sweep catches whatever was keyed up as it went past; FT8 is forty
stations transmitting simultaneously for twelve and a half seconds out of
every fifteen, and the interesting thing is all of them.
The band plan is mostly shortwave, which a plain receiver of this kind
cannot reach without help -- so the default here is the two-metre channel
at 144.174 MHz, which it can. The shortwave channels are all in the list
and work perfectly well through an upconverter or a receiver in direct
sampling mode; they are simply not what you get by not choosing.
Nothing here transmits. This listens, decodes and writes down.
"""
from __future__ import annotations
import math
import time
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from pathlib import Path
import numpy as np
import yaml
from . import ft8code as code
from . import ft8wave as wave
from .settings import Setting, format_value
__all__ = ["Ft8Options", "OPTIONS", "OPTION_GROUPS", "defaults", "in_group",
"by_key", "format_option", "describe", "summarise",
"Station", "Band", "Heard", "listen", "open_device", "open_log",
"pump", "finish", "report", "load_options", "save_options",
"options_path", "logs_in", "BANDS", "band_named", "band_text",
"use_band", "slot_start", "next_slot", "grid_at", "grid_away",
"SLOT", "audio_from"]
# ---------------------------------------------------------------------------
# Where FT8 lives
# ---------------------------------------------------------------------------
# name, hertz, what it needs. The dial frequency, in the usual convention:
# signals sit from about 200 to 3000 Hz above it.
BANDS: tuple[tuple[str, float, str], ...] = (
("160m", 1_840_000.0, "shortwave"),
("80m", 3_573_000.0, "shortwave"),
("60m", 5_357_000.0, "shortwave"),
("40m", 7_074_000.0, "shortwave"),
("30m", 10_136_000.0, "shortwave"),
("20m", 14_074_000.0, "shortwave"),
("17m", 18_100_000.0, "shortwave"),
("15m", 21_074_000.0, "shortwave"),
("12m", 24_915_000.0, "shortwave"),
("10m", 28_074_000.0, "shortwave"),
("6m", 50_313_000.0, "direct"),
("2m", 144_174_000.0, "direct"),
("70cm", 432_174_000.0, "direct"),
)
SLOT = code.SLOT_S # fifteen seconds, and everybody keeps to it
def band_named(name: str) -> float:
"""The dial frequency for a band, or the two-metre one."""
wanted = (name or "").strip().lower()
for key, hz, _needs in BANDS:
if wanted == key:
return hz
return 144_174_000.0
def band_needs(name: str) -> str:
wanted = (name or "").strip().lower()
for key, _hz, needs in BANDS:
if wanted == key:
return needs
return "direct"
def use_band(options: "Ft8Options", name: str) -> None:
"""Point the receiver at a band, frequency and all."""
options.band = (name or "").strip().lower()
options.frequency = band_named(options.band)
def band_text(options: "Ft8Options") -> str:
where = f"{options.frequency / 1e6:g} MHz"
needs = band_needs(options.band)
if needs == "shortwave":
return f"{options.band} — {where}, needs an upconverter"
return f"{options.band} — {where}"
# ---------------------------------------------------------------------------
# Slots
# ---------------------------------------------------------------------------
def slot_start(when: float | None = None) -> float:
"""The top of the fifteen-second slot that ``when`` falls in.
Against UTC, not against the local clock and not against when this
program happened to start: the whole protocol is built on every station
on earth agreeing about which quarter-minute it is. A receiver whose
clock is a second out still decodes -- the search covers a few seconds
either way -- but one that is a slot out hears every transmission
split across two captures and decodes none of them.
"""
when = time.time() if when is None else when
return math.floor(when / SLOT) * SLOT
def next_slot(when: float | None = None) -> float:
return slot_start(when) + SLOT
def slot_name(at: float) -> str:
"""The slot, as the six digits every FT8 program labels it with."""
return datetime.fromtimestamp(at, timezone.utc).strftime("%H%M%S")
# ---------------------------------------------------------------------------
# Grid squares
# ---------------------------------------------------------------------------
def grid_at(grid: str):
"""The middle of a Maidenhead square, as latitude and longitude.
Four characters is about seventy miles by a hundred, which is all the
position FT8 carries and quite enough to say which continent somebody
is on. Six characters is finer and turns up in free text rather than
in the position field.
"""
grid = (grid or "").strip().upper()
if len(grid) < 4 or not grid[:2].isalpha() or not grid[2:4].isdigit():
return None
if not ("A" <= grid[0] <= "R" and "A" <= grid[1] <= "R"):
return None
lon = (ord(grid[0]) - 65) * 20.0 + int(grid[2]) * 2.0
lat = (ord(grid[1]) - 65) * 10.0 + int(grid[3]) * 1.0
if len(grid) >= 6 and grid[4].isalpha() and grid[5].isalpha():
lon += (ord(grid[4]) - 65) * (2.0 / 24.0) + (1.0 / 24.0)
lat += (ord(grid[5]) - 65) * (1.0 / 24.0) + (0.5 / 24.0)
else:
lon += 1.0
lat += 0.5
return lat - 90.0, lon - 180.0
def grid_away(here: str, there: str):
"""How far apart two grid squares are, in kilometres, and on what bearing.
Great-circle, from the middles of the squares -- so it is good to the
size of a square and no better, which for FT8 is the honest answer and
is why nothing here prints a distance to the nearest kilometre.
"""
a, b = grid_at(here), grid_at(there)
if a is None or b is None:
return None
lat1, lon1 = math.radians(a[0]), math.radians(a[1])
lat2, lon2 = math.radians(b[0]), math.radians(b[1])
d = 2 * math.asin(math.sqrt(
math.sin((lat2 - lat1) / 2) ** 2
+ math.cos(lat1) * math.cos(lat2) * math.sin((lon2 - lon1) / 2) ** 2))
y = math.sin(lon2 - lon1) * math.cos(lat2)
x = (math.cos(lat1) * math.sin(lat2)
- math.sin(lat1) * math.cos(lat2) * math.cos(lon2 - lon1))
return 6371.0088 * d, (math.degrees(math.atan2(y, x)) + 360.0) % 360.0
# ---------------------------------------------------------------------------
# The options
# ---------------------------------------------------------------------------
@dataclass
class Ft8Options:
"""Everything the FT8 mode can be told, in one place."""
# -- receiver -------------------------------------------------------
device: int = 0
gain: str = "auto"
rate: float = 240_000.0
band: str = "2m"
frequency: float = 144_174_000.0
simulate: bool = False
# -- listening ------------------------------------------------------
seconds: float = 0.0
slots: int = 0 # or stop after this many slots
log: bool = True
decodes_seen: bool = False # a line per decode rather than a table
hold: float = 3600.0
# -- decoding -------------------------------------------------------
lowest: float = 200.0
highest: float = 3000.0
most: int = 300 # candidates examined per slot
rounds: int = 30 # how hard to try to repair one
# -- showing --------------------------------------------------------
grid: str = "" # where the aerial is, as a grid square
units: str = "metric"
calls_only: bool = False # leave out free text and telemetry
# -- afterwards -----------------------------------------------------
report: bool = True
csv: bool = False
adif: bool = False
@property
def imperial(self) -> bool:
return str(self.units).lower().startswith("imp")
def validate(self) -> list[str]:
out = []
if self.rate < 96_000:
out.append("the sample rate must be at least 96 kS/s to hold "
"three kilohertz of audio with room to filter it")
if self.seconds < 0 or self.slots < 0:
out.append("times cannot be negative")
if self.hold <= 0:
out.append("a station must stay on the display for some time")
if not 1e5 < self.frequency < 2e9:
out.append("that is not a frequency a receiver can be tuned to")
if self.highest <= self.lowest:
out.append("the top of the search must be above the bottom")
if self.highest - self.lowest < 100.0:
out.append("a search narrower than 100 Hz would miss almost "
"everything: one signal is 50 Hz wide and they are "
"spread across three kilohertz")
if self.most < 1:
out.append("at least one candidate has to be looked at")
if self.rounds < 1:
out.append("the error correction needs at least one pass")
if self.grid and grid_at(self.grid) is None:
out.append(f"{self.grid!r} is not a grid square — four "
"characters like IO91 or FN31")
return out
def to_dict(self) -> dict:
return asdict(self)
def defaults() -> Ft8Options:
return Ft8Options()
O = Setting
_BANDS = tuple(key for key, _hz, _needs in BANDS)
OPTIONS: tuple[Setting, ...] = (
# -- receiver -------------------------------------------------------
O("device", "Receiver", "Receiver", "int",
"which receiver to use, when more than one is plugged in",
"The index shown by `bandsaunter devices`. Zero unless you have "
"several dongles.",
minimum=0, flags=("--device",), example="0"),
O("gain", "Gain", "Receiver", "gain",
"tuner gain in dB, or automatic",
"FT8 is forty signals at once spanning fifty decibels of strength, "
"and what matters is that none of them clips rather than that any of "
"them is loud. Automatic copes. A fixed gain makes the signal "
"reports comparable between one evening and the next, which is the "
"whole point of them if you are judging an aerial or a band opening.",
flags=("--gain",), example="auto"),
O("rate", "Sample rate", "Receiver", "float",
"how fast to sample; 96 kS/s is the least that holds the channel",
"Only three kilohertz of it is used, so this is about having room to "
"filter cleanly rather than about bandwidth. The default costs "
"little and decimates to the twelve kilohertz the decoding wants by "
"a whole number.",
unit="Hz", minimum=96_000.0, flags=("--rate",), example="240000"),
O("band", "Band", "Receiver", "choice",
"which FT8 channel to listen on",
"Every band has one agreed dial frequency and everybody uses it. "
"Two metres is the default because a plain receiver of this kind can "
"reach it; the shortwave bands, which is where almost all the "
"activity is, need an upconverter or a receiver in direct sampling "
"mode. Choosing a band sets the frequency with it.",
choices=_BANDS, flags=("--band",), example="20m"),
O("frequency", "Dial frequency", "Receiver", "float",
"the dial frequency, if you want one the band list does not have",
"Signals sit from about two hundred to three thousand hertz above "
"this, which is the upper-sideband convention the whole mode is "
"built on. Setting a band sets this; setting this directly leaves "
"the band label alone, so use it for an upconverter's offset or a "
"channel that is not in the list.",
unit="Hz", minimum=1e5, maximum=2e9, flags=("--frequency", "--freq"),
example="14074000"),
O("simulate", "Simulate", "Receiver", "bool",
"invent a band instead of using a receiver",
"A made-up band of stations, on a real clock, for trying the display "
"and the report without an aerial. What it cannot tell you is "
"whether your receiver hears anything.",
flags=("--simulate",), off_flags=("--no-simulate",)),
# -- listening ------------------------------------------------------
O("seconds", "Listen for", "Listening", "float",
"how long to listen; zero means until stopped",
"Rounded up to a whole slot, because half a transmission decodes to "
"nothing at all.",
unit="s", minimum=0.0, flags=("--seconds",), example="600"),
O("slots", "Or this many slots", "Listening", "int",
"stop after this many fifteen-second slots; zero means no limit",
"The same thing as the time limit, counted the way the band counts "
"it. Whichever of the two is reached first stops the run.",
minimum=0, flags=("--slots",), example="40"),
O("log", "Write a log", "Listening", "bool",
"keep every decode in a file",
"One line per decode, with the slot, the frequency, how late it was "
"and how strongly it came in. The report afterwards is built from "
"this, and so is anything you want to do with it later.",
flags=("--log",), off_flags=("--no-log",)),
O("decodes_seen", "Print each decode", "Listening", "bool",
"a line per decode instead of a table that refreshes",
"What to use when the output is going into a pipe or a file. The "
"table is easier to watch; the lines are easier to grep.",
flags=("--decodes-seen",), off_flags=("--no-decodes-seen",)),
O("hold", "Keep on display for", "Listening", "float",
"how long a station stays on the table after its last decode",
"A station transmits every other slot at most, and many call once "
"and go away. Long enough that the table is a picture of the "
"evening rather than of the last thirty seconds.",
unit="s", minimum=1.0, flags=("--hold",), example="3600"),
# -- decoding -------------------------------------------------------
O("lowest", "Search from", "Decoding", "float",
"the lowest audio frequency to look for signals at",
"Below a couple of hundred hertz there is nothing but the "
"receiver's own rumble, and searching it costs time for nothing.",
unit="Hz", minimum=0.0, flags=("--lowest",), example="200"),
O("highest", "Search to", "Decoding", "float",
"the highest audio frequency to look for signals at",
"Three kilohertz is where a normal sideband receiver stops. Going "
"higher finds nothing unless your receiver is wider than that, and "
"costs time in proportion.",
unit="Hz", minimum=100.0, flags=("--highest",), example="3000"),
O("most", "Candidates per slot", "Decoding", "int",
"how many possible transmissions to try to decode each slot",
"The sync search ranks every place a transmission could be and this "
"is how far down the list to go. A busy shortwave band puts forty "
"real signals in a slot and several hundred plausible-looking "
"places; more than a few hundred has been measured to add nothing "
"but time.",
minimum=1, flags=("--most",), example="300"),
O("rounds", "Repair passes", "Decoding", "int",
"how many passes of error correction to make before giving up",
"The code either converges in twenty or so passes or it does not "
"converge at all -- raising this has been measured to find nothing "
"further, and it is here so that can be checked rather than taken "
"on trust.",
minimum=1, flags=("--rounds",), example="30"),
# -- showing --------------------------------------------------------
O("grid", "Aerial at", "Showing", "text",
"your own grid square, so distances can be worked out",
"Four characters, like IO91 or FN31. Every FT8 station that calls "
"CQ says where it is this way, so with your own square filled in "
"the report can say how far each of them is and on what bearing — "
"which on shortwave is the whole interest of the thing. Left blank "
"the decodes are still recorded; there is simply nothing to measure "
"them from.",
flags=("--grid",), example="IO91", metavar="SQUARE"),
O("units", "Show readings in", "Showing", "choice",
"metric or imperial, for the display and the export",
"Distances only. Signal reports are in decibels either way, that "
"being what the mode measures in and what gets sent back on the "
"air.",
choices=("metric", "imperial"), flags=("--units",), example="metric"),
O("calls_only", "Callsigns only", "Showing", "bool",
"leave out free text and telemetry",
"Most of what passes on FT8 is two callsigns and a report. The rest "
"is free text -- thirteen characters, no callsign structure, so "
"nothing can be said about who sent it -- and telemetry, which is "
"eighteen hex digits. Turn this on to see only the exchanges.",
flags=("--calls-only",), off_flags=("--no-calls-only",)),
# -- afterwards -----------------------------------------------------
O("report", "Report at the end", "Afterwards", "bool",
"print the tables when the listening stops",
"Who was heard, how far off, and what was exchanged.",
flags=("--report",), off_flags=("--no-report",)),
O("csv", "Also write a CSV", "Afterwards", "bool",
"a spreadsheet of every decode beside the log",
"The same rows as the log, in the form a spreadsheet opens without "
"being told anything.",
flags=("--csv",), off_flags=("--no-csv",)),
O("adif", "Also write an ADIF", "Afterwards", "bool",
"the log again, in the form logging programs read",
"ADIF is what every amateur logging program imports. What is "
"written is what was heard rather than a contact made -- nothing "
"here transmits, so nothing here is a QSO — so the records are "
"marked as received-only and are for feeding a propagation map or a "
"spotting tool rather than for claiming anything.",
flags=("--adif",), off_flags=("--no-adif",)),
)
OPTION_GROUPS = ("Receiver", "Listening", "Decoding", "Showing", "Afterwards")
def in_group(group: str) -> tuple[Setting, ...]:
return tuple(o for o in OPTIONS if o.group == group)
def by_key(key: str):
for option in OPTIONS:
if option.key == key:
return option
return None
# Options where nothing set means something, rather than nothing.
_ZERO_MEANS = {
"seconds": "until stopped",
"slots": "no limit",
"grid": "not set, so no distances",
}
def format_option(option: Setting, value) -> str:
"""Render an option the way the menu and the manual should show it."""
if not value and option.key in _ZERO_MEANS:
return _ZERO_MEANS[option.key]
return format_value(option, value)
def describe(options: Ft8Options) -> str:
"""The one-line summary the menu shows beside Listen."""
bits = [band_text(options)]
if options.seconds:
bits.append(f"{options.seconds:g} s")
elif options.slots:
bits.append(f"{options.slots} slots")
if options.grid:
bits.append(f"from {options.grid.upper()}")
if options.simulate:
bits.append("simulated")
return ", ".join(bits)
def summarise(options: Ft8Options) -> str:
return describe(options)
def options_path(directory=None) -> Path:
from .config import DEFAULT_CONFIG_DIR
return Path(directory or DEFAULT_CONFIG_DIR) / "ft8.yaml"
def load_options(directory=None) -> Ft8Options:
"""The saved options, or the defaults. A broken file is not an error."""
options = Ft8Options()
try:
body = yaml.safe_load(options_path(directory).read_text()) or {}
except (OSError, ValueError, yaml.YAMLError):
return options
if not isinstance(body, dict):
return options
known = set(options.__dict__)
for key, value in body.items():
if key in known and value is not None:
try:
setattr(options, key, type(getattr(options, key))(value))
except (TypeError, ValueError):
pass
return options
def save_options(options: Ft8Options, directory=None) -> Path:
path = options_path(directory)
path.parent.mkdir(parents=True, exist_ok=True)
with open(path, "w") as fh:
yaml.safe_dump(options.to_dict(), fh, sort_keys=False,
default_flow_style=False)
return path
def logs_in(directory) -> list[Path]:
try:
return sorted(Path(directory).glob("ft8-*.txt"))
except OSError:
return []
# ---------------------------------------------------------------------------
# Who was heard
# ---------------------------------------------------------------------------
@dataclass
class Station:
"""One callsign, and everything heard from it."""
call: str = ""
grid: str = ""
first: float = 0.0
last: float = 0.0
decodes: int = 0
best_snr: float = -99.0
worst_snr: float = 99.0
hertz: float = 0.0
calling: int = 0 # how many times it called CQ
worked: set = field(default_factory=set) # who it was talking to
said: list = field(default_factory=list) # the lines, newest last
def away(self, here: str):
"""How far off and on what bearing, if both squares are known."""
if not here or not self.grid:
return None
return grid_away(here, self.grid)
@dataclass
class Band:
"""Every station heard, and every exchange between them.
Kept apart the way the APRS section keeps messages apart from stations,
and for the same reason: a decode is about a moment and a station is
about an evening, and a table that tried to be both would be neither.
"""
stations: dict = field(default_factory=dict)
decodes: list = field(default_factory=list)
book: code.CallBook = field(default_factory=code.CallBook)
slots: int = 0
def add(self, found) -> None:
self.decodes.append(found)
# The caller is the second callsign in a standard exchange -- the
# first is who is being answered. A CQ has no first, so the one
# station named is the one transmitting.
sender = found.calls[1] if len(found.calls) > 1 else (
found.calls[0] if found.calls else "")
if not sender or sender == "<...>":
return
station = self.stations.get(sender)
if station is None:
station = Station(call=sender, first=found.at)
self.stations[sender] = station
station.last = found.at
station.decodes += 1
station.hertz = found.hertz
station.best_snr = max(station.best_snr, found.snr_db)
station.worst_snr = min(station.worst_snr, found.snr_db)
if found.grid:
station.grid = found.grid
if found.calling:
station.calling += 1
elif len(found.calls) > 1 and found.calls[0] != "<...>":
station.worked.add(found.calls[0])
station.said.append(found.text)
if len(station.said) > 32:
del station.said[0]
def all(self) -> list:
return list(self.stations.values())
@dataclass
class Heard:
"""What the front ends share: the running totals of one run."""
band: Band = field(default_factory=Band)
decodes: int = 0
slots: int = 0
started: float = 0.0
latest: list = field(default_factory=list)
# ---------------------------------------------------------------------------
# The receiver
# ---------------------------------------------------------------------------
AUDIO_RATE = 12_000 # what the decoding wants, and a whole division
def open_device(console, options: Ft8Options):
"""The receiver, or a made-up band, or None if neither can be had."""
from rich.panel import Panel
from rich.text import Text
from .device import RtlSdrDevice, RtlSdrError
if options.simulate:
console.print("[yellow]simulated: these stations are not there."
"[/yellow]")
from .ft8sim import SimulatedBand
return SimulatedBand(rate=options.rate, realtime=True)
try:
device = RtlSdrDevice(index=options.device,
sample_rate=int(options.rate),
gain=options.gain,
agc=options.gain == "auto",
# Which is what makes the shortwave bands
# reachable at all on a dongle that supports
# it: below 24 MHz the tuner is bypassed and
# the sampler is fed straight off the aerial.
direct_sampling="auto")
device.open()
except RtlSdrError as exc:
console.print(Panel(Text(str(exc)),
title="[red]cannot open the receiver",
border_style="red"))
return None
return device
def audio_from(samples, options: Ft8Options, demod=None):
"""Upper-sideband audio at twelve kilohertz, from the receiver's samples.
Upper sideband because that is the convention the whole mode is built
on: the dial frequency is the bottom edge and every signal sits above
it, which is why two stations three hundred hertz apart are two
different signals rather than the same one heard twice.
"""
from .demod import make_demodulator
if demod is None:
demod = make_demodulator("usb", sample_rate=int(options.rate),
bandwidth=3200.0, audio_rate=AUDIO_RATE)
audio, _iq = demod.step(samples)
return audio, demod
def pump(device, options: Ft8Options, heard: Heard, log, started: float,
on_slot=None, on_decode=None, stopping=None, clock=time.time) -> int:
"""Read the receiver, a slot at a time, until it stops or is told to.
The one loop both front ends are driven from, so that what is written
to the log cannot depend on which one you happened to be looking at.
Audio is collected continuously and cut on the slot boundaries rather
than being captured slot by slot. A receiver that started reading when
the slot began would miss however long it took to start reading, and
everything on the band begins in the first half-second.
"""
demod = None
audio = np.zeros(0, dtype=np.float32)
audio_at = 0.0 # wall time of audio[0]
want = next_slot(clock()) # the first whole slot we can get
total = 0
device.tune(options.frequency)
block = int(options.rate) # a second of samples at a time
while stopping is None or not stopping():
at = clock()
samples = device.read_samples(block)
if samples is None or samples.size == 0:
break
fresh, demod = audio_from(samples, options, demod)
if fresh.size:
if audio.size == 0:
# The clock was read before asking for the samples, and the
# read returns once they have arrived -- so they cover the
# block that *starts* here. Dating them a block earlier
# instead slides every slice a second late, which cuts the
# first half-second off every transmission: three symbols,
# part of the opening Costas array, and most of the
# decodes with it.
audio_at = at
audio = np.concatenate([audio, fresh])
# Every whole slot now sitting in the buffer.
while audio.size and audio_at <= want and \
audio_at + audio.size / AUDIO_RATE >= want + SLOT:
first = int(round((want - audio_at) * AUDIO_RATE))
piece = audio[first:first + int(round(SLOT * AUDIO_RATE))]
found = wave.listen_to(piece, AUDIO_RATE, book=heard.band.book,
most=options.most, rounds=options.rounds,
at=want)
found = [d for d in found
if options.lowest <= d.hertz <= options.highest]
if options.calls_only:
found = [d for d in found if d.kind in ("standard",
"non-standard")]
heard.slots += 1
heard.band.slots += 1
heard.latest = found
for d in found:
heard.band.add(d)
heard.decodes += 1
total += 1
if log is not None:
log.append(d)
if on_decode is not None:
on_decode(d)
if on_slot is not None:
on_slot(want, found)
want += SLOT
# Keep a little before the next slot: a transmission can start
# before the slot it belongs to.
keep = int(round(max(0.0, want - 3.0 - audio_at) * AUDIO_RATE))
if keep > 0:
audio = audio[keep:]
audio_at += keep / AUDIO_RATE
if options.seconds and clock() - started >= options.seconds:
break
if options.slots and heard.slots >= options.slots:
break
return total
def open_log(console, options: Ft8Options, output_dir, started: float):
from . import ft8log
if not options.log:
return None
log = ft8log.open_log(output_dir, started, options.frequency,
options.band)
if log is None:
console.print("[yellow]could not open a log; carrying on without "
"one[/yellow]")
return log
def listen(console, options: Ft8Options, output_dir: str,
heard: Heard | None = None, stopping=None) -> int:
"""Listen on one channel and write down everything decoded."""
from .ui import Ft8Display
started = time.time()
heard = heard if heard is not None else Heard()
heard.started = started
# The receiver first. Announcing that it is listening and then failing
# to open a dongle reads as though the listening went wrong, when what
# went wrong happened before any of it started.
device = open_device(console, options)
if device is None:
return 0
log = open_log(console, options, output_dir, started)
_say_where(console, options)
console.print(f"[grey62]listening on {band_text(options)} at "
f"{options.rate / 1e6:g} MS/s — control-C to stop"
"[/grey62]")
if not options.decodes_seen:
console.print("[grey62]the first decodes appear when the slot after "
"next ends, which is up to thirty seconds away"
"[/grey62]")
display = None
live = None
if not options.decodes_seen and getattr(console, "is_terminal", False):
from rich.live import Live
display = Ft8Display(console, hold=options.hold,
imperial=options.imperial, grid=options.grid,
band=band_text(options))
display.started = started
live = Live(display.render(), console=console, refresh_per_second=2,
screen=False, transient=False, vertical_overflow="crop")
live.start()
def show_slot(at, found):
if display is not None:
display.update(heard, at)
live.update(display.render())
def show_decode(d):
if options.decodes_seen:
console.print(f"{slot_name(d.at)} {d.snr_db:>4.0f} "
f"{d.offset:>5.1f} {d.hertz:>7.1f} ~ {d.text}")
try:
pump(device, options, heard, log, started, on_slot=show_slot,
on_decode=show_decode, stopping=stopping)
except KeyboardInterrupt:
console.print("\n[grey62]stopped[/grey62]")
finally:
if live is not None:
live.stop()
try:
device.close()
except Exception:
pass
return finish(console, options, output_dir, heard, log)
def _say_where(console, options: Ft8Options) -> None:
"""Whether distances can be worked out, and what it costs if not."""
if options.grid:
where = grid_at(options.grid)
console.print(f"[grey62]aerial at {options.grid.upper()} "
f"({where[0]:.2f},{where[1]:.2f}) — distances and "
f"bearings will be worked out[/grey62]")
return
console.print("[grey62]no grid square set, so no distances and no "
"bearings — `--grid IO91` gives both[/grey62]")
if band_needs(options.band) == "shortwave":
console.print("[grey62]this band is shortwave: a plain receiver "
"needs an upconverter or direct sampling to hear it"
"[/grey62]")
def finish(console, options: Ft8Options, output_dir: str, heard: Heard,
log=None) -> int:
"""Close the log, write the exports, and print the report."""
from . import ft8log
if log is not None:
log.close()
console.print(f"[grey62]{log.lines} decodes written to "
f"{log.path}[/grey62]")
if options.csv:
where = ft8log.write_csv(log.path.with_suffix(".csv"),
heard.band.decodes)
if where is not None:
console.print(f"[grey62]and as a spreadsheet: {where}"
"[/grey62]")
if options.adif:
where = ft8log.write_adif(log.path.with_suffix(".adi"),
heard.band.decodes, band=options.band,
frequency=options.frequency,
my_grid=options.grid)
if where is not None:
console.print(f"[grey62]and as ADIF, marked as heard rather "
f"than worked: {where}[/grey62]")
if options.report:
report(console, heard.band, options)
return heard.decodes
def _away_text(station: Station, options: Ft8Options) -> str:
away = station.away(options.grid)
if away is None:
return ""
km, bearing = away
if options.imperial:
return f"{km * 0.621371:.0f} mi {bearing:.0f}°"
return f"{km:.0f} km {bearing:.0f}°"
def report(console, band: Band, options: Ft8Options) -> None:
"""Three tables, because they answer three different questions."""
from rich.table import Table
stations = sorted(band.all(), key=lambda s: (-s.decodes, s.call))
if not stations and not band.decodes:
console.print("[yellow]nothing decoded[/yellow]")
if band.slots:
console.print("[grey62]the band was quiet, the aerial is not "
"hearing it, or the clock is out: FT8 needs the "
"clock right to a second or two[/grey62]")
return
console.print()
console.print(f"[bold]Stations heard[/bold] — {len(stations)} in "
f"{band.slots} slots")
t = Table(box=None, header_style="bold", pad_edge=False)
t.add_column("station")
t.add_column("grid")
if options.grid:
t.add_column("away")
t.add_column("decodes", justify="right")
t.add_column("best", justify="right")
t.add_column("worst", justify="right")
t.add_column("audio", justify="right")
t.add_column("CQ", justify="right")
for s in stations[:60]:
row = [s.call, s.grid or "—"]
if options.grid:
row.append(_away_text(s, options) or "—")
row += [str(s.decodes), f"{s.best_snr:.0f}", f"{s.worst_snr:.0f}",
f"{s.hertz:.0f} Hz", str(s.calling) if s.calling else ""]
t.add_row(*row)
console.print(t)
if len(stations) > 60:
console.print(f"[grey62]…and {len(stations) - 60} more[/grey62]")
calling = [s for s in stations if s.calling]
if calling:
console.print()
console.print(f"[bold]Calling CQ[/bold] — {len(calling)} stations "
"looking for a contact")
t = Table(box=None, header_style="bold", pad_edge=False)
t.add_column("station")
t.add_column("grid")
if options.grid:
t.add_column("away")
t.add_column("times", justify="right")
t.add_column("best", justify="right")
for s in sorted(calling, key=lambda s: -s.calling)[:30]:
row = [s.call, s.grid or "—"]
if options.grid:
row.append(_away_text(s, options) or "—")
row += [str(s.calling), f"{s.best_snr:.0f}"]
t.add_row(*row)
console.print(t)
worked = [s for s in stations if s.worked]
if worked:
console.print()
console.print("[bold]Who was working whom[/bold] — the exchanges "
"that passed while this was listening")
t = Table(box=None, header_style="bold", pad_edge=False)
t.add_column("station")
t.add_column("answered")
for s in sorted(worked, key=lambda s: -len(s.worked))[:30]:
t.add_row(s.call, ", ".join(sorted(s.worked)[:8]))
console.print(t)
if options.grid:
far = [(s, s.away(options.grid)) for s in stations if s.grid]
far = [(s, a) for s, a in far if a is not None]
if far:
s, (km, bearing) = max(far, key=lambda x: x[1][0])
unit = (f"{km * 0.621371:.0f} miles" if options.imperial
else f"{km:.0f} km")
console.print()
console.print(f"[grey62]furthest heard: {s.call} in {s.grid}, "
f"{unit} on {bearing:.0f}°[/grey62]")

712
bandsaunter/ft8code.py Normal file
View file

@ -0,0 +1,712 @@
"""What an FT8 transmission carries, and how it is wrapped.
No radio in here at all. This is the part between a string of one hundred
and seventy-four bits and a line somebody can read: the checksum, the
error-correcting code that makes FT8 work at signal levels where the
operator hears nothing, the Gray mapping onto eight tones, and the
seventy-seven bits that hold two callsigns and a grid square.
The shape of it, outward from the message:
77 bits what was said: two callsigns and a report, or free text
+ 14 bits a CRC, so a wrong answer is caught rather than printed
= 91 bits
+ 83 bits LDPC parity, so a wrong answer is usually *repaired*
= 174 bits
/ 3 three bits to a tone, in Gray order
= 58 tones
+ 21 tones three Costas arrays, at the start, the middle and the end
= 79 tones at 6.25 Hz apart and 6.25 to the second: 12.64 seconds
The error correction is the whole trick. Fifty-eight tones of payload
carried in one hundred and seventy-four bits is a rate of about one half:
half of what is transmitted is redundancy, and that is what buys a protocol
that decodes at twenty-odd decibels below the noise in the same bandwidth.
A receiver that only took the strongest tone in each symbol and hoped would
decode almost nothing -- which is why the tone detector below reports how
confident it is, bit by bit, rather than just what it thinks it heard.
"""
from __future__ import annotations
from dataclasses import dataclass, field
import numpy as np
from .ft8tables import (BITS, CHECKS, COSTAS, CRC_BITS, CRC_POLYNOMIAL,
GENERATOR, GRAY, PARITY, PAYLOAD, TONES)
__all__ = ["crc14", "with_crc", "crc_holds", "encode", "repair",
"parity_holds", "tones_of",
"bits_of", "unpack", "pack", "CallBook", "Message", "MESSAGE_BITS",
"SYMBOL_HZ", "SYMBOL_S", "SLOT_S", "SENDING_S", "COSTAS", "TONES",
"hash_of", "free_text", "is_standard_call"]
MESSAGE_BITS = 77 # what is actually said, before the checksum
# The channel, in numbers. Six and a quarter of everything: the tones are
# 6.25 Hz apart and there are 6.25 of them a second, which makes each tone
# exactly one cycle of separation long and the whole thing orthogonal.
SYMBOL_HZ = 6.25
SYMBOL_S = 1.0 / SYMBOL_HZ
SENDING_S = TONES * SYMBOL_S # 12.64 s of transmission
SLOT_S = 15.0 # in a slot of fifteen
# ---------------------------------------------------------------------------
# The checksum
# ---------------------------------------------------------------------------
def crc14(bits) -> int:
"""CRC-14 over a sequence of bits, most significant first.
The one thing standing between a repaired codeword and a confident lie.
The error correction below will happily converge on *a* valid codeword
from noise; whether it is the codeword that was sent is what this
answers, and one in sixteen thousand wrong answers gets through, which
over an evening is a handful of lines nobody should trust. Every
decoder prints them anyway, and so does this one -- but only after the
CRC has agreed, which is the difference between a handful and a flood.
"""
top = 1 << (CRC_BITS - 1)
mask = (top << 1) - 1
remainder = 0
for i, bit in enumerate(bits):
if i % 8 == 0:
# A byte at a time into the top of the register, exactly as the
# specification frames it: the message is a byte sequence and
# the odd bits at the end are zeros.
byte = 0
for k in range(8):
byte = (byte << 1) | (bits[i + k] if i + k < len(bits) else 0)
remainder ^= byte << (CRC_BITS - 8)
remainder = ((remainder << 1) ^ CRC_POLYNOMIAL if remainder & top
else remainder << 1)
return remainder & mask
def with_crc(payload) -> np.ndarray:
"""The seventy-seven bits said, plus the fourteen that check them.
The checksum covers eighty-two bits rather than seventy-seven: the
message zero-extended by five, which is what the specification says and
is not the same answer as checksumming seventy-seven.
"""
payload = np.asarray(payload, dtype=np.uint8)
if payload.size != MESSAGE_BITS:
raise ValueError(f"a message is {MESSAGE_BITS} bits, not "
f"{payload.size}")
extended = np.concatenate([payload, np.zeros(5, dtype=np.uint8)])
check = crc14(extended.tolist())
tail = np.array([(check >> (CRC_BITS - 1 - k)) & 1
for k in range(CRC_BITS)], dtype=np.uint8)
return np.concatenate([payload, tail])
def crc_holds(bits91) -> bool:
"""Whether the checksum on a decoded block agrees with its message."""
bits91 = np.asarray(bits91, dtype=np.uint8)
if bits91.size != PAYLOAD:
return False
return bool((with_crc(bits91[:MESSAGE_BITS]) == bits91).all())
# ---------------------------------------------------------------------------
# The error-correcting code
# ---------------------------------------------------------------------------
def _generator() -> np.ndarray:
"""The parity half of the generator, unpacked to bits."""
rows = np.zeros((PARITY, PAYLOAD), dtype=np.uint8)
for m, row in enumerate(GENERATOR):
packed = bytes.fromhex(row)
for k in range(PAYLOAD):
rows[m, k] = (packed[k // 8] >> (7 - k % 8)) & 1
return rows
GEN = _generator()
def encode(bits91) -> np.ndarray:
"""Ninety-one bits in, one hundred and seventy-four out.
Systematic: the message comes out unchanged at the front and the parity
is appended, which is why the decoder can read the message straight off
a repaired codeword without undoing anything.
"""
bits91 = np.asarray(bits91, dtype=np.uint8)
if bits91.size != PAYLOAD:
raise ValueError(f"expected {PAYLOAD} bits, got {bits91.size}")
parity = (GEN @ bits91) & 1
return np.concatenate([bits91, parity.astype(np.uint8)])
def _sparse():
"""The parity-check matrix as flat slots, for the message passing.
Every codeword bit sits in exactly three checks and every check covers
six or seven bits, so the whole graph fits in two small index arrays and
the decoding never needs a scatter-add. The short checks are padded to
seven with a slot pointing at a dummy bit that is held at no opinion.
"""
width = max(len(c) for c in CHECKS)
bit_of_slot = np.full(PARITY * width, BITS, dtype=np.int64) # BITS = dummy
live = np.zeros(PARITY * width, dtype=bool)
for m, check in enumerate(CHECKS):
for j, one_based in enumerate(check):
bit_of_slot[m * width + j] = one_based - 1
live[m * width + j] = True
# And the other way: for each bit, the slots that talk about it.
per_bit = [[] for _ in range(BITS)]
for slot in np.flatnonzero(live):
per_bit[bit_of_slot[slot]].append(slot)
degree = max(len(s) for s in per_bit)
slots_of_bit = np.zeros((BITS, degree), dtype=np.int64)
for n, slots in enumerate(per_bit):
slots_of_bit[n] = slots + [slots[-1]] * (degree - len(slots))
return width, bit_of_slot, live, slots_of_bit
WIDTH, BIT_OF_SLOT, LIVE, SLOTS_OF_BIT = _sparse()
def parity_holds(bits174) -> bool:
"""Whether every one of the eighty-three checks comes out even.
Here rather than left to callers because the padding convention -- short
checks point their spare slots at a dummy bit past the end -- is an
implementation detail of the message passing, and anything outside this
module that had to know about it would be a bug waiting to be written.
"""
bits174 = np.asarray(bits174, dtype=np.uint8)
if bits174.size != BITS:
return False
padded = np.concatenate([bits174, np.zeros(1, dtype=np.uint8)])
sums = padded[BIT_OF_SLOT].copy()
sums[~LIVE] = 0
return bool((sums.reshape(PARITY, WIDTH).sum(axis=1) % 2 == 0).all())
def repair(llr, rounds: int = 30, alpha: float = 0.75):
"""Belief propagation over one or many candidate codewords.
``llr`` is how strongly each bit is believed to be a zero: positive for
zero, negative for one, and the size of it is the confidence. Shape
(174,) for one candidate or (n, 174) for a batch, which is how it is
actually used -- a busy slot throws up hundreds of candidates and doing
them one at a time is most of the decoding time.
Normalised min-sum rather than the exact sum-product: it is within a
few tenths of a decibel of it, and it is multiplication-free in the
inner loop, which at this size matters more than the tenths.
Returns the repaired bits, or None where the checks never came out
even. An answer here still has to pass the CRC before it is believed:
this finds *a* codeword, and noise has valid codewords in it too.
"""
single = np.ndim(llr) == 1
belief = np.atleast_2d(np.asarray(llr, dtype=np.float32))
n = belief.shape[0]
if belief.shape[1] != BITS:
raise ValueError(f"a codeword is {BITS} bits, not {belief.shape[1]}")
messages = np.zeros((n, PARITY * WIDTH), dtype=np.float32)
dead = ~LIVE
dead_grid = dead.reshape(PARITY, WIDTH)
out = np.zeros((n, BITS), dtype=np.uint8)
done = np.zeros(n, dtype=bool)
def totals(msgs):
"""What every bit is believed to be, once its three checks are in.
A gather and a sum rather than a scatter-add: every bit sits in
exactly three checks, so the three places to look are known in
advance and np.add.at -- which is where an earlier version of this
spent most of its time -- is not needed at all.
"""
return belief + msgs[:, SLOTS_OF_BIT].sum(axis=2)
for _ in range(rounds):
total = totals(messages)
# What each check hears about a bit, less what it said itself.
wide = np.concatenate([total, np.zeros((n, 1), dtype=np.float32)],
axis=1)
heard = (wide[:, BIT_OF_SLOT] - messages).reshape(n, PARITY, WIDTH)
size = np.abs(heard)
size[:, dead_grid] = np.inf
# The smallest and the next smallest in each check, so "the
# smallest of the others" is a lookup rather than a loop.
order = np.argsort(size, axis=2)
smallest = np.take_along_axis(size, order[:, :, :1], axis=2)
next_up = np.take_along_axis(size, order[:, :, 1:2], axis=2)
others = np.where(size == smallest, next_up, smallest)
odd = (heard < 0).sum(axis=2, keepdims=True) % 2 == 1
mine = np.where(heard < 0, -1.0, 1.0).astype(np.float32)
whole = np.where(odd, -1.0, 1.0).astype(np.float32)
messages = (alpha * whole * mine
* np.minimum(others, 1e4)).astype(np.float32)
messages = messages.reshape(n, -1)
messages[:, dead] = 0.0
hard = (totals(messages) < 0).astype(np.uint8)
wide_hard = np.concatenate([hard, np.zeros((n, 1), dtype=np.uint8)],
axis=1)
sums = wide_hard[:, BIT_OF_SLOT].reshape(n, PARITY, WIDTH).copy()
sums[:, dead_grid] = 0
even = (sums.sum(axis=2) % 2 == 0).all(axis=1)
fresh = even & ~done
if fresh.any():
out[fresh] = hard[fresh]
done |= fresh
if done.all():
break
if single:
return out[0] if done[0] else None
return out, done
# ---------------------------------------------------------------------------
# Tones
# ---------------------------------------------------------------------------
UNGRAY = np.zeros(8, dtype=np.uint8)
for _value, _tone in enumerate(GRAY):
UNGRAY[_tone] = _value
SYNC_AT = (0, 36, 72) # where the three Costas arrays sit
def tones_of(bits174) -> np.ndarray:
"""The seventy-nine tones that carry these bits, sync included."""
bits174 = np.asarray(bits174, dtype=np.uint8)
if bits174.size != BITS:
raise ValueError(f"a codeword is {BITS} bits, not {bits174.size}")
trips = bits174.reshape(-1, 3)
values = trips[:, 0] * 4 + trips[:, 1] * 2 + trips[:, 2]
data = np.array([GRAY[v] for v in values], dtype=np.uint8)
out = np.zeros(TONES, dtype=np.uint8)
costas = np.array(COSTAS, dtype=np.uint8)
taken = 0
for i in range(TONES):
which = i // 36 if i % 36 < 7 else -1
if i in range(0, 7) or i in range(36, 43) or i in range(72, 79):
out[i] = costas[(i - (0 if i < 7 else 36 if i < 43 else 72))]
else:
out[i] = data[taken]
taken += 1
return out
def bits_of(tones) -> np.ndarray:
"""The bits those tones carry, sync thrown away."""
tones = np.asarray(tones, dtype=np.uint8)
if tones.size != TONES:
raise ValueError(f"a transmission is {TONES} tones, not {tones.size}")
data = [tones[i] for i in range(TONES)
if not (i < 7 or 36 <= i < 43 or 72 <= i)]
out = np.zeros(BITS, dtype=np.uint8)
for k, tone in enumerate(data):
value = UNGRAY[tone]
out[3 * k] = (value >> 2) & 1
out[3 * k + 1] = (value >> 1) & 1
out[3 * k + 2] = value & 1
return out
# ---------------------------------------------------------------------------
# Callsigns
# ---------------------------------------------------------------------------
ALPHANUM_SPACE = " 0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ" # 37
ALPHANUM = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ" # 36
NUMERIC = "0123456789" # 10
LETTERS_SPACE = " ABCDEFGHIJKLMNOPQRSTUVWXYZ" # 27
FULL = " 0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ+-./?" # 42
WITH_SLASH = " 0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ/" # 38
TOKENS = 2_063_592 # DE, QRZ, CQ and the CQ-with-an-argument forms
MAX22 = 4_194_304 # room for a hashed callsign
MAX_GRID = 32_400 # four-character grids, before the report forms
def hash_of(call: str, width: int = 22) -> int:
"""The short hash a compound callsign is carried as.
A callsign that will not fit in twenty-eight bits -- anything with a
slash in it, and anything long -- travels as a hash, and the receiver
is expected to have heard the full call earlier in the exchange and to
remember it. That is why a fresh receiver prints <...>: not a decoding
failure, but a station whose full name has not been said yet in
anything this receiver heard.
"""
packed = 0
padded = (call.upper() + " ")[:11]
for ch in padded:
packed = packed * 38 + (WITH_SLASH.index(ch) if ch in WITH_SLASH else 0)
return int((47_055_833_459 * packed) >> (64 - width)) & ((1 << width) - 1)
def is_standard_call(call: str) -> bool:
"""Whether a callsign fits the twenty-eight bit form.
One or two characters of prefix, a digit, and up to three letters. Most
callsigns on earth; not the ones with a slash in them.
"""
return _pack_standard(call) is not None
def _pack_standard(call: str):
"""A plain callsign as its number, or None if it is not a plain one.
The form is one or two characters of prefix, exactly one digit, and up
to three letters -- and the twenty-eight bit packing wants the digit in
the third place. Which character is *the* digit cannot be found by
looking at the second one: Z33Z has a digit there and it belongs to the
prefix. It is the last character that is not one of the trailing
letters, so that is how it is found.
"""
call = (call or "").strip().upper()
if not call or "/" in call or len(call) > 6:
return None
at = len(call) - 1
while at >= 0 and call[at].isalpha():
at -= 1
if at < 0 or at > 2 or not call[at].isdigit():
return None
call = " " * (2 - at) + call
if len(call) > 6:
return None
call = (call + " ")[:6]
if (call[0] not in ALPHANUM_SPACE or call[1] not in ALPHANUM
or call[2] not in NUMERIC):
return None
if any(c not in LETTERS_SPACE for c in call[3:6]):
return None
n = ALPHANUM_SPACE.index(call[0])
n = n * 36 + ALPHANUM.index(call[1])
n = n * 10 + NUMERIC.index(call[2])
n = n * 27 + LETTERS_SPACE.index(call[3])
n = n * 27 + LETTERS_SPACE.index(call[4])
n = n * 27 + LETTERS_SPACE.index(call[5])
return n
def _unpack_standard(n: int) -> str:
out = [""] * 6
out[5] = LETTERS_SPACE[n % 27]
n //= 27
out[4] = LETTERS_SPACE[n % 27]
n //= 27
out[3] = LETTERS_SPACE[n % 27]
n //= 27
out[2] = NUMERIC[n % 10]
n //= 10
out[1] = ALPHANUM[n % 36]
n //= 36
out[0] = ALPHANUM_SPACE[n % 37]
return "".join(out).strip()
@dataclass
class CallBook:
"""Full callsigns heard earlier, so a hash can be turned back into one.
A compound callsign travels as a hash and is expected to be remembered
from earlier in the same exchange. This is that memory. It is only
ever added to by callsigns that arrived in full and passed their CRC,
so a station that was never heard properly stays <...> rather than being
guessed at -- a wrong callsign in a log is worse than a missing one.
"""
by_hash: dict = field(default_factory=dict)
def remember(self, call: str) -> None:
call = (call or "").strip().upper()
if len(call) < 3 or call in ("CQ", "DE", "QRZ"):
return
for width in (10, 12, 22):
self.by_hash[(width, hash_of(call, width))] = call
def look_up(self, value: int, width: int = 22) -> str:
return self.by_hash.get((width, value), "<...>")
def _unpack28(n: int, book: CallBook) -> str:
"""One of the two callsign fields of a standard message."""
if n < TOKENS:
if n == 0:
return "DE"
if n == 1:
return "QRZ"
if n == 2:
return "CQ"
if n <= 1002:
return f"CQ {n - 3:03d}"
if n <= 532_443:
rest = n - 1003
letters = [""] * 4
for i in (3, 2, 1, 0):
letters[i] = LETTERS_SPACE[rest % 27]
rest //= 27
return "CQ " + "".join(letters).strip()
return "<...>"
n -= TOKENS
if n < MAX22:
return book.look_up(n, 22)
return _unpack_standard(n - MAX22)
def _unpack_grid(value: int, rover: int) -> str:
"""The fifteen-bit field: a grid square, a signal report, or a sign-off."""
if value <= MAX_GRID:
n = value
out = ["", "", "", ""]
out[3] = NUMERIC[n % 10]
n //= 10
out[2] = NUMERIC[n % 10]
n //= 10
out[1] = chr(ord("A") + n % 18)
n //= 18
out[0] = chr(ord("A") + n % 18)
grid = "".join(out)
return ("R " + grid) if rover else grid
report = value - MAX_GRID
if report == 1:
return ""
if report == 2:
return "RRR"
if report == 3:
return "RR73"
if report == 4:
return "73"
decibels = report - 35
return f"{'R' if rover else ''}{decibels:+03d}"
def _pack_grid(text: str):
"""The inverse: a grid, a report or a token as fifteen bits and a flag."""
text = (text or "").strip().upper()
if not text:
return MAX_GRID + 1, 0
if text == "RRR":
return MAX_GRID + 2, 0
if text == "RR73":
return MAX_GRID + 3, 0
if text == "73":
return MAX_GRID + 4, 0
rover = 0
if text.startswith("R ") or (text.startswith("R") and len(text) > 1
and text[1] in "+-"):
rover, text = 1, text[2:].strip() if text[1] == " " else text[1:]
if (len(text) == 4 and "A" <= text[0] <= "R" and "A" <= text[1] <= "R"
and text[2].isdigit() and text[3].isdigit()):
value = ((ord(text[0]) - 65) * 18 + (ord(text[1]) - 65)) * 10
value = (value + int(text[2])) * 10 + int(text[3])
return value, rover
try:
return MAX_GRID + 35 + int(text), rover
except ValueError:
return None
# ---------------------------------------------------------------------------
# Messages
# ---------------------------------------------------------------------------
@dataclass
class Message:
"""One decoded line, and what could be picked out of it."""
text: str = ""
kind: str = "" # standard, free text, telemetry, ...
calls: tuple = () # the callsigns in it, in order
grid: str = "" # a four-character grid, where there was one
report: str = "" # a signal report, where there was one
calling: bool = False # whether this is a CQ
def _number(bits, start: int, count: int) -> int:
value = 0
for k in range(count):
value = (value << 1) | int(bits[start + k])
return value
def _put(bits, start: int, count: int, value: int) -> None:
for k in range(count):
bits[start + k] = (value >> (count - 1 - k)) & 1
def free_text(bits77) -> str:
"""The thirteen-character form: anything at all, at a price.
Base forty-two over seventy-one bits, which is thirteen characters and
no callsign structure -- so a receiver cannot tell who sent it from the
message alone, and nor can the network that reports these things.
"""
n = _number(bits77, 0, 71)
out = [""] * 13
for i in range(12, -1, -1):
out[i] = FULL[n % 42]
n //= 42
return "".join(out).strip()
def unpack(bits77, book: CallBook | None = None) -> Message:
"""Seventy-seven bits as something to read.
The type is the last three bits, which is an odd place for it until you
remember that everything before it is a different length in each form.
"""
bits77 = np.asarray(bits77, dtype=np.uint8)
book = book or CallBook()
i3 = _number(bits77, 74, 3)
if i3 == 0:
n3 = _number(bits77, 71, 3)
if n3 == 0:
return Message(text=free_text(bits77), kind="free text")
if n3 == 5:
value = _number(bits77, 0, 71)
return Message(text=f"{value:018X}", kind="telemetry")
return Message(text=free_text(bits77), kind="free text")
if i3 in (1, 2):
a = _number(bits77, 0, 28)
a_rover = _number(bits77, 28, 1)
b = _number(bits77, 29, 28)
b_rover = _number(bits77, 57, 1)
rover = _number(bits77, 58, 1)
grid = _number(bits77, 59, 15)
one = _unpack28(a, book)
two = _unpack28(b, book)
if a_rover:
one += "/R"
if b_rover:
two += "/R"
extra = _unpack_grid(grid, rover)
text = " ".join(x for x in (one, two, extra) if x)
looks_like_grid = (len(extra) == 4 and extra[:2].isalpha()
and extra[2:].isdigit())
return Message(text=text, kind="standard",
calls=tuple(c for c in (one, two)
if c not in ("CQ", "DE", "QRZ")
and not c.startswith("CQ ")),
grid=extra if looks_like_grid else "",
report="" if looks_like_grid else extra,
calling=one.startswith("CQ") or one == "QRZ")
if i3 == 4:
# A compound callsign: one hashed, one carried in full as eleven
# characters of base thirty-eight.
short = _number(bits77, 0, 12)
n58 = _number(bits77, 12, 58)
flip = _number(bits77, 70, 1)
kind = _number(bits77, 71, 2)
rest = n58
letters = [""] * 11
for i in range(10, -1, -1):
letters[i] = WITH_SLASH[rest % 38]
rest //= 38
full = "".join(letters).strip()
other = book.look_up(short, 12)
one, two = (full, other) if flip else (other, full)
tail = {1: "RRR", 2: "RR73", 3: "73"}.get(kind, "")
text = " ".join(x for x in (one, two, tail) if x)
return Message(text=text, kind="non-standard",
calls=tuple(c for c in (one, two) if c != "<...>"),
report=tail, calling=one.startswith("CQ"))
return Message(text=free_text(bits77), kind=f"type {i3}")
def pack(text: str, book: CallBook | None = None):
"""The inverse, for the standard and free-text forms.
Here so the decoder can be tested against something that is not itself.
A receiver never needs to build a message -- this one does not transmit
-- but a test that encodes with the same misunderstanding it decodes
with agrees with itself about anything they are both wrong about.
"""
text = (text or "").strip().upper()
bits = np.zeros(MESSAGE_BITS, dtype=np.uint8)
words = text.split()
if len(words) in (2, 3) or (len(words) == 4 and words[0] == "CQ"):
made = _pack_standard_message(words, bits)
if made is not None:
return made
return _pack_free_text(text, bits)
def _pack_standard_message(words, bits):
if words[0] == "CQ" and len(words) >= 3 and len(words[1]) <= 4 \
and not _is_call(words[1]):
one, rest = f"CQ {words[1]}", words[2:]
else:
one, rest = words[0], words[1:]
if not rest:
return None
two, extra = rest[0], (rest[1] if len(rest) > 1 else "")
a = _pack28(one)
b = _pack28(two)
if a is None or b is None:
return None
grid = _pack_grid(extra)
if grid is None:
return None
value, rover = grid
_put(bits, 0, 28, a)
_put(bits, 28, 1, 0)
_put(bits, 29, 28, b)
_put(bits, 57, 1, 0)
_put(bits, 58, 1, rover)
_put(bits, 59, 15, value)
_put(bits, 74, 3, 1)
return bits
def _is_call(word: str) -> bool:
return any(c.isdigit() for c in word) and _pack_standard(word) is not None
def _pack28(call: str):
call = call.strip().upper()
if call == "DE":
return 0
if call == "QRZ":
return 1
if call == "CQ":
return 2
if call.startswith("CQ "):
rest = call[3:].strip()
if rest.isdigit() and len(rest) == 3:
return 3 + int(rest)
if 1 <= len(rest) <= 4 and all(c in LETTERS_SPACE for c in rest):
n = 0
for ch in (rest + " ")[:4]:
n = n * 27 + LETTERS_SPACE.index(ch)
return 1003 + n
return None
plain = _pack_standard(call)
if plain is not None:
return TOKENS + MAX22 + plain
# Deliberately not a hash. A hash always succeeds, and a packer that
# always succeeds would send "HELLO WORLD" as two hashed callsigns
# rather than as the free text it plainly is.
return None
def _pack_free_text(text: str, bits):
text = "".join(c if c in FULL else " " for c in text)[:13]
text = text.rjust(13)
n = 0
for ch in text:
n = n * 42 + FULL.index(ch)
_put(bits, 0, 71, n)
_put(bits, 71, 3, 0)
_put(bits, 74, 3, 0)
return bits

177
bandsaunter/ft8log.py Normal file
View file

@ -0,0 +1,177 @@
"""Writing FT8 decodes down.
Three forms, because three different things want to read them. The log is
for a person and for this program's own report; the CSV is for a
spreadsheet; the ADIF is for the logging and spotting programs every
amateur already has.
The ADIF is the one worth a note. It is the format for recording contacts,
and nothing here makes a contact -- this receiver does not transmit. So
what is written is a record of a station *heard*, marked as such, which is
what a propagation map or a reverse-beacon feed wants. Passing it off as a
worked contact would put claims into somebody's log that they cannot make.
"""
from __future__ import annotations
import csv
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from . import __version__
__all__ = ["Ft8Log", "open_log", "write_csv", "write_adif", "read_log",
"LOG_VERSION"]
LOG_VERSION = 1
@dataclass
class Ft8Log:
"""A file of decodes, one to a line, written as they arrive."""
path: Path
handle: object = None
lines: int = 0
def append(self, found) -> None:
if self.handle is None:
return
when = datetime.fromtimestamp(found.at, timezone.utc)
self.handle.write(
f"{when.strftime('%Y-%m-%d %H:%M:%S')} "
f"{found.snr_db:>4.0f} {found.offset:>5.1f} "
f"{found.hertz:>7.1f} ~ {found.text}\n")
self.handle.flush()
self.lines += 1
def close(self) -> None:
if self.handle is not None:
self.handle.close()
self.handle = None
def open_log(directory, started: float, frequency: float,
band: str = "") -> Ft8Log | None:
"""A new log for this run, or nothing if it cannot be opened."""
when = datetime.fromtimestamp(started, timezone.utc)
path = Path(directory) / f"ft8-{when.strftime('%Y%m%d-%H%M%S')}.txt"
try:
path.parent.mkdir(parents=True, exist_ok=True)
handle = open(path, "w")
except OSError:
return None
handle.write(f"# bandsaunter {__version__} FT8 log v{LOG_VERSION}\n")
handle.write(f"# {frequency / 1e6:g} MHz"
f"{(' ' + band) if band else ''}, times are UTC\n")
handle.write("# when snr dt hz ~ message\n")
handle.flush()
return Ft8Log(path=path, handle=handle)
def read_log(path) -> list[dict]:
"""A log back off the disk, for reading an evening again.
Deliberately forgiving: a line that cannot be parsed is skipped rather
than raising, because a log is often read while it is still being
written and the last line may be half there.
"""
out = []
try:
text = Path(path).read_text()
except OSError:
return out
for line in text.splitlines():
if line.startswith("#") or "~" not in line:
continue
head, message = line.split("~", 1)
parts = head.split()
if len(parts) < 5:
continue
try:
when = datetime.strptime(f"{parts[0]} {parts[1]}",
"%Y-%m-%d %H:%M:%S")
out.append({"at": when.replace(tzinfo=timezone.utc).timestamp(),
"snr_db": float(parts[2]), "offset": float(parts[3]),
"hertz": float(parts[4]), "text": message.strip()})
except ValueError:
continue
return out
def write_csv(path, decodes) -> Path | None:
try:
with open(path, "w", newline="") as fh:
writer = csv.writer(fh)
writer.writerow(["utc", "snr_db", "offset_s", "hertz", "kind",
"calls", "grid", "report", "cq", "message"])
for d in decodes:
when = datetime.fromtimestamp(d.at, timezone.utc)
writer.writerow([when.strftime("%Y-%m-%d %H:%M:%S"),
f"{d.snr_db:.0f}", f"{d.offset:.1f}",
f"{d.hertz:.1f}", d.kind,
" ".join(d.calls), d.grid, d.report,
"yes" if d.calling else "", d.text])
except OSError:
return None
return Path(path)
def _adif(field: str, value: str) -> str:
value = str(value)
return f"<{field}:{len(value)}>{value} "
def write_adif(path, decodes, band: str = "", frequency: float = 0.0,
my_grid: str = "") -> Path | None:
"""The decodes as ADIF records of stations heard.
One record per station rather than per decode: a logging program that
was handed forty identical records for one evening's CQ calls would be
right to complain. The strongest report is the one kept, that being
the one worth passing on.
"""
best: dict = {}
for d in decodes:
sender = d.calls[1] if len(d.calls) > 1 else (
d.calls[0] if d.calls else "")
if not sender or sender == "<...>":
continue
was = best.get(sender)
if was is None or d.snr_db > was.snr_db:
best[sender] = d
if d.grid and not getattr(best[sender], "grid", ""):
best[sender].grid = d.grid
try:
with open(path, "w") as fh:
fh.write(f"bandsaunter {__version__} — FT8 stations heard.\n")
fh.write("These are receptions, not contacts: this program "
"does not transmit.\n")
fh.write(_adif("ADIF_VER", "3.1.4") + "\n")
fh.write(_adif("PROGRAMID", "bandsaunter") + "\n")
fh.write(_adif("PROGRAMVERSION", __version__) + "\n")
fh.write("<EOH>\n")
for call, d in sorted(best.items()):
when = datetime.fromtimestamp(d.at, timezone.utc)
row = (_adif("CALL", call)
+ _adif("QSO_DATE", when.strftime("%Y%m%d"))
+ _adif("TIME_ON", when.strftime("%H%M%S"))
+ _adif("MODE", "FT8"))
if band:
row += _adif("BAND", band)
if frequency:
row += _adif("FREQ", f"{frequency / 1e6:.6f}")
if d.grid:
row += _adif("GRIDSQUARE", d.grid)
if my_grid:
row += _adif("MY_GRIDSQUARE", my_grid.upper())
row += _adif("RST_RCVD", f"{d.snr_db:.0f}")
# Said plainly in the record itself, so that a log this is
# imported into cannot quietly become a claim.
row += _adif("COMMENT", "heard only, not worked")
row += _adif("QSO_RANDOM", "Y")
fh.write(row + "<EOR>\n")
except OSError:
return None
return Path(path)

172
bandsaunter/ft8sim.py Normal file
View file

@ -0,0 +1,172 @@
"""A made-up FT8 band, on a real clock.
For trying the display, the report and the exports without an aerial, and
for testing the whole path from samples to tables. It stands in for a
receiver: same `tune`, `read_samples` and `close`, same sample rate, same
complex samples.
What it cannot tell you is whether your receiver hears anything, which is
the one question a simulator is never allowed to answer. It is also the
lesson this project learned expensively on another band: an encoder tested
against its own decoder agrees with it about anything they are both wrong
about. So the decoding is tested against real off-air recordings; this is
here for the parts above the decoding.
"""
from __future__ import annotations
import math
import time
import numpy as np
from . import ft8code as code
from . import ft8wave as wave
__all__ = ["SimulatedBand", "CALLS", "GRIDS"]
# A plausible spread: a few continents, a few strengths, a few habits.
CALLS = ("G4UJS", "SP4FCA", "IK4LZH", "EA1ABT", "DL1UDO", "OH8GDU",
"JA2GQT", "VK4BLE", "RA6ABO", "F4FSY", "K1ABC", "W2XYZ",
"PA3EPP", "SM0ABC", "LZ1LZ", "9A9TT", "ZS6ABC", "LU1AAA")
GRIDS = ("IO83", "KO03", "JN54", "IN73", "JO31", "KP24",
"PM85", "QG62", "KN96", "JN25", "FN42", "FN20",
"JO21", "JO99", "KN12", "JN75", "KG33", "GF05")
class SimulatedBand:
"""A receiver that invents a band of FT8 stations.
Paced against the real clock, because everything about FT8 is: the
slots are quarter-minutes of UTC, and a simulator that produced fifteen
seconds of audio instantly would make the section look as though it
worked while testing none of the timing that makes it work.
"""
def __init__(self, rate: float = 240_000.0, seed: int = 0,
stations: int = 12, realtime: bool = True,
clock=time.time, sleep=time.sleep):
self.rate = float(rate)
self.frequency = 0.0
self.rng = np.random.default_rng(seed)
self.stations = max(1, int(stations))
self.realtime = realtime
self.clock = clock
self.sleep = sleep
self.closed = False
self._audio_rate = 12_000
self._made: dict[float, np.ndarray] = {}
self._one_sided: dict[float, np.ndarray] = {}
self._at = None
# -- the receiver's shape -------------------------------------------
def tune(self, hz: float) -> None:
self.frequency = float(hz)
def close(self) -> None:
self.closed = True
def read_samples(self, count: int):
"""A block of samples, paced to take as long as it would on the air."""
if self.closed:
return None
now = self.clock()
if self._at is None:
self._at = now
wanted = count / self.rate
if self.realtime:
behind = self._at + wanted - self.clock()
if behind > 0:
self.sleep(behind)
block = self._samples_for(self._at, count)
self._at += wanted
return block
# -- making the band ------------------------------------------------
def _slot_audio(self, slot_at: float) -> np.ndarray:
"""One slot of the band, at twelve kilohertz, made once and kept."""
have = self._made.get(slot_at)
if have is not None:
return have
rng = np.random.default_rng(int(slot_at) & 0xFFFFFFFF)
total = int(self._audio_rate * code.SLOT_S)
audio = rng.standard_normal(total).astype(np.float32) * 0.02
# Stations pick a frequency and keep it, mostly, and take turns.
even = int(slot_at / code.SLOT_S) % 2 == 0
for i in range(self.stations):
if (i % 2 == 0) != even:
continue
call = CALLS[(i + int(slot_at / code.SLOT_S) // 2) % len(CALLS)]
grid = GRIDS[CALLS.index(call)]
other = CALLS[(i * 7 + 3) % len(CALLS)]
pick = rng.integers(0, 3)
if pick == 0:
text = f"CQ {call} {grid}"
elif pick == 1:
text = f"{other} {call} {grid}"
else:
text = f"{other} {call} {rng.integers(-24, 5):+03d}"
hertz = 300.0 + i * 190.0 + float(rng.integers(-30, 30))
strength = float(rng.uniform(-18.0, 10.0))
one = wave.transmit(text, rate=float(self._audio_rate),
hertz=hertz,
offset=0.5 + float(rng.uniform(-0.3, 0.6)),
seconds=code.SLOT_S)
scale = 10.0 ** (strength / 20.0) * 0.05
audio[:one.size] += (one * scale).astype(np.float32)
if len(self._made) > 2:
self._made.clear()
self._made[slot_at] = audio
return audio
def _samples_for(self, at: float, count: int):
"""Complex samples covering ``count`` samples from ``at``.
The band is made as audio and then put back on a carrier, which is
the reverse of what the section does to it. One-sided, so that
what comes out of the demodulator is what went in: a real audio
tone at 1000 Hz has to arrive as a complex tone at +1000 Hz, or
upper sideband and lower sideband would be the same thing.
Held at the audio rate and stepped through, rather than properly
resampled. The images that leaves sit at multiples of twelve
kilohertz and the demodulator's own decimation filter removes them
-- which is worth saying out loud, because it means this simulator
leans on the thing it is being used to test. That is tolerable for
the display and the report, and is exactly why the decoding itself
is tested against real recordings instead.
"""
out = np.zeros(count, dtype=np.complex64)
step = 1.0 / self.rate
ratio = self._audio_rate / self.rate
for k in range(0, count, 4096):
piece = min(4096, count - k)
when = at + k * step
slot_at = math.floor(when / code.SLOT_S) * code.SLOT_S
audio = self._analytic_slot(slot_at)
into = (when - slot_at) * self._audio_rate
idx = np.clip((into + np.arange(piece) * ratio).astype(np.int64),
0, audio.size - 1)
out[k:k + piece] = audio[idx].astype(np.complex64)
return out
def _analytic_slot(self, slot_at: float) -> np.ndarray:
have = self._one_sided.get(slot_at)
if have is None:
have = _analytic(self._slot_audio(slot_at))
# Two slots kept: a block of samples commonly straddles a
# boundary, and rebuilding a slot for every such block is most
# of the simulator's time.
if len(self._one_sided) > 2:
self._one_sided.clear()
self._one_sided[slot_at] = have
return have
def _analytic(real_audio: np.ndarray) -> np.ndarray:
"""The one-sided version of a real signal: no energy below zero."""
spectrum = np.fft.fft(real_audio)
n = spectrum.size
spectrum[n // 2 + 1:] = 0.0
spectrum[1:(n + 1) // 2] *= 2.0
return np.fft.ifft(spectrum)

219
bandsaunter/ft8tables.py Normal file
View file

@ -0,0 +1,219 @@
"""The two fixed tables the FT8 code is defined by.
Everything else in this section is worked out from first principles -- the
tones, the sync, the parity arithmetic, the way a callsign is squeezed into
twenty-eight bits. These two cannot be, because they are not derived from
anything. They are the code, chosen once by its designers and published, and
a receiver that guessed at them would be speaking a different protocol.
Taken from ft8_lib (https://github.com/kgoba/ft8_lib), MIT licensed,
Copyright (c) 2018 K\u0101rlis Goba, which took them in turn from WSJT-X.
Reproduced here under that licence; the MIT terms permit it and this file
carries the attribution they ask for. Nothing else in bandsaunter comes from
there: the decoder below is its own.
GENERATOR is the parity half of the systematic generator: row m gives the
message bits that are exclusive-ored to make parity bit m, as ninety-one bits
in twelve bytes, most significant bit first.
CHECKS is the sparse parity-check matrix, as the bit positions each of the
eighty-three checks covers, one-based as published. It is *not* derivable
from GENERATOR even though both describe the same code: the generator's
parity half runs to a weight of fifty-odd bits per row, and belief
propagation on a matrix that dense says nothing useful. The sparse one is a
different basis for the same dual space, six or seven bits to a check, and it
is the one the decoding works on. That they agree is checked by a test
rather than assumed -- encode at random, and every sparse check must come out
even.
"""
__all__ = ["GENERATOR", "CHECKS", "COSTAS", "GRAY", "BITS", "PAYLOAD",
"PARITY", "TONES", "CRC_POLYNOMIAL", "CRC_BITS"]
# The 7x7 Costas array that starts, middles and ends every transmission.
COSTAS = (3, 1, 4, 0, 6, 5, 2)
# Three bits to a tone, in Gray order, so that mistaking a tone for its
# neighbour costs one bit rather than three.
GRAY = (0, 1, 3, 2, 5, 6, 4, 7)
BITS = 174 # bits in a codeword
PAYLOAD = 91 # of which are message and checksum
PARITY = BITS - PAYLOAD
TONES = 79 # channel symbols, sync included
# CRC-14, without the leading bit.
CRC_POLYNOMIAL = 0x2757
CRC_BITS = 14
GENERATOR: tuple[str, ...] = (
"8329CE11BF31EAF509F27FC0",
"761C264E25C2593354931320",
"DC265902FB277C6410A1BDC0",
"1B3F417858CD2DD33EC7F620",
"09FDA4FEE04195FD034783A0",
"077CCCC11B8873ED5C3D48A0",
"29B62AFE3CA036F4FE1A9DA0",
"6054FAF5F35D96D3B0C8C3E0",
"E20798E4310EED27884AE900",
"775C9C08E80E26DDAE563180",
"B0B811028C2BF997213487C0",
"18A0C9231FC60ADF5C5EA320",
"76471E8302A0721E01B12B80",
"FFBCCB80CA8341FAFB47B2E0",
"66A72A158F9325A2BF671700",
"C4243689FE85B1C51363A180",
"0DFF739414D1A1B34B1C2700",
"15B48830636C8B99894972E0",
"29A89C0D3DE81D665489B0E0",
"4F126F37FA51CBE61BD6B940",
"99C47239D0D97D3C84E09400",
"1919B75119765621BB4F1E80",
"09DB12D731FAEE0B86DF6B80",
"488FC33DF43FBDEEA4EAFB40",
"827423EE40B675F756EB5FE0",
"ABE197C484CB74757144A9A0",
"2B500E4BC0EC5A6D2BDBDD00",
"C474AA53D702187616693600",
"8EBA1A13DB3390BD6718CEC0",
"753844673A27782CC42012E0",
"06FF83A145C37035A5C12680",
"3B37417858CC2DD33EC3F620",
"9A4A5A28EE17CA9C324842C0",
"BC29F465309C977E89610A40",
"2663AE6DDF8B5CE2BB294880",
"46F231EFE457034C18144180",
"3FB2CE85ABE9B0C72E06FBE0",
"DE87481F282C153971A0A2E0",
"FCD7CCF23C69FA99BBA14120",
"F0261447E9490CA8E474CEC0",
"4410115818196F95CDD70120",
"088FC31DF4BFBDE2A4EAFB40",
"B8FEF1B6307729FB0A078C00",
"5AFEA7ACCCB77BBC9D99A900",
"49A7016AC653F65ECDC90760",
"1944D085BE4E7DA8D6CC7D00",
"251F62ADC4032F0EE7140020",
"56471F8702A0721E00B12B80",
"2B8E4923F2DD51E2D537FA00",
"6B550A40A66F4755DE95C260",
"A18AD28D4E27FE92A4F6C840",
"10C2E586388CB82A3D807580",
"EF34A41817EE02133DB2EB00",
"7E9C0C54325A9C15836E0000",
"3693E572D1FDE4CDF079E860",
"BFB2CEC5ABE1B0C72E07FBE0",
"7EE18230C583CCCC57D4B080",
"A066CB2FEDAFC9F526641260",
"BB23725ABC47CC5F4CC4CD20",
"DED9DBA3BEE40C59B5609B40",
"D9A7016AC653E6DECDC90360",
"9AD46AED5F707F280AB5FC40",
"E5921C77822587316D7D3C20",
"4F14DA8242A8B86DCA733520",
"8B8B507AD467D4441DF770E0",
"22831C9CF1169467AD04B680",
"213B838FE2AE54C38EE71800",
"5D926B6DD71F085181A4E120",
"66AB79D4B29EE6E69509E560",
"958148682D748A38DD68BAA0",
"B8CE020CF069C32A723AB140",
"F4331D6D461607E957527460",
"6DA23BA424B9596133CF9C80",
"A636BCBC7B30C5FBEAE67FE0",
"5CB0D86A07DF654A9089A200",
"F11F106848780FC9ECDD80A0",
"1FBB5364FB8D2C9D730D5BA0",
"FCB86BC70A50C9D02A5D0340",
"A534433029EAC15F322E34C0",
"C989D9C7C3D3B8C55D751300",
"7BB38B2F0186D46643AE9620",
"2644EBADEB44B9467D1F42C0",
"608CC857594BFBB55D696000",
)
CHECKS: tuple[tuple[int, ...], ...] = (
(4, 31, 59, 91, 92, 96, 153),
(5, 32, 60, 93, 115, 146),
(6, 24, 61, 94, 122, 151),
(7, 33, 62, 95, 96, 143),
(8, 25, 63, 83, 93, 96, 148),
(6, 32, 64, 97, 126, 138),
(5, 34, 65, 78, 98, 107, 154),
(9, 35, 66, 99, 139, 146),
(10, 36, 67, 100, 107, 126),
(11, 37, 67, 87, 101, 139, 158),
(12, 38, 68, 102, 105, 155),
(13, 39, 69, 103, 149, 162),
(8, 40, 70, 82, 104, 114, 145),
(14, 41, 71, 88, 102, 123, 156),
(15, 42, 59, 106, 123, 159),
(1, 33, 72, 106, 107, 157),
(16, 43, 73, 108, 141, 160),
(17, 37, 74, 81, 109, 131, 154),
(11, 44, 75, 110, 121, 166),
(45, 55, 64, 111, 130, 161, 173),
(8, 46, 71, 112, 119, 166),
(18, 36, 76, 89, 113, 114, 143),
(19, 38, 77, 104, 116, 163),
(20, 47, 70, 92, 138, 165),
(2, 48, 74, 113, 128, 160),
(21, 45, 78, 83, 117, 121, 151),
(22, 47, 58, 118, 127, 164),
(16, 39, 62, 112, 134, 158),
(23, 43, 79, 120, 131, 145),
(19, 35, 59, 73, 110, 125, 161),
(20, 36, 63, 94, 136, 161),
(14, 31, 79, 98, 132, 164),
(3, 44, 80, 124, 127, 169),
(19, 46, 81, 117, 135, 167),
(7, 49, 58, 90, 100, 105, 168),
(12, 50, 61, 118, 119, 144),
(13, 51, 64, 114, 118, 157),
(24, 52, 76, 129, 148, 149),
(25, 53, 69, 90, 101, 130, 156),
(20, 46, 65, 80, 120, 140, 170),
(21, 54, 77, 100, 140, 171),
(35, 82, 133, 142, 171, 174),
(14, 30, 83, 113, 125, 170),
(4, 29, 68, 120, 134, 173),
(1, 4, 52, 57, 86, 136, 152),
(26, 51, 56, 91, 122, 137, 168),
(52, 84, 110, 115, 145, 168),
(7, 50, 81, 99, 132, 173),
(23, 55, 67, 95, 172, 174),
(26, 41, 77, 109, 141, 148),
(2, 27, 41, 61, 62, 115, 133),
(27, 40, 56, 124, 125, 126),
(18, 49, 55, 124, 141, 167),
(6, 33, 85, 108, 116, 156),
(28, 48, 70, 85, 105, 129, 158),
(9, 54, 63, 131, 147, 155),
(22, 53, 68, 109, 121, 174),
(3, 13, 48, 78, 95, 123),
(31, 69, 133, 150, 155, 169),
(12, 43, 66, 89, 97, 135, 159),
(5, 39, 75, 102, 136, 167),
(2, 54, 86, 101, 135, 164),
(15, 56, 87, 108, 119, 171),
(10, 44, 82, 91, 111, 144, 149),
(23, 34, 71, 94, 127, 153),
(11, 49, 88, 92, 142, 157),
(29, 34, 87, 97, 147, 162),
(30, 50, 60, 86, 137, 142, 162),
(10, 53, 66, 84, 112, 128, 165),
(22, 57, 85, 93, 140, 159),
(28, 32, 72, 103, 132, 166),
(28, 29, 84, 88, 117, 143, 150),
(1, 26, 45, 80, 128, 147),
(17, 27, 89, 103, 116, 153),
(51, 57, 98, 163, 165, 172),
(21, 37, 73, 138, 152, 169),
(16, 47, 76, 130, 137, 154),
(3, 24, 30, 72, 104, 139),
(9, 40, 90, 106, 134, 151),
(15, 58, 60, 74, 111, 150, 163),
(18, 42, 79, 144, 146, 152),
(25, 38, 65, 99, 122, 160),
(17, 42, 75, 129, 170, 172),
)

411
bandsaunter/ft8wave.py Normal file
View file

@ -0,0 +1,411 @@
"""Finding FT8 in fifteen seconds of audio.
The coding is in ft8code; this is the part that has to deal with the air.
A slot of audio three kilohertz wide holds anything from nothing to forty
transmissions, each fifty hertz wide, each starting when its operator's
clock said to rather than when ours did, and most of them below the noise.
The shape of the search:
1. a waterfall -- the whole slot as power against time and frequency, at
half a symbol and half a tone, so nothing falls between two stools;
2. the sync search -- every place a Costas array could be, scored;
3. for each candidate, the eight tone powers of each of seventy-nine
symbols, turned into how strongly each bit is believed;
4. the error correction, which either converges or does not;
5. the checksum, which decides whether to believe it.
Steps four and five are why this works at all. Nothing up to step three is
clever enough to read a signal that is twenty decibels under the noise; what
it produces is a wash of barely-tilted opinions, and the code turns a
hundred and seventy-four of those into ninety-one bits of certainty -- or
into nothing, which is the other important answer.
"""
from __future__ import annotations
import math
from dataclasses import dataclass
import numpy as np
from . import ft8code as code
from .ft8code import COSTAS, SYMBOL_S, TONES
__all__ = ["Waterfall", "Candidate", "Decode", "listen_to", "waterfall",
"candidates", "soft_bits", "TIME_STEPS", "FREQ_STEPS"]
# Half a symbol and half a tone. A transmission that started a quarter of a
# symbol late, or sits a quarter of a tone off, is still caught: with steps
# a whole symbol wide the worst case loses most of the signal to the gap
# between two bins, and the worst case is common because nobody's clock and
# nobody's dial agree with ours.
TIME_STEPS = 2
FREQ_STEPS = 2
SYNC_AT = (0, 36, 72) # where the three Costas arrays sit, in symbols
# A perfectly-timed station does not start at the top of the slot. The
# transmission is 12.64 seconds long in a slot of fifteen, and the
# convention every FT8 program follows is to begin half a second in and to
# report lateness against that -- so a station whose clock is right reports
# as dt 0.0 rather than dt 0.5. Followed here so the figures mean the same
# thing as everybody else's.
NOMINAL_START = 0.5
@dataclass
class Waterfall:
"""The slot as power against time and frequency."""
power: np.ndarray # (times, bins), already in decibels
rate: float
hz_per_bin: float
seconds_per_step: float
@property
def times(self) -> int:
return self.power.shape[0]
@property
def bins(self) -> int:
return self.power.shape[1]
@dataclass
class Candidate:
"""Somewhere a transmission might start, and how much it looks like one."""
step: int # in half-symbols from the top of the slot
bin: int # in half-tones up the waterfall
score: float
def seconds(self, fall: Waterfall) -> float:
"""How late this started, against the nominal start of sending."""
return self.step * fall.seconds_per_step - NOMINAL_START
def hertz(self, fall: Waterfall) -> float:
return self.bin * fall.hz_per_bin
@dataclass
class Decode:
"""One transmission, read."""
text: str = ""
kind: str = ""
calls: tuple = ()
grid: str = ""
report: str = ""
calling: bool = False
hertz: float = 0.0
offset: float = 0.0 # how late it started, in seconds
snr_db: float = 0.0
score: float = 0.0
at: float = 0.0 # when the slot began, as a clock time
def waterfall(audio, rate: float) -> Waterfall:
"""The slot, as power against time and frequency.
One symbol of samples to a row, stepped half a symbol at a time, and
zero-padded to twice its length before the transform so the bins come
out half a tone apart. Padding rather than a longer window on purpose:
a longer window would average two symbols together, and the thing being
looked for changes every symbol.
"""
audio = np.asarray(audio, dtype=np.float32)
per_symbol = int(round(rate * SYMBOL_S))
if per_symbol < 8:
raise ValueError("the audio rate is too low to hold FT8 tones")
step = per_symbol // TIME_STEPS
size = per_symbol * FREQ_STEPS
window = np.hanning(per_symbol).astype(np.float32)
rows = 1 + max(0, (audio.size - per_symbol) // step)
if rows < 1:
raise ValueError("not enough audio for a single symbol")
# One strided view of the whole slot, transformed in a single call:
# a loop over two hundred rows in Python is most of the decode time.
frames = np.lib.stride_tricks.sliding_window_view(
audio, per_symbol)[::step][:rows]
spectrum = np.fft.rfft(frames * window, n=size, axis=1)
power = np.abs(spectrum).astype(np.float32)
# Decibels, because everything downstream adds and subtracts these and
# the useful comparisons are all ratios.
floor = max(float(np.median(power)) * 1e-4, 1e-12)
power = 20.0 * np.log10(np.maximum(power, floor))
return Waterfall(power=power, rate=float(rate),
hz_per_bin=rate / size,
seconds_per_step=step / rate)
def _sync_offsets():
"""Where the twenty-one sync tones sit, relative to a candidate."""
steps, bins = [], []
for block in SYNC_AT:
for k, tone in enumerate(COSTAS):
steps.append((block + k) * TIME_STEPS)
bins.append(tone * FREQ_STEPS)
return np.array(steps), np.array(bins)
SYNC_STEP, SYNC_BIN = _sync_offsets()
def candidates(fall: Waterfall, most: int = 300, lowest: float = 200.0,
highest: float = 3000.0, earliest: float = -2.5,
latest: float = 3.5) -> list[Candidate]:
"""Every place a transmission plausibly starts, best first.
Scored as how far the twenty-one sync tones stand above the other seven
tones at the same instants. Not above the noise floor of the band --
that would rank a strong signal's neighbours above a weak signal, and
the weak ones are the whole point of the protocol.
Only local peaks are kept. A real transmission lights up every
neighbouring offset as well, and without this one loud station would
fill the list and crowd out forty quiet ones.
"""
power = fall.power
span = TONES * TIME_STEPS
first = int(math.floor(earliest / fall.seconds_per_step))
last = int(math.ceil(latest / fall.seconds_per_step))
starts = np.arange(first, last + 1)
starts = starts[(starts >= 0) | (starts + SYNC_STEP.min() >= 0)]
starts = starts[starts + span <= power.shape[0]]
starts = starts[starts >= 0]
low = max(0, int(lowest / fall.hz_per_bin))
high = min(power.shape[1] - 8 * FREQ_STEPS,
int(highest / fall.hz_per_bin))
if starts.size == 0 or high <= low:
return []
tops = np.arange(low, high)
# The sync tones, and all eight tones at the same instants, for every
# (start, frequency) at once.
times = starts[:, None] + SYNC_STEP[None, :] # (S, 21)
here = power[times] # (S, 21, bins)
want = tops[None, None, :] + SYNC_BIN[None, :, None]
sync = np.take_along_axis(here, want, axis=2).mean(axis=1) # (S, tops)
whole = np.zeros_like(sync)
for tone in range(8):
idx = tops[None, None, :] + tone * FREQ_STEPS
whole += np.take_along_axis(here, idx, axis=2).mean(axis=1)
score = sync - whole / 8.0
# Local peaks only, in both directions.
keep = np.ones_like(score, dtype=bool)
keep[1:, :] &= score[1:, :] >= score[:-1, :]
keep[:-1, :] &= score[:-1, :] >= score[1:, :]
keep[:, 1:] &= score[:, 1:] >= score[:, :-1]
keep[:, :-1] &= score[:, :-1] >= score[:, 1:]
rows, cols = np.nonzero(keep)
values = score[rows, cols]
order = np.argsort(values)[::-1][:most]
return [Candidate(step=int(starts[rows[i]]), bin=int(tops[cols[i]]),
score=float(values[i])) for i in order]
def soft_bits(fall: Waterfall, spot: Candidate) -> np.ndarray:
"""How strongly each of the hundred and seventy-four bits is a zero.
Every symbol gives eight tone powers. A bit is a zero in four of the
eight tones and a one in the other four, so its evidence is the best of
the four against the best of the other four -- in decibels, which is
already a log-likelihood up to a scale factor, and the scale does not
matter to the min-sum decoding downstream.
Taking only the loudest tone and calling that three bits, which is the
obvious thing to do, throws away exactly the information the error
correction runs on, and decodes almost nothing.
"""
power = fall.power
steps = spot.step + np.arange(TONES) * TIME_STEPS
bins = spot.bin + np.arange(8) * FREQ_STEPS
if steps[-1] >= power.shape[0] or bins[-1] >= power.shape[1]:
return np.zeros(code.BITS, dtype=np.float32)
grid = power[np.ix_(steps, bins)] # (79, 8)
# Sync symbols carry nothing; drop them and put the tones back in the
# order the bits were in before the Gray mapping.
data = np.array([i for i in range(TONES)
if not (i < 7 or 36 <= i < 43 or 72 <= i)])
values = grid[data][:, list(code.GRAY)] # (58, 8) by value
out = np.zeros(code.BITS, dtype=np.float32)
for b in range(3):
mask = (np.arange(8) >> (2 - b)) & 1
zero = values[:, mask == 0].max(axis=1)
one = values[:, mask == 1].max(axis=1)
out[b::3] = zero - one
return out
# Worked out by measuring against transmissions of known strength and
# against off-air recordings with published readings, not derived. Turning
# a ratio of bins into the decibels-in-2500-Hz everybody quotes depends on
# the window, the padding and the shape of the noise, and calibrating it is
# both easier and more honest than deriving it and hoping.
SNR_TRIM = -37.0
# Making the transmitter above label its signals the way the reading below
# reports them. The reading is the one tied to reality -- it matches the
# published figures for the off-air recordings to within half a decibel
# across a hundred of them -- and a bare ratio of signal power to noise in
# 2500 Hz comes out eight decibels away from it, consistently enough that
# the difference is a definition rather than an error. Whose definition is
# "right" is not a question this can settle; what matters is that a signal
# built here at -15 reads as -15, so a test of sensitivity means what it
# says.
SNR_MATCHES = 10.0 ** (8.0 / 10.0)
SNR_NEAR_HZ = 200.0 # how far either side to look for the noise
SNR_PERCENTILE = 30 # low enough to ignore the neighbours
def _snr(fall: Waterfall, spot: Candidate, tones) -> float:
"""A signal-to-noise in the usual 2500 Hz reference bandwidth.
Measured in the tone that was actually sent, which is known by the time
this is called because the message decoded. Taking the loudest of the
eight instead looks like the same thing and is not: the largest of
eight noisy numbers is well above their mean even when there is no
signal at all, so weak transmissions all read as though they were
stronger than they are, and the scale compresses where it matters.
The noise is measured beside the signal rather than across the band. A
receiver with a three-kilohertz passband has nothing above it but the
noise floor of the sound card, and a median taken over the whole
spectrum sits twelve to sixteen decibels below the real one -- which is
a receiver flattering every station it hears by that much.
Good to a few decibels, honestly: the readings here track the published
ones for the same recordings with a correlation of about 0.83 and no
systematic bias between -25 and +10, which is the range that matters.
Very strong signals read a little low, as they do everywhere.
"""
power = (10.0 ** (fall.power / 20.0)) ** 2
steps = spot.step + np.arange(TONES) * TIME_STEPS
live = (steps >= 0) & (steps < power.shape[0])
steps, sent = steps[live], np.asarray(tones)[live]
bins = spot.bin + np.arange(8) * FREQ_STEPS
if steps.size == 0 or bins[-1] >= power.shape[1]:
return -99.0
near = int(SNR_NEAR_HZ / fall.hz_per_bin)
lo = max(0, spot.bin - near)
high = min(power.shape[1], bins[-1] + near)
columns = np.arange(lo, high)
# A guard either side of the signal: its own skirts are not noise.
beside = ((columns < spot.bin - 2 * FREQ_STEPS)
| (columns > bins[-1] + 2 * FREQ_STEPS))
if beside.sum() < 8:
return -99.0
noise = float(np.percentile(power[np.ix_(steps, columns[beside])],
SNR_PERCENTILE))
grid = power[np.ix_(steps, bins)]
here = float(grid[np.arange(grid.shape[0]), sent].mean())
if noise <= 0 or here <= noise:
return -99.0
return round(10.0 * math.log10((here - noise) / noise) + SNR_TRIM, 0)
def listen_to(audio, rate: float, book=None, most: int = 300,
rounds: int = 30, at: float = 0.0) -> list[Decode]:
"""Everything decodable in one slot of audio.
Candidates are decoded in one batch rather than one at a time: a busy
slot throws up two or three hundred of them, the error correction is
most of the work, and doing them together is the difference between a
slot that decodes in a second and one that does not keep up with the
air.
"""
book = book if book is not None else code.CallBook()
fall = waterfall(audio, rate)
spots = candidates(fall, most=most)
if not spots:
return []
beliefs = np.stack([soft_bits(fall, spot) for spot in spots])
repaired, worked = code.repair(beliefs, rounds=rounds)
out: list[Decode] = []
seen: dict[str, Decode] = {}
for i, spot in enumerate(spots):
if not worked[i]:
continue
bits = repaired[i]
if not code.crc_holds(bits[:code.PAYLOAD]):
continue
message = code.unpack(bits[:code.MESSAGE_BITS], book)
if not message.text:
continue
found = Decode(text=message.text, kind=message.kind,
calls=message.calls, grid=message.grid,
report=message.report, calling=message.calling,
hertz=round(spot.hertz(fall), 1),
offset=round(spot.seconds(fall), 2),
snr_db=_snr(fall, spot, code.tones_of(bits)),
score=spot.score, at=at)
# The same transmission is found at several neighbouring offsets.
# Keep the one with the best sync, which is the one nearest the
# truth about where and when it actually was.
was = seen.get(found.text)
if was is None or found.score > was.score:
seen[found.text] = found
out = sorted(seen.values(), key=lambda d: d.hertz)
for found in out:
for call in found.calls:
book.remember(call)
return out
def transmit(text: str, rate: float = 12000.0, hertz: float = 1000.0,
offset: float = 0.5, seconds: float = code.SLOT_S,
snr_db: float | None = None, seed: int = 0,
book=None) -> np.ndarray:
"""A slot of audio with one transmission in it.
Here rather than in a test because the tests need it, the simulator
needs it, and something that generates the signal this module claims to
understand is the only way to ask a question of the decoder whose answer
is known in advance -- where a transmission started, to the sample.
It does not transmit anything anywhere. It builds samples.
"""
payload = code.pack(text, book)
tones = code.tones_of(code.encode(code.with_crc(payload)))
per_symbol = int(round(rate * SYMBOL_S))
total = int(round(rate * seconds))
audio = np.zeros(total, dtype=np.float64)
# Continuous phase across the whole transmission. A tone that restarted
# its phase every symbol would splatter across the band, which is both
# unneighbourly on the air and not what a decoder should be tested with.
phase = 0.0
start = int(round(offset * rate))
for i, tone in enumerate(tones):
freq = hertz + float(tone) * code.SYMBOL_HZ
at = start + i * per_symbol
if at >= total or at + per_symbol <= 0:
phase += 2 * math.pi * freq * per_symbol / rate
continue
step = 2 * math.pi * freq / rate
angles = phase + step * np.arange(per_symbol)
phase = angles[-1] + step
first = max(0, at)
last = min(total, at + per_symbol)
audio[first:last] += np.sin(angles[first - at:last - at])
if snr_db is not None:
# The usual reference: the signal's own power against the noise in
# 2500 Hz, which is how every FT8 program quotes a report. The
# noise added here is white across the whole sampled band, so the
# part of it that lands in the reference 2500 Hz is that fraction
# of the total -- which is the step it is easy to get wrong, and
# getting it wrong makes a decoder look deaf when it is the test
# signal that was quietly attenuated.
rng = np.random.default_rng(seed)
signal = float(np.mean(audio[audio != 0] ** 2)) if audio.any() else 1.0
bandwidth = rate / 2.0
whole = (signal / (10.0 ** (snr_db / 10.0))
* (bandwidth / 2500.0) / SNR_MATCHES)
audio = audio + rng.standard_normal(total) * math.sqrt(whole)
return audio.astype(np.float32)

200
bandsaunter/tui.py
View file

@ -24,7 +24,8 @@ from .config import (DEFAULT_CONFIG_DIR, DEFAULT_CONFIG_PATH,
from .ranges import RangeError, ScanRange, parse_frequency from .ranges import RangeError, ScanRange, parse_frequency
__all__ = ["run_tui", "show_ranges", "settings_menu", "help_screen", __all__ = ["run_tui", "show_ranges", "settings_menu", "help_screen",
"aircraft_menu", "weather_menu", "aprs_menu", "first_run_setup", "aircraft_menu", "weather_menu", "aprs_menu", "ft8_menu",
"first_run_setup",
"TUIAbort"] "TUIAbort"]
_BACK = ("", "b", "back", "q", "quit", "x") _BACK = ("", "b", "back", "q", "quit", "x")
@ -1676,6 +1677,9 @@ def _main_loop(console: Console, cfg: ScanConfig) -> ScanConfig | None:
f" [cyan]7[/cyan] APRS (144 MHz packet) " f" [cyan]7[/cyan] APRS (144 MHz packet) "
f"[grey62]positions, weather and messages from amateurs" f"[grey62]positions, weather and messages from amateurs"
f"[/grey62]\n" f"[/grey62]\n"
f" [cyan]8[/cyan] FT8 "
f"[grey62]fifteen-second slots, whole bands at once, mostly "
f"under the noise[/grey62]\n"
f" [cyan]h[/cyan] Help\n" f" [cyan]h[/cyan] Help\n"
f" [cyan]s[/cyan] [bold green]Start scanning[/bold green]\n" f" [cyan]s[/cyan] [bold green]Start scanning[/bold green]\n"
f" [cyan]q[/cyan] Quit\n") f" [cyan]q[/cyan] Quit\n")
@ -1695,6 +1699,8 @@ def _main_loop(console: Console, cfg: ScanConfig) -> ScanConfig | None:
weather_menu(console, cfg) weather_menu(console, cfg)
elif choice == "7": elif choice == "7":
aprs_menu(console, cfg) aprs_menu(console, cfg)
elif choice == "8":
ft8_menu(console, cfg)
elif choice in ("h", "?", "help"): elif choice in ("h", "?", "help"):
help_screen(console) help_screen(console)
elif choice in ("s", "start", "go"): elif choice in ("s", "start", "go"):
@ -1706,3 +1712,195 @@ def _main_loop(console: Console, cfg: ScanConfig) -> ScanConfig | None:
return cfg return cfg
elif choice in ("q", "quit", "exit"): elif choice in ("q", "quit", "exit"):
return None return None
_FT8_INTRO = (
"[bold]FT8[/bold] is fifteen seconds of everybody at once. Every station "
"on the band transmits in the same quarter-minute slots, on the same "
"dial frequency, fifty hertz wide each, stacked across three kilohertz "
"of audio — so one receiver parked on one frequency hears the whole "
"band's worth of stations at the same time.\n\n"
"Most of them arrive [bold]below the noise[/bold], and decode anyway: "
"half of what is sent is error-correcting code, which is what buys a "
"mode that works twenty decibels under what an operator can hear.\n\n"
"Two things it needs. The [bold]clock[/bold] has to be right to a second "
"or two, because the slots are quarter-minutes of UTC and every station "
"on earth agrees about which one it is. And almost all the activity is "
"on [bold]shortwave[/bold], which a plain receiver of this kind cannot "
"reach without an upconverter or direct sampling — so the default here "
"is the two-metre channel, which it can.\n\n"
"Nothing here transmits. It listens, decodes and writes down."
)
def ft8_menu(console: Console, cfg: ScanConfig) -> None:
"""Listen to FT8, without a command line."""
from . import ft8
options = ft8.load_options()
while True:
_rule(console, "FT8")
console.print(Panel(Text.from_markup(_FT8_INTRO),
border_style="blue", padding=(0, 1)))
_option_groups(console, options, ft8)
logs = ft8.logs_in(cfg.output_dir)
kept = "no logs yet" if not logs else \
f"{len(logs)} log{'s' if len(logs) != 1 else ''}"
console.print(
f"\n [cyan]l[/cyan] [bold green]Listen[/bold green]"
f" [grey62]{ft8.describe(options)}[/grey62]\n"
# The band is the setting that decides whether anything is heard
# at all, and whether the receiver can reach it, so it sits here
# rather than a level down among the gain and the sample rate.
f" [cyan]c[/cyan] Band / channel "
f"[bold]{ft8.band_text(options)}[/bold]\n"
f" [cyan]g[/cyan] Where you are "
f"[grey62]{options.grid.upper() or 'not set — no distances'}"
f"[/grey62]\n"
f" [cyan]r[/cyan] Read a log back [grey62]{kept} in "
f"{cfg.output_dir}[/grey62]\n"
f" [cyan]N[/cyan] open group N "
f"[grey62]or type part of an option's name to find it[/grey62]\n"
f" [cyan]s[/cyan] Save these as default "
f"[grey62]kept in {ft8.options_path()}[/grey62]\n"
f" [cyan]d[/cyan] Reset them\n"
f" [cyan]b[/cyan] Back\n")
answer = _ask(console, " choice", "l").strip().lower()
if answer in _BACK:
return
if answer in ("l", "listen", "p"):
_ft8_listen(console, cfg, options)
elif answer in ("c", "channel", "band"):
_ft8_band(console, options)
elif answer in ("g", "grid", "where"):
_ft8_grid(console, options)
elif answer in ("r", "read", "m"):
_ft8_read(console, cfg, options, logs)
elif answer == "s":
try:
where = ft8.save_options(options)
console.print(f" [green]saved to {where}[/green]")
except OSError as exc:
console.print(f" [red]could not save: {exc}[/red]")
elif answer == "d":
if _confirm(" reset every FT8 option"):
options = ft8.Ft8Options()
console.print(" [green]reset[/green]")
elif answer.isdigit() and 1 <= int(answer) <= len(ft8.OPTION_GROUPS):
_option_group_menu(console, options,
ft8.OPTION_GROUPS[int(answer) - 1], ft8)
elif answer.lstrip("?").strip().isdigit():
_edit_option(console, options, answer, ft8)
elif answer:
found = _find_options(answer, ft8)
if not found:
console.print(f" [yellow]nothing matches {answer!r} — "
f"enter a group number, or l, c, g, r, s, d "
f"or b[/yellow]")
elif len(found) == 1:
_edit_option(console, options,
str(ft8.OPTIONS.index(found[0]) + 1), ft8)
else:
_option_list(console, options, found, f"matching {answer!r}",
ft8)
_pick_option(console, options, ft8)
def _ft8_listen(console: Console, cfg: ScanConfig, options) -> None:
from . import ft8
errs = options.validate()
if errs:
for e in errs:
console.print(f" [red]{e}[/red]")
return
try:
ft8.listen(console, options, cfg.output_dir)
except Exception as exc: # a menu must survive it
console.print(f" [red]{exc}[/red]")
def _ft8_band(console: Console, options) -> None:
"""Choose the band, which sets the dial frequency with it.
Which band is also the question of whether the receiver can hear it at
all, so the list says so rather than leaving somebody to find out by
listening to silence for ten minutes.
"""
from . import ft8
console.print("\n [bold]Which band[/bold]\n")
for i, (name, hz, needs) in enumerate(ft8.BANDS, 1):
note = ("[yellow]needs an upconverter or direct sampling[/yellow]"
if needs == "shortwave"
else "[green]a plain receiver reaches this[/green]")
here = " [green]<- now[/green]" if options.band == name else ""
console.print(f" [cyan]{i:>2}[/cyan] {name:<5} "
f"{hz / 1e6:>10.3f} MHz {note}{here}")
console.print("\n [grey62]Almost all the activity is on shortwave, and "
"20m is the busiest band on earth. Two metres is quiet by "
"comparison and is what a plain dongle can reach.[/grey62]")
answer = _ask(console, "\n band (number, name, or blank to keep)",
"").strip().lower()
if not answer:
return
if answer.isdigit() and 1 <= int(answer) <= len(ft8.BANDS):
ft8.use_band(options, ft8.BANDS[int(answer) - 1][0])
elif any(answer == name for name, _hz, _n in ft8.BANDS):
ft8.use_band(options, answer)
else:
console.print(f" [yellow]no band called {answer!r}[/yellow]")
return
console.print(f" [green]{ft8.band_text(options)}[/green]")
def _ft8_grid(console: Console, options) -> None:
"""Set the grid square, which is what distances are measured from."""
from . import ft8
console.print("\n [grey62]Your Maidenhead grid square: four characters "
"like IO91 or FN31. Every station calling CQ says where it "
"is this way, so with yours filled in the report can say "
"how far each of them is and on what bearing — which on "
"shortwave is the whole interest of the thing.[/grey62]")
answer = _ask(console, "\n grid square (blank to clear)",
options.grid).strip()
if not answer:
options.grid = ""
console.print(" [green]cleared — no distances[/green]")
return
if ft8.grid_at(answer) is None:
console.print(f" [yellow]{answer!r} is not a grid square[/yellow]")
return
options.grid = answer.upper()
lat, lon = ft8.grid_at(options.grid)
console.print(f" [green]{options.grid} — {lat:.2f},{lon:.2f}[/green]")
def _ft8_read(console: Console, cfg: ScanConfig, options, logs) -> None:
"""Read an evening's decodes back off the disk."""
from . import ft8, ft8log
if not logs:
console.print(" [yellow]no FT8 logs yet — listen first[/yellow]")
return
console.print("\n [bold]Which log[/bold]\n")
for i, path in enumerate(logs[-20:], 1):
console.print(f" [cyan]{i:>2}[/cyan] {path.name}")
answer = _ask(console, "\n log (number, or blank to go back)",
"").strip()
if not answer.isdigit():
return
picked = logs[-20:][int(answer) - 1] if 1 <= int(answer) <= len(logs[-20:]) \
else None
if picked is None:
return
rows = ft8log.read_log(picked)
console.print(f"\n [grey62]{len(rows)} decodes in {picked.name}"
f"[/grey62]\n")
for row in rows[:40]:
console.print(f" {row['snr_db']:>4.0f} {row['offset']:>5.1f} "
f"{row['hertz']:>7.1f} ~ {row['text']}")
if len(rows) > 40:
console.print(f" [grey62]…and {len(rows) - 40} more[/grey62]")

115
bandsaunter/ui.py
View file

@ -1071,3 +1071,118 @@ class AprsDisplay:
if not out.plain: if not out.plain:
out.append(station.kind, style="grey62") out.append(station.kind, style="grey62")
return out return out
class Ft8Display:
"""The last slot's decodes, above a table of who has been heard.
Two things at once because FT8 is two things at once. A slot is an
event -- forty stations transmitted, here is what they said -- and the
evening is an accumulation. A display that showed only the latest slot
would throw away the band; one that showed only the running table would
never show the thing that just happened.
"""
def __init__(self, console: Console, hold: float = 3600.0,
imperial: bool = False, grid: str = "", band: str = ""):
self.console = console
self.hold = hold
self.imperial = imperial
self.grid = grid
self.band = band
self.started = time.time()
self.heard = None
self.slot_at = 0.0
def update(self, heard, slot_at: float = 0.0) -> None:
self.heard = heard
if slot_at:
self.slot_at = slot_at
def showing(self, now: float | None = None) -> list:
if self.heard is None:
return []
now = time.time() if now is None else now
return [s for s in self.heard.band.all()
if now - s.last <= self.hold]
def render(self, now: float | None = None, width: int | None = None):
now = time.time() if now is None else now
parts = [self._header(now)]
latest = list(getattr(self.heard, "latest", []) or [])
if latest:
parts.append(self._slot(latest))
here = self.showing(now)
if here:
parts.append(self._table(here, now))
elif not latest:
parts.append(Panel(_one_line(
"[grey62]nothing decoded yet — the first decodes arrive "
"when the slot after next ends, up to thirty seconds away. "
"If nothing comes at all, check the clock: FT8 needs it "
"right to a second or two[/grey62]"),
border_style="grey37", padding=(0, 1)))
return Group(*parts)
def _header(self, now: float):
heard = self.heard
slots = getattr(heard, "slots", 0)
decodes = getattr(heard, "decodes", 0)
stations = len(getattr(heard, "band", None).stations) if heard else 0
elapsed = max(1e-9, now - self.started)
bits = [f"[bold]FT8[/bold] {self.band}",
f"{slots} slots",
f"{decodes} decodes",
f"{stations} stations"]
if slots:
bits.append(f"{decodes / max(1, slots):.1f} a slot")
bits.append(_dur(elapsed))
if self.grid:
bits.append(f"from {self.grid.upper()}")
return Text.from_markup(" ".join(f"[grey62]{b}[/grey62]"
if i else b
for i, b in enumerate(bits)))
def _slot(self, latest):
table = Table(box=None, header_style="bold", pad_edge=False,
title=f"last slot — {len(latest)} decoded",
title_justify="left", title_style="bold")
table.add_column("snr", justify="right")
table.add_column("dt", justify="right")
table.add_column("hz", justify="right")
table.add_column("message")
for d in latest[:24]:
table.add_row(f"{d.snr_db:.0f}", f"{d.offset:+.1f}",
f"{d.hertz:.0f}",
f"[bold]{d.text}[/bold]" if d.calling else d.text)
return table
def _table(self, here, now: float):
from .ft8 import grid_away
here = sorted(here, key=lambda s: (-s.last, s.call))
table = Table(box=None, header_style="bold", pad_edge=False,
title=f"heard so far — {len(here)} stations",
title_justify="left", title_style="bold")
table.add_column("station")
table.add_column("grid")
if self.grid:
table.add_column("away", justify="right")
table.add_column("n", justify="right")
table.add_column("best", justify="right")
table.add_column("hz", justify="right")
table.add_column("last", justify="right")
for s in here[:20]:
row = [s.call, s.grid or "—"]
if self.grid:
away = grid_away(self.grid, s.grid) if s.grid else None
if away is None:
row.append("—")
elif self.imperial:
row.append(f"{away[0] * 0.621371:.0f} mi")
else:
row.append(f"{away[0]:.0f} km")
row += [str(s.decodes), f"{s.best_snr:.0f}", f"{s.hertz:.0f}",
f"{now - s.last:.0f}s"]
table.add_row(*row)
return table

202
packaging/bandsaunter.1
View file

@ -1,5 +1,5 @@
.\" Generated by packaging/make-man.py -- do not edit by hand. .\" Generated by packaging/make-man.py -- do not edit by hand.
.TH BANDSAUNTER 1 "2026-09-21" "bandsaunter 2026-09-21_03" "User Commands" .TH BANDSAUNTER 1 "2026-09-21" "bandsaunter 2026-09-21_04" "User Commands"
.SH NAME .SH NAME
bandsaunter \- scan, record and identify radio signals with an RTL-SDR bandsaunter \- scan, record and identify radio signals with an RTL-SDR
.SH SYNOPSIS .SH SYNOPSIS
@ -2958,6 +2958,196 @@ Also write a map \[em] write the stations and their tracks for Google Earth.
.br .br
Setting name \fBkml\fR, default \fBno\fR. Setting name \fBkml\fR, default \fBno\fR.
.PP .PP
.SH FT8
Fifteen seconds of everybody at once. Every station on the band transmits in
the same quarter-minute slots, on the same dial frequency, fifty hertz wide
each, stacked across three kilohertz of audio \[em] so one receiver parked on
one frequency hears the whole band's worth of stations at the same time, and
hears most of them well below the noise.
.PP
Half of what is transmitted is error-correcting code, and that is the trick:
it is what buys a mode that decodes twenty-odd decibels under what an operator
can hear. A receiver that took the loudest tone of each symbol and hoped would
decode almost nothing, which is why the tone detector reports how confident it
is bit by bit rather than what it thinks it heard.
.SS What it needs
The clock has to be right to a second or two. The slots are quarter-minutes of
UTC and every station on earth agrees about which one it is. A receiver a
second out still decodes; one a slot out hears every transmission split across
two captures and decodes none of them. This is the one failure that looks
exactly like a dead band, so the display and the report both say so when
nothing arrives.
.PP
Almost all the activity is on shortwave, which a plain receiver of this kind
cannot reach. The default is therefore the two-metre channel at 144.174 MHz,
which it can. All thirteen channels are in the list and the shortwave ones
work through an upconverter or a receiver in direct sampling mode; the menu
says which is which rather than leaving somebody to find out by listening to
silence.
.SS What comes out
.BI \-\-grid " SQUARE"
is what turns decodes into geography: every station calling CQ says where it
is, so with your own square filled in each gets a distance and a bearing and
the furthest heard is named. Three tables afterwards \[em] stations heard,
calling CQ, and who was working whom.
.PP
.B \-\-adif
writes the log again in the form every amateur logging program imports,
marked as heard rather than worked. Nothing here transmits, so nothing here is
a contact, and an ADIF that let a logging program treat these as worked would
put claims into somebody's log that they cannot make.
.SS How well it works
Checked against eleven off-air recordings with published decodes, which is the
only honest way to test a decoder: an encoder tested against its own decoder
agrees with it about anything they are both wrong about. Ninety-seven of a
hundred and fifty messages, with no false decodes; timing within a hundredth
of a second, frequency within a hertz, signal reports within half a decibel on
average. The third not decoded are the weakest in each slot: a mature decoder
subtracts what it has decoded and looks again in the remainder, which is not
built here.
.SH FT8 OPTIONS
Every option the FT8 side takes, in the five groups the menu shows them in.
Each is a flag here and a line in the menu, and both come from one table in
the program, so they cannot disagree.
.SS Receiver
.TP
.B --device
Receiver \[em] which receiver to use, when more than one is plugged in.
.br
Setting name \fBdevice\fR, default \fB0\fR.
.br
Accepts: at least 0.
.TP
.B --gain
Gain \[em] tuner gain in dB, or automatic.
.br
Setting name \fBgain\fR, default \fBauto\fR.
.TP
.B --rate
Sample rate \[em] how fast to sample; 96 kS/s is the least that holds the channel (Hz).
.br
Setting name \fBrate\fR, default \fB240 kHz\fR.
.br
Accepts: at least 96000.
.TP
.B --band
Band \[em] which FT8 channel to listen on.
.br
Setting name \fBband\fR, default \fB2m\fR.
.br
Accepts: one of: 160m, 80m, 60m, 40m, 30m, 20m, 17m, 15m, 12m, 10m, 6m, 2m, 70cm.
.TP
.B --frequency --freq
Dial frequency \[em] the dial frequency, if you want one the band list does not have (Hz).
.br
Setting name \fBfrequency\fR, default \fB144.174 MHz\fR.
.br
Accepts: at least 100000, at most 2e+09.
.TP
.B --simulate / --no-simulate
Simulate \[em] invent a band instead of using a receiver.
.br
Setting name \fBsimulate\fR, default \fBno\fR.
.PP
.SS Listening
.TP
.B --seconds
Listen for \[em] how long to listen; zero means until stopped (s).
.br
Setting name \fBseconds\fR, default \fBuntil stopped\fR.
.br
Accepts: at least 0.
.TP
.B --slots
Or this many slots \[em] stop after this many fifteen-second slots; zero means no limit.
.br
Setting name \fBslots\fR, default \fBno limit\fR.
.br
Accepts: at least 0.
.TP
.B --log / --no-log
Write a log \[em] keep every decode in a file.
.br
Setting name \fBlog\fR, default \fByes\fR.
.TP
.B --decodes-seen / --no-decodes-seen
Print each decode \[em] a line per decode instead of a table that refreshes.
.br
Setting name \fBdecodes_seen\fR, default \fBno\fR.
.TP
.B --hold
Keep on display for \[em] how long a station stays on the table after its last decode (s).
.br
Setting name \fBhold\fR, default \fB3600 s\fR.
.br
Accepts: at least 1.
.PP
.SS Decoding
.TP
.B --lowest
Search from \[em] the lowest audio frequency to look for signals at (Hz).
.br
Setting name \fBlowest\fR, default \fB200 Hz\fR.
.br
Accepts: at least 0.
.TP
.B --highest
Search to \[em] the highest audio frequency to look for signals at (Hz).
.br
Setting name \fBhighest\fR, default \fB3 kHz\fR.
.br
Accepts: at least 100.
.TP
.B --most
Candidates per slot \[em] how many possible transmissions to try to decode each slot.
.br
Setting name \fBmost\fR, default \fB300\fR.
.br
Accepts: at least 1.
.TP
.B --rounds
Repair passes \[em] how many passes of error correction to make before giving up.
.br
Setting name \fBrounds\fR, default \fB30\fR.
.br
Accepts: at least 1.
.PP
.SS Showing
.TP
.B --grid
Aerial at \[em] your own grid square, so distances can be worked out.
.br
Setting name \fBgrid\fR, default \fBnot set, so no distances\fR.
.TP
.B --units
Show readings in \[em] metric or imperial, for the display and the export.
.br
Setting name \fBunits\fR, default \fBmetric\fR.
.br
Accepts: one of: metric, imperial.
.TP
.B --calls-only / --no-calls-only
Callsigns only \[em] leave out free text and telemetry.
.br
Setting name \fBcalls_only\fR, default \fBno\fR.
.PP
.SS Afterwards
.TP
.B --report / --no-report
Report at the end \[em] print the tables when the listening stops.
.br
Setting name \fBreport\fR, default \fByes\fR.
.TP
.B --csv / --no-csv
Also write a CSV \[em] a spreadsheet of every decode beside the log.
.br
Setting name \fBcsv\fR, default \fBno\fR.
.TP
.B --adif / --no-adif
Also write an ADIF \[em] the log again, in the form logging programs read.
.br
Setting name \fBadif\fR, default \fBno\fR.
.PP
.SH FILES .SH FILES
.TP .TP
.I ~/.config/bandsaunter/config.yaml .I ~/.config/bandsaunter/config.yaml
@ -3090,6 +3280,16 @@ terms, in
or at or at
.UR https://www.gnu.org/licenses/ .UR https://www.gnu.org/licenses/
.UE . .UE .
.PP
One file is not original work.
.I ft8tables.py
holds the two fixed tables that define the FT8 error-correcting code, taken
from ft8_lib (https://github.com/kgoba/ft8_lib), MIT licensed, copyright 2018
K\[u0101]rlis Goba, which took them in turn from WSJT\-X. They are reproduced
under the MIT terms and that file carries the attribution they ask for. They
are there because they cannot be derived: everything else about FT8 here is
worked out from first principles, but those two tables are the code itself,
chosen once by its designers and published. No decoding logic was taken.
.SH BUGS .SH BUGS
The DVB-T television driver claims these dongles on sight. If the receiver The DVB-T television driver claims these dongles on sight. If the receiver
cannot be opened, that is almost always why: the package blacklists the cannot be opened, that is almost always why: the package blacklists the

70
packaging/make-man.py
View file

@ -77,6 +77,13 @@ def aprs_section() -> list[str]:
return options_section(ap) return options_section(ap)
def ft8_section() -> list[str]:
"""Every FT8 option, from the same table again."""
from bandsaunter import ft8
return options_section(ft8)
def options_section(air) -> list[str]: def options_section(air) -> list[str]:
"""One section's options, written out from the table the program uses. """One section's options, written out from the table the program uses.
@ -1792,6 +1799,58 @@ Every option the APRS side takes, in the four groups the menu shows them in.
Each is a flag here and a line in the menu, and both come from one table in Each is a flag here and a line in the menu, and both come from one table in
the program, so they cannot disagree. the program, so they cannot disagree.
.APRS_OPTIONS_HERE .APRS_OPTIONS_HERE
.SH FT8
Fifteen seconds of everybody at once. Every station on the band transmits in
the same quarter-minute slots, on the same dial frequency, fifty hertz wide
each, stacked across three kilohertz of audio \[em] so one receiver parked on
one frequency hears the whole band's worth of stations at the same time, and
hears most of them well below the noise.
.PP
Half of what is transmitted is error-correcting code, and that is the trick:
it is what buys a mode that decodes twenty-odd decibels under what an operator
can hear. A receiver that took the loudest tone of each symbol and hoped would
decode almost nothing, which is why the tone detector reports how confident it
is bit by bit rather than what it thinks it heard.
.SS What it needs
The clock has to be right to a second or two. The slots are quarter-minutes of
UTC and every station on earth agrees about which one it is. A receiver a
second out still decodes; one a slot out hears every transmission split across
two captures and decodes none of them. This is the one failure that looks
exactly like a dead band, so the display and the report both say so when
nothing arrives.
.PP
Almost all the activity is on shortwave, which a plain receiver of this kind
cannot reach. The default is therefore the two-metre channel at 144.174 MHz,
which it can. All thirteen channels are in the list and the shortwave ones
work through an upconverter or a receiver in direct sampling mode; the menu
says which is which rather than leaving somebody to find out by listening to
silence.
.SS What comes out
.BI \-\-grid " SQUARE"
is what turns decodes into geography: every station calling CQ says where it
is, so with your own square filled in each gets a distance and a bearing and
the furthest heard is named. Three tables afterwards \[em] stations heard,
calling CQ, and who was working whom.
.PP
.B \-\-adif
writes the log again in the form every amateur logging program imports,
marked as heard rather than worked. Nothing here transmits, so nothing here is
a contact, and an ADIF that let a logging program treat these as worked would
put claims into somebody's log that they cannot make.
.SS How well it works
Checked against eleven off-air recordings with published decodes, which is the
only honest way to test a decoder: an encoder tested against its own decoder
agrees with it about anything they are both wrong about. Ninety-seven of a
hundred and fifty messages, with no false decodes; timing within a hundredth
of a second, frequency within a hertz, signal reports within half a decibel on
average. The third not decoded are the weakest in each slot: a mature decoder
subtracts what it has decoded and looks again in the remainder, which is not
built here.
.SH FT8 OPTIONS
Every option the FT8 side takes, in the five groups the menu shows them in.
Each is a flag here and a line in the menu, and both come from one table in
the program, so they cannot disagree.
.FT8_OPTIONS_HERE
.SH FILES .SH FILES
.TP .TP
.I ~/.config/bandsaunter/config.yaml .I ~/.config/bandsaunter/config.yaml
@ -1924,6 +1983,16 @@ terms, in
or at or at
.UR https://www.gnu.org/licenses/ .UR https://www.gnu.org/licenses/
.UE . .UE .
.PP
One file is not original work.
.I ft8tables.py
holds the two fixed tables that define the FT8 error-correcting code, taken
from ft8_lib (https://github.com/kgoba/ft8_lib), MIT licensed, copyright 2018
K\[u0101]rlis Goba, which took them in turn from WSJT\-X. They are reproduced
under the MIT terms and that file carries the attribution they ask for. They
are there because they cannot be derived: everything else about FT8 here is
worked out from first principles, but those two tables are the code itself,
chosen once by its designers and published. No decoding logic was taken.
.SH BUGS .SH BUGS
The DVB-T television driver claims these dongles on sight. If the receiver The DVB-T television driver claims these dongles on sight. If the receiver
cannot be opened, that is almost always why: the package blacklists the cannot be opened, that is almost always why: the package blacklists the
@ -1944,6 +2013,7 @@ def main() -> int:
text = text.replace(".WEATHER_OPTIONS_HERE", text = text.replace(".WEATHER_OPTIONS_HERE",
"\n".join(weather_section())) "\n".join(weather_section()))
text = text.replace(".APRS_OPTIONS_HERE", "\n".join(aprs_section())) text = text.replace(".APRS_OPTIONS_HERE", "\n".join(aprs_section()))
text = text.replace(".FT8_OPTIONS_HERE", "\n".join(ft8_section()))
text = text.replace("\n\n", "\n") # troff dislikes blank lines text = text.replace("\n\n", "\n") # troff dislikes blank lines
target = Path(sys.argv[1] if len(sys.argv) > 1 target = Path(sys.argv[1] if len(sys.argv) > 1
else Path(__file__).parent / "bandsaunter.1") else Path(__file__).parent / "bandsaunter.1")

839
tests/test_ft8.py Normal file
View file

@ -0,0 +1,839 @@
"""The FT8 section: the code, the radio, the options and the front ends.
The important thing about this file is where its truth comes from. An
encoder tested against its own decoder agrees with it about anything they
are both wrong about -- this project learned that expensively on 433 MHz --
so the decoding here is checked against signals that came off the air and
were decoded by somebody else's program, not against anything written here.
Those recordings live outside the repository, and the tests that use them
skip when they are absent. What does not skip is REAL: whole codewords
lifted off the air, which pin the checksum, the parity and every field of
the message format to reality in forty-four characters apiece.
"""
import math
import time
import wave
from pathlib import Path
import numpy as np
import pytest
from bandsaunter import ft8, ft8code as code, ft8log, ft8sim, ft8wave
from bandsaunter import ft8tables as tables
REFERENCE = Path("/mnt/global/bandsaunter/ft8ref")
needs_recordings = pytest.mark.skipif(
not REFERENCE.is_dir(), reason="the off-air recordings are not here")
# Whole codewords, off the air, with what they say. A hundred and
# seventy-four bits each: seventy-seven of message, fourteen of checksum and
# eighty-three of parity, so one of these exercises the lot.
REAL = [
("B79A7EA67655CA92DB4D342FDD55B071779A631CFB4C",
"PA3EPP SP8NFO KN09", "standard"),
("0000002046E853111D48153FD733DA62C110A5E053FC",
"CQ F4FSY JN25", "standard"),
("E20AFC7590F9DA9FA7C9EEEA8A1A96D06F6855C82B58",
"VK4BLE OH1EDK -20", "standard"),
("70D84EAC55E5FB9FA70CD7148B9180DD1CD36A7EFC6C",
"ET3RFG/R IN3ADG -23", "standard"),
("1E6191F629601212F10D6992973A9B5AEA0596877498",
"2M0OGG RA6ABO KN96", "standard"),
("70AFEC9338AD521FA50DE077F5C94CAD362B2CC3A364",
"ES5GI DD3SF 73", "standard"),
("00000024519BF311248987ED8D2C4AA596CE262E1E68",
"CQ IK4LZH JN54", "standard"),
("4B9ACCF4519BF31FAA4F0C50A7B6CEA7F9C51BED8E80",
"9A9TT IK4LZH -10", "standard"),
("0B13C7046826781FA50F539DFE6D9FAF63F7C21EC268",
"R2EA IZ4OUL 73", "standard"),
("CBEC04F45AC9D49FA509770E71FEDE65EE2EFE66E8A8",
"SA5QED IQ5PJ 73", "standard"),
]
def bits_of_hex(text: str) -> np.ndarray:
raw = np.unpackbits(np.frombuffer(bytes.fromhex(text), dtype=np.uint8))
return raw[:code.BITS].astype(np.uint8)
# ---------------------------------------------------------------------------
# The two tables the code is defined by
# ---------------------------------------------------------------------------
def test_the_generator_and_the_parity_matrix_describe_the_same_code():
"""The two published tables are not derivable from one another -- the
generator's parity half runs to fifty-odd bits a row and the sparse one
to six or seven -- so that they agree is checked rather than assumed.
If they disagreed, every codeword this builds would fail every check
and nothing would ever decode, which is a long way to find out."""
rng = np.random.default_rng(11)
for _ in range(50):
message = rng.integers(0, 2, code.PAYLOAD).astype(np.uint8)
assert code.parity_holds(code.encode(message))
def test_every_codeword_bit_is_checked_three_times():
"""A regular column weight, which is what the message passing assumes
when it gathers three slots per bit instead of scattering."""
assert code.SLOTS_OF_BIT.shape == (code.BITS, 3)
assert sorted({len(c) for c in tables.CHECKS}) == [6, 7]
def test_a_broken_codeword_fails_its_parity():
rng = np.random.default_rng(3)
word = code.encode(rng.integers(0, 2, code.PAYLOAD).astype(np.uint8))
for flip in (0, 40, 90, 173):
bad = word.copy()
bad[flip] ^= 1
assert not code.parity_holds(bad)
# ---------------------------------------------------------------------------
# Reality
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("packed,text,kind", REAL)
def test_a_codeword_off_the_air_says_what_it_said(packed, text, kind):
"""The whole chain against something nobody here made up."""
word = bits_of_hex(packed)
assert code.parity_holds(word), "the parity of a real transmission"
assert code.crc_holds(word[:code.PAYLOAD]), "the checksum of a real one"
message = code.unpack(word[:code.MESSAGE_BITS])
assert message.text == text
assert message.kind == kind
def test_a_real_codeword_is_repaired_from_a_damaged_copy():
"""What the error correction is for, on a real transmission rather than
a made-up one.
Ten of the hundred and seventy-four bits, each of them *confidently*
wrong, which is the hard case: a bit the detector is unsure about costs
the decoding far less than one it is certain about and mistaken. Real
errors arrive uncertain, which is why the same code reads signals off
the air with far more than ten bits astray."""
word = bits_of_hex(REAL[0][0])
rng = np.random.default_rng(5)
belief = np.where(word == 1, -4.0, 4.0).astype(np.float32)
for flip in rng.choice(code.BITS, 10, replace=False):
belief[flip] = -belief[flip]
got = code.repair(belief)
assert got is not None and (got == word).all()
assert code.unpack(got[:code.MESSAGE_BITS]).text == REAL[0][1]
def test_noise_does_not_decode_into_a_message():
"""The checksum's job. The error correction will converge on *a*
codeword given enough noise, and the fourteen bits are what stop that
becoming a line in somebody's log."""
rng = np.random.default_rng(17)
lied = 0
for _ in range(200):
belief = rng.standard_normal(code.BITS).astype(np.float32) * 2.0
got = code.repair(belief)
if got is not None and code.crc_holds(got[:code.PAYLOAD]):
lied += 1
# One in sixteen thousand gets through by arithmetic; two hundred tries
# should see none, and seeing several would mean the CRC is not being
# checked at all.
assert lied == 0, f"{lied} runs of noise passed the checksum"
# ---------------------------------------------------------------------------
# The checksum and the coding
# ---------------------------------------------------------------------------
def test_the_checksum_covers_the_message_zero_extended():
"""Eighty-two bits, not seventy-seven -- which is not the same answer
and is the sort of thing that decodes nothing at all."""
payload = np.zeros(code.MESSAGE_BITS, dtype=np.uint8)
payload[3] = payload[70] = 1
block = code.with_crc(payload)
assert block.size == code.PAYLOAD
assert code.crc_holds(block)
broken = block.copy()
broken[10] ^= 1
assert not code.crc_holds(broken)
def test_the_tones_carry_the_bits_and_the_sync_is_where_it_should_be():
rng = np.random.default_rng(2)
word = code.encode(rng.integers(0, 2, code.PAYLOAD).astype(np.uint8))
tones = code.tones_of(word)
assert tones.size == code.TONES
assert (code.bits_of(tones) == word).all()
for start in (0, 36, 72):
assert list(tones[start:start + 7]) == list(tables.COSTAS)
def test_a_tone_mistaken_for_its_neighbour_costs_one_bit():
"""Which is the whole reason for the Gray mapping, and is worth a test
because getting the map backwards still round-trips perfectly."""
for tone in range(7):
a = code.UNGRAY[tone]
b = code.UNGRAY[tone + 1]
assert bin(int(a) ^ int(b)).count("1") == 1
# ---------------------------------------------------------------------------
# Messages
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("text", [
"CQ DL1UDO JO31", "CQ DX Z33Z KN11", "CQ JA OH1LWZ KP11",
"VK4BLE OH8JK R-17", "JR5MJS OH8NW 73", "LZ1CWK DC8VA RR73",
"G4CUS SP4FCA +10", "2M0OGG RA6ABO KN96", "YO7CGS A41ZZ -11",
"R2EA IZ4OUL R-08", "W2XYZ K1ABC RRR",
])
def test_a_standard_message_survives_the_whole_chain(text):
payload = code.pack(text)
word = code.encode(code.with_crc(payload))
assert code.parity_holds(word)
back = code.bits_of(code.tones_of(word))
assert (back == word).all()
got = code.repair(np.where(word == 1, -4.0, 4.0).astype(np.float32))
assert got is not None and (got == word).all()
assert code.unpack(got[:code.MESSAGE_BITS]).text == text
def test_the_digit_of_a_callsign_is_found_rather_than_guessed_at():
"""Z33Z has a digit in the second place that belongs to the prefix, so
a packer that assumes the second character is the separating digit
turns it into a hash and loses the callsign."""
assert code.is_standard_call("Z33Z")
assert code.is_standard_call("K1ABC")
assert code.is_standard_call("DL1UDO")
assert code.is_standard_call("2M0OGG")
assert not code.is_standard_call("ET3RFG/R")
assert not code.is_standard_call("")
def test_free_text_is_not_dressed_up_as_two_callsigns():
"""A packer that falls back to hashing any word it does not recognise
always succeeds, and would send HELLO WORLD as two hashed callsigns."""
got = code.unpack(code.pack("HELLO WORLD"))
assert got.kind == "free text"
assert got.text == "HELLO WORLD"
def test_free_text_is_thirteen_characters_and_says_so_by_truncating():
got = code.unpack(code.pack("TNX FER QSO 73"))
assert got.kind == "free text"
assert len(got.text) <= 13
@pytest.mark.parametrize("text,grid,report,calling", [
("CQ DL1UDO JO31", "JO31", "", True),
("VK4BLE OH8JK R-17", "", "R-17", False),
("JR5MJS OH8NW 73", "", "73", False),
("W2XYZ K1ABC RRR", "", "RRR", False),
])
def test_what_can_be_picked_out_of_a_message(text, grid, report, calling):
got = code.unpack(code.pack(text))
assert got.grid == grid and got.report == report
assert got.calling is calling
def test_a_hashed_callsign_stays_unknown_until_it_has_been_heard_in_full():
"""A wrong callsign in a log is worse than a missing one, so a hash
that has not been heard spelled out stays <...> rather than guessed."""
book = code.CallBook()
assert book.look_up(code.hash_of("GM4ABC", 22), 22) == "<...>"
book.remember("GM4ABC")
assert book.look_up(code.hash_of("GM4ABC", 22), 22) == "GM4ABC"
assert book.look_up(code.hash_of("GM4ABD", 22), 22) == "<...>"
def test_a_callsign_too_short_to_mean_anything_is_not_remembered():
book = code.CallBook()
for rubbish in ("", "A", "CQ", "DE", "QRZ"):
book.remember(rubbish)
assert book.by_hash == {}
# ---------------------------------------------------------------------------
# The radio
# ---------------------------------------------------------------------------
def test_a_transmission_is_found_where_it_was_put():
"""The decoder's own clock and dial, against a signal whose time and
frequency are known to the sample."""
for hertz in (500.0, 1200.0, 2400.0):
for offset in (0.2, 0.5, 1.4):
audio = ft8wave.transmit("CQ DL1UDO JO31", hertz=hertz,
offset=offset)
got = ft8wave.listen_to(audio, 12000.0)
assert len(got) == 1, (hertz, offset)
assert got[0].text == "CQ DL1UDO JO31"
assert abs(got[0].hertz - hertz) < 3.5
# Reported against the nominal start of sending, half a second
# into the slot, which is what every FT8 program reports.
assert abs(got[0].offset - (offset - 0.5)) < 0.12
def test_the_nominal_start_is_half_a_second_into_the_slot():
"""A station whose clock is right reports as dt 0.0, not dt 0.5: the
transmission is 12.64 seconds long in a slot of fifteen and everybody
begins half a second in."""
audio = ft8wave.transmit("CQ DL1UDO JO31", offset=ft8wave.NOMINAL_START)
got = ft8wave.listen_to(audio, 12000.0)
assert got and abs(got[0].offset) < 0.1
def test_several_transmissions_in_one_slot_all_come_out():
"""Which is the whole point of the mode: they overlap in time entirely
and are told apart by frequency alone."""
said = [("CQ DL1UDO JO31", 500.0), ("VK4BLE OH8JK R-17", 1000.0),
("JR5MJS OH8NW 73", 1600.0), ("CQ DX Z33Z KN11", 2300.0)]
audio = np.zeros(int(12000 * code.SLOT_S), dtype=np.float32)
for text, hertz in said:
audio += ft8wave.transmit(text, hertz=hertz, offset=0.5)
got = {d.text for d in ft8wave.listen_to(audio, 12000.0)}
assert got == {t for t, _hz in said}
def test_a_slot_of_noise_decodes_to_nothing():
rng = np.random.default_rng(9)
noise = rng.standard_normal(int(12000 * code.SLOT_S)).astype(np.float32)
assert ft8wave.listen_to(noise, 12000.0) == []
def test_it_hears_a_signal_well_below_the_noise():
"""The claim the mode is built on. Fifteen decibels under is a signal
an operator hears nothing of at all."""
heard = 0
for seed in range(5):
audio = ft8wave.transmit("CQ DL1UDO JO31", hertz=1200.0, offset=0.5,
snr_db=-15.0, seed=seed)
got = ft8wave.listen_to(audio, 12000.0)
heard += any(d.text == "CQ DL1UDO JO31" for d in got)
assert heard == 5
def test_the_signal_report_tracks_the_signal():
"""Not to a decibel -- it is an estimate -- but a stronger signal has
to read stronger, or the number is worse than not printing one."""
readings = []
for want in (-15.0, -5.0, 5.0):
audio = ft8wave.transmit("CQ DL1UDO JO31", hertz=1200.0, offset=0.5,
snr_db=want, seed=1)
got = ft8wave.listen_to(audio, 12000.0)
assert got
readings.append(got[0].snr_db)
assert readings[0] < readings[1] < readings[2]
assert all(abs(r - w) < 8.0
for r, w in zip(readings, (-15.0, -5.0, 5.0)))
def test_the_waterfall_is_half_a_symbol_and_half_a_tone():
"""A transmission a quarter of a symbol late, or a quarter of a tone
off, still has to be caught."""
fall = ft8wave.waterfall(np.zeros(12000 * 15, dtype=np.float32), 12000.0)
assert fall.hz_per_bin == pytest.approx(code.SYMBOL_HZ
/ ft8wave.FREQ_STEPS)
assert fall.seconds_per_step == pytest.approx(code.SYMBOL_S
/ ft8wave.TIME_STEPS)
def test_audio_too_short_to_hold_a_symbol_is_refused_rather_than_guessed():
with pytest.raises(ValueError):
ft8wave.waterfall(np.zeros(10, dtype=np.float32), 12000.0)
# ---------------------------------------------------------------------------
# Against the air
# ---------------------------------------------------------------------------
def _recording(name):
with wave.open(str(REFERENCE / f"{name}.wav")) as handle:
raw = handle.readframes(handle.getnframes())
rate = float(handle.getframerate())
audio = np.frombuffer(raw, dtype=np.int16).astype(np.float32) / 32768.0
want = {}
for line in (REFERENCE / f"{name}.txt").read_text().splitlines():
parts = line.split(None, 4)
if len(parts) >= 5:
want[" ".join(parts[4].lstrip("~ ").split())] = (
float(parts[1]), float(parts[2]), float(parts[3]))
return audio, rate, want
@needs_recordings
@pytest.mark.parametrize("name,least", [
("191111_110145", 2), ("websdr_test3", 7), ("20m_busy_test_02", 14),
])
def test_it_decodes_signals_that_came_off_the_air(name, least):
"""The only test here whose answers were not written by this program."""
audio, rate, want = _recording(name)
got = {d.text: d for d in ft8wave.listen_to(audio, rate)}
matched = [w for w in want if any(w.startswith(t) for t in got)]
assert len(matched) >= least, f"decoded {len(matched)} of {len(want)}"
# And nothing it decoded may be something nobody else heard: a false
# line in a log is worse than a missing one.
unknown = [t for t in got if not any(w.startswith(t) for w in want)]
assert len(unknown) <= 1, unknown
@needs_recordings
def test_the_time_and_frequency_agree_with_what_else_heard_them():
audio, rate, want = _recording("websdr_test3")
errors = []
for d in ft8wave.listen_to(audio, rate):
for text, (snr, dt, hz) in want.items():
if text.startswith(d.text):
errors.append((d.offset - dt, d.hertz - hz))
assert len(errors) >= 6
assert max(abs(dt) for dt, _hz in errors) < 0.2
assert max(abs(hz) for _dt, hz in errors) < 4.0
# ---------------------------------------------------------------------------
# Slots and grids
# ---------------------------------------------------------------------------
def test_slots_are_quarter_minutes_of_utc():
"""Not of the local clock and not of when this program started: the
protocol is built on every station on earth agreeing which one it is."""
assert ft8.slot_start(1_700_000_007.0) == 1_700_000_000.0 - \
(1_700_000_000.0 % 15)
for when in (0.0, 1_700_000_000.0, 1_700_000_014.9):
assert ft8.slot_start(when) % ft8.SLOT == 0
assert 0 <= when - ft8.slot_start(when) < ft8.SLOT
assert ft8.next_slot(when) == ft8.slot_start(when) + ft8.SLOT
@pytest.mark.parametrize("grid,lat,lon", [
("IO91", 51.5, -1.0), ("FN31", 41.5, -73.0), ("JN54", 44.5, 11.0),
("RE78", -41.5, 175.0),
])
def test_a_grid_square_is_where_it_says(grid, lat, lon):
got = ft8.grid_at(grid)
assert got == pytest.approx((lat, lon), abs=0.01)
def test_a_grid_that_is_not_one_is_refused():
for rubbish in ("", "IO", "9191", "ZZ99", "IO9", "ABCD"):
assert ft8.grid_at(rubbish) is None
def test_the_distance_between_two_squares_is_the_great_circle_one():
km, bearing = ft8.grid_away("IO91", "FN31")
assert km == pytest.approx(5390, abs=60)
assert bearing == pytest.approx(288, abs=3)
assert ft8.grid_away("IO91", "IO91")[0] < 1.0
assert ft8.grid_away("IO91", "") is None
# ---------------------------------------------------------------------------
# The options
# ---------------------------------------------------------------------------
def test_every_option_names_a_field_that_exists():
fields = set(ft8.Ft8Options().__dict__)
for option in ft8.OPTIONS:
assert option.key in fields, f"{option.key} is not an option"
def test_every_field_is_an_option_somebody_can_reach():
assert set(ft8.Ft8Options().__dict__) == {o.key for o in ft8.OPTIONS}
def test_every_option_is_in_a_group_and_says_what_it_does():
for option in ft8.OPTIONS:
assert option.group in ft8.OPTION_GROUPS
assert option.help and option.detail and option.flags
if option.kind == "bool":
assert option.off_flags, f"{option.key} cannot be turned off"
for group in ft8.OPTION_GROUPS:
assert ft8.in_group(group)
def test_every_option_flag_is_one_the_parser_accepts():
from bandsaunter.cli import build_parser
known = set()
for action in build_parser()._subparsers._group_actions[0].choices[
"ft8"]._actions:
known.update(action.option_strings)
for option in ft8.OPTIONS:
for flag in option.flags + option.off_flags:
assert flag in known, f"{flag} is in the table and not the parser"
def test_every_command_line_option_is_reachable_from_the_menu():
"""Asked for explicitly, and worth a test rather than a promise: the
menu builds itself from the groups, so an option in no group would be
settable from the command line and invisible in the menu."""
from bandsaunter import tui
text = Path(tui.__file__).read_text()
assert "def ft8_menu(" in text
assert "ft8_menu(console, cfg)" in text, "not reachable from the front page"
reachable = set()
for group in ft8.OPTION_GROUPS:
reachable |= {o.key for o in ft8.in_group(group)}
assert reachable == {o.key for o in ft8.OPTIONS}
def test_the_options_survive_being_saved_and_read_back(tmp_path):
options = ft8.Ft8Options(band="20m", frequency=14_074_000.0,
grid="IO91", units="imperial", slots=12)
ft8.save_options(options, tmp_path)
assert ft8.load_options(tmp_path) == options
def test_a_settings_file_with_a_mistake_in_it_costs_the_defaults(tmp_path):
(tmp_path / "ft8.yaml").write_text("band: [not a band\n")
assert ft8.load_options(tmp_path) == ft8.Ft8Options()
@pytest.mark.parametrize("options,broken", [
(ft8.Ft8Options(rate=48_000.0), "sample rate"),
(ft8.Ft8Options(seconds=-1.0), "negative"),
(ft8.Ft8Options(hold=0.0), "display"),
(ft8.Ft8Options(frequency=10.0), "frequency"),
(ft8.Ft8Options(lowest=2000.0, highest=500.0), "above the bottom"),
(ft8.Ft8Options(lowest=1000.0, highest=1050.0), "narrower"),
(ft8.Ft8Options(most=0), "candidate"),
(ft8.Ft8Options(rounds=0), "error correction"),
(ft8.Ft8Options(grid="nowhere"), "grid square"),
])
def test_a_setting_that_cannot_work_is_refused(options, broken):
assert any(broken in err for err in options.validate())
def test_the_defaults_are_all_workable():
assert ft8.Ft8Options().validate() == []
def test_choosing_a_band_sets_the_frequency_with_it():
options = ft8.defaults()
ft8.use_band(options, "20m")
assert options.frequency == 14_074_000.0 and options.band == "20m"
ft8.use_band(options, "nonsense")
assert options.frequency == 144_174_000.0
def test_the_band_list_says_which_ones_a_plain_receiver_can_reach():
"""Almost all the activity is on shortwave, which this kind of receiver
cannot hear without help, and finding that out by listening to silence
for ten minutes is the wrong way to learn it."""
assert "upconverter" in ft8.band_text(ft8.Ft8Options(band="20m"))
assert "upconverter" not in ft8.band_text(ft8.Ft8Options(band="2m"))
assert ft8.defaults().band == "2m"
def test_the_command_line_beats_the_saved_settings(tmp_path, monkeypatch):
from bandsaunter.cli import build_parser
ft8.save_options(ft8.Ft8Options(band="2m", grid="AA00"), tmp_path)
monkeypatch.setattr(ft8, "options_path",
lambda directory=None: tmp_path / "ft8.yaml")
args = build_parser().parse_args(["ft8", "--band", "20m",
"--grid", "IO91"])
assert args.band == "20m" and args.grid == "IO91"
# ---------------------------------------------------------------------------
# End to end
# ---------------------------------------------------------------------------
class _Clock:
"""A clock the test winds, so a run of slots takes no real time."""
def __init__(self, at=1_700_000_000.0):
self.at = at
def now(self):
return self.at
def sleep(self, seconds):
self.at += seconds
def _run(slots=2, **kw):
clock = _Clock()
band = ft8sim.SimulatedBand(rate=240_000.0, realtime=True, stations=12,
clock=clock.now, sleep=clock.sleep)
options = ft8.defaults()
options.slots = slots
for key, value in kw.items():
setattr(options, key, value)
heard = ft8.Heard()
heard.started = clock.now()
total = ft8.pump(band, options, heard, None, clock.now(),
clock=clock.now)
return heard, total
def test_a_made_up_band_comes_out_the_other_end():
"""The whole path: samples, sideband, slot boundaries, sync, decoding,
and into the book of stations."""
heard, total = _run(slots=2)
assert heard.slots == 2
assert total >= 8, f"only {total} decodes in two slots"
assert len(heard.band.stations) >= 6
for station in heard.band.all():
assert station.decodes >= 1
assert station.call and station.call != "<...>"
def test_the_slot_boundaries_are_found_rather_than_assumed():
"""A receiver that dated its samples a block out would slice every slot
late and cut the first half-second -- three symbols and part of the
opening sync -- off every transmission on the band."""
heard, _total = _run(slots=2)
assert heard.band.decodes
# Everything on this band starts within half a second of the nominal
# start, so everything should be reported within about that of zero.
worst = max(abs(d.offset) for d in heard.band.decodes)
assert worst < 1.0, f"the worst decode was {worst:.2f} s out"
def test_callsigns_only_leaves_out_the_free_text():
heard, _ = _run(slots=1, calls_only=True)
assert all(d.kind in ("standard", "non-standard")
for d in heard.band.decodes)
def test_the_search_can_be_narrowed_and_the_narrowing_is_obeyed():
heard, _ = _run(slots=1, lowest=800.0, highest=1600.0)
assert heard.band.decodes
assert all(800.0 <= d.hertz <= 1600.0 for d in heard.band.decodes)
def test_a_station_that_calls_cq_is_counted_as_calling():
heard, _ = _run(slots=2)
calling = [s for s in heard.band.all() if s.calling]
assert calling, "nobody called CQ on a band that is mostly CQ"
assert all(s.grid for s in calling), "a CQ says where it is"
# ---------------------------------------------------------------------------
# Writing it down
# ---------------------------------------------------------------------------
def _decode(text="CQ DL1UDO JO31", **kw):
fields = dict(text=text, kind="standard", calls=("DL1UDO",),
grid="JO31", report="", calling=True, hertz=1200.0,
offset=0.2, snr_db=-7.0, at=1_700_000_000.0)
fields.update(kw)
return ft8wave.Decode(**fields)
def test_a_log_is_written_and_read_back(tmp_path):
log = ft8log.open_log(tmp_path, 1_700_000_000.0, 14_074_000.0, "20m")
assert log is not None
for i in range(3):
log.append(_decode(hertz=1000.0 + i * 100))
log.close()
rows = ft8log.read_log(log.path)
assert len(rows) == 3
assert rows[0]["text"] == "CQ DL1UDO JO31"
assert rows[1]["hertz"] == 1100.0
def test_a_half_written_log_line_is_skipped_rather_than_raising(tmp_path):
path = tmp_path / "ft8-part.txt"
path.write_text("# header\n2023-11-14 22:13:20 -7 0.2 1200.0 ~ OK\n"
"2023-11-14 22:13:35 -7 0.2\n")
rows = ft8log.read_log(path)
assert len(rows) == 1 and rows[0]["text"] == "OK"
def test_a_csv_has_a_row_for_every_decode(tmp_path):
where = ft8log.write_csv(tmp_path / "x.csv",
[_decode(), _decode("W2XYZ K1ABC RRR")])
assert where is not None
body = where.read_text().splitlines()
assert len(body) == 3 and body[0].startswith("utc,")
def test_an_adif_says_these_were_heard_rather_than_worked(tmp_path):
"""Nothing here transmits, so nothing here is a contact. An ADIF that
let a logging program treat these as worked would put claims into
somebody's log that they cannot make."""
where = ft8log.write_adif(tmp_path / "x.adi",
[_decode(), _decode("W2XYZ K1ABC RRR",
calls=("W2XYZ", "K1ABC"))],
band="20m", frequency=14_074_000.0,
my_grid="IO91")
body = where.read_text()
assert "heard only, not worked" in body
assert "not contacts" in body
assert "<CALL:6>DL1UDO" in body
assert "<MODE:3>FT8" in body
assert "<MY_GRIDSQUARE:4>IO91" in body
def test_an_adif_keeps_one_record_a_station_rather_than_one_a_decode(tmp_path):
many = [_decode(snr_db=s) for s in (-20.0, -3.0, -11.0)]
where = ft8log.write_adif(tmp_path / "y.adi", many)
body = where.read_text()
assert body.count("<EOR>") == 1
# And the one kept is the best report, that being the one worth passing on.
assert "<RST_RCVD:2>-3" in body
# ---------------------------------------------------------------------------
# The display
# ---------------------------------------------------------------------------
def test_the_display_says_what_it_is_waiting_for_before_anything_arrives():
from rich.console import Console
from bandsaunter.ui import Ft8Display
console = Console(width=160, force_terminal=False)
display = Ft8Display(console, band="20m — 14.074 MHz")
display.update(ft8.Heard())
with console.capture() as caught:
console.print(display.render())
shown = caught.get()
assert "nothing decoded yet" in shown
# And the one thing that silently stops FT8 working is worth saying.
assert "clock" in shown
def test_the_display_shows_the_last_slot_and_the_running_total():
from rich.console import Console
from bandsaunter.ui import Ft8Display
console = Console(width=110, force_terminal=False)
display = Ft8Display(console, grid="IO91", band="20m — 14.074 MHz")
heard = ft8.Heard()
for text in ("CQ DL1UDO JO31", "W2XYZ K1ABC RRR"):
found = _decode(text, calls=tuple(text.split()[1:2] or ["DL1UDO"]),
grid="JO31" if "CQ" in text else "")
heard.band.add(found)
heard.decodes += 1
heard.latest = heard.band.decodes
heard.slots = 4
display.update(heard, 1_700_000_000.0)
with console.capture() as caught:
console.print(display.render(now=1_700_000_005.0))
shown = caught.get()
assert "last slot" in shown and "heard so far" in shown
assert "DL1UDO" in shown and "JO31" in shown
assert "4 slots" in shown
def test_the_report_says_nothing_rather_than_an_empty_table():
from rich.console import Console
console = Console(width=100, force_terminal=False)
band = ft8.Band()
band.slots = 12
with console.capture() as caught:
ft8.report(console, band, ft8.defaults())
shown = caught.get()
assert "nothing decoded" in shown
assert "clock" in shown, "the usual cause is worth naming"
def test_the_report_names_the_furthest_station_when_it_can():
from rich.console import Console
console = Console(width=120, force_terminal=False)
band = ft8.Band()
band.slots = 4
band.add(_decode("CQ VK4BLE QG62", calls=("VK4BLE",), grid="QG62"))
band.add(_decode("CQ DL1UDO JO31", calls=("DL1UDO",), grid="JO31"))
options = ft8.defaults()
options.grid = "IO91"
with console.capture() as caught:
ft8.report(console, band, options)
shown = caught.get()
assert "furthest heard" in shown and "VK4BLE" in shown
def test_the_report_leaves_distances_out_when_it_has_nowhere_to_measure_from():
from rich.console import Console
console = Console(width=120, force_terminal=False)
band = ft8.Band()
band.add(_decode("CQ DL1UDO JO31", calls=("DL1UDO",), grid="JO31"))
with console.capture() as caught:
ft8.report(console, band, ft8.defaults())
shown = caught.get()
assert "DL1UDO" in shown and "away" not in shown
# ---------------------------------------------------------------------------
# Opening a receiver
# ---------------------------------------------------------------------------
def test_every_name_these_modules_import_actually_exists():
"""Imports inside a function are not checked until that function runs.
Which is how `open_rtlsdr` -- a name nothing in this program has ever
exported -- sat in the one line that opens the receiver, past every
test, until somebody chose Listen and was told the section was broken.
The tests reached that line only through the simulator, which takes the
other branch.
"""
import ast
import importlib
missing = []
for name in ("ft8", "ft8code", "ft8wave", "ft8log", "ft8sim",
"ft8tables"):
module = importlib.import_module(f"bandsaunter.{name}")
tree = ast.parse(Path(module.__file__).read_text())
for node in ast.walk(tree):
if not isinstance(node, ast.ImportFrom) or node.level != 1:
continue
if not node.module:
continue
target = importlib.import_module(f"bandsaunter.{node.module}")
for alias in node.names:
if not hasattr(target, alias.name):
missing.append(f"{name}: {node.module}.{alias.name}")
assert not missing, missing
def test_the_made_up_band_needs_no_receiver_and_says_it_is_made_up():
from rich.console import Console
console = Console(width=90, force_terminal=False)
options = ft8.defaults()
options.simulate = True
with console.capture() as caught:
device = ft8.open_device(console, options)
assert isinstance(device, ft8sim.SimulatedBand)
assert "not there" in caught.get(), "a simulation must say so"
device.close()
def test_a_receiver_that_cannot_be_opened_is_reported_rather_than_raising():
"""A menu has to survive it, and the person has to be told which of the
two things went wrong: the receiver, or the listening."""
from rich.console import Console
console = Console(width=90, force_terminal=False)
options = ft8.defaults()
options.device = 99 # no such dongle
with console.capture() as caught:
assert ft8.open_device(console, options) is None
assert "cannot open the receiver" in caught.get()
def test_nothing_claims_to_be_listening_before_there_is_a_receiver(tmp_path):
"""Announcing the frequency and then failing to open a dongle reads as
though the listening went wrong, when what went wrong happened before
any of it started."""
from rich.console import Console
console = Console(width=90, force_terminal=False)
options = ft8.defaults()
options.device = 99
options.log = False
with console.capture() as caught:
assert ft8.listen(console, options, str(tmp_path)) == 0
shown = caught.get()
assert "cannot open the receiver" in shown
assert "listening on" not in shown