68b05a031c
Much of what a scanner finds is not speech. Doorbells, tyre-pressure sensors, weather stations, remote controls, paging and packet radio all carry something a receiver can read, and until now the answer was "OOK / ASK data burst" and a WAV file. Now the bits come out. The observation the whole thing is built on is that whatever the modulation, a data signal is the same shape once it has been sliced: a train of alternating runs whose lengths carry the information. On-off keying gives that directly -- the carrier is up or it is down -- and two-level FSK gives exactly the same thing from the discriminator, one tone or the other. So both reduce to a run-length train and everything after that is shared. What the runs mean is the line code, and it is worked out from the runs alone rather than configured, because each code makes a different prediction about which of the two histograms is the bimodal one: PWM (EV1527, PT2262, and nearly every 433 MHz remote), PPM, Manchester, and plain NRZ. Four-level FSK is recognised as such and read as symbols rather than sliced down the middle, which produces bits that mean nothing; where a frame sync word appears the system is named outright. Two protocols carry their own framing and checksums and so are read in full. POCSAG paging: all three rates tried because nothing in the signal says which it is, every codeword checked and single-bit errors corrected against the BCH code, and the address, function letter and message text reported. AX.25 as APRS uses it: the frame check has to come out right before a frame is reported at all, and the sender's callsign goes onto the map with everyone else's. The hard half is refusing what is not data. Noise sliced at a threshold produces runs and runs produce bits, so three things guard against it: the runs have to quantise to the line code's own grid; most of the bursts in a capture have to decode the same way, because one lucky window in eight is a coincidence and that is exactly what SSB voice produced; and, much the strongest, the packet has to repeat, because bits that come back identical six times did not come from noise. A reading with none of that behind it is reported as nothing at all rather than as a bit string with a low number beside it that somebody will read anyway. Across 27 recordings of speech, music, static, a bare carrier, Morse and PSK it returns nothing 27 times. A firm decode also outranks the content check, which is statistical: a burst of keying demodulated as FM audio is a buzz and the speech detector likes a buzz, but a frame whose own checksum came out right is not a statistic. Such a capture is kept and filed as data, not as voice. What comes out is written to a _data.txt beside the recording, shown on the live display and in the line-per-hit output, and takes the place of the transcript at the top of saunterbrowse -- where it is searchable, so "which page mentioned engine 4" is a question that can be asked. `bandsaunter analyze` decodes a file you already have. The simulator gained two honest transmitters to test against: a pulse-width remote that repeats a real payload, and a pager that sends real POCSAG batches with real BCH check bits. Random keying exercises the classifier but leaves a decoder nothing to get right. The POCSAG encoder lives next to the decoder rather than in the test helpers, so a bug shared by both cannot hide. Fixed along the way: - Rich reads a square bracket as markup, and a decoded page is arbitrary text off the air. "[/x]" in a message ended the live display with a MarkupError; so did typing "[/" at saunterbrowse's search prompt. Everything that did not come from this program is escaped now. - Otsu returned the first bin of a plateau. Two populations with nothing between them -- silence and full carrier, which is what on-off keying is -- make every threshold in the gap equally good, and taking the first put it hard against the lower population with the hysteresis band outside the data entirely, so nothing sliced at all. - Estimating the symbol clock by counting along a cumulative grid is a fixed point: a unit two per cent small produces two per cent more symbols and reproduces itself exactly. Rounding each run on its own converges instead, because every run votes independently. The grid is then the right way to extract the bits, where rounding runs one at a time drifts. - A clipped first repeat used to truncate every other repeat to its length. The consensus is taken over the commonest length now. 761 -> 869 tests.
227 lines
8.6 KiB
Python
227 lines
8.6 KiB
Python
"""Synthetic test signals shared by the test modules."""
|
|
import numpy as np
|
|
from scipy.signal import butter, hilbert, lfilter
|
|
|
|
FS = 32000.0
|
|
|
|
|
|
def _noise(x, snr_db, rng):
|
|
p = float(np.mean(np.abs(x) ** 2))
|
|
n = np.sqrt(p / (2 * 10 ** (snr_db / 10.0)))
|
|
return (x + n * (rng.standard_normal(x.size)
|
|
+ 1j * rng.standard_normal(x.size))).astype(np.complex64)
|
|
|
|
|
|
def voice(n, fs=FS, seed=0):
|
|
"""Band-limited noise with a syllabic envelope -- a good speech stand-in."""
|
|
rng = np.random.default_rng(seed)
|
|
b, a = butter(4, [300 / (fs / 2), 2700 / (fs / 2)], btype="band")
|
|
v = lfilter(b, a, rng.standard_normal(n))
|
|
v /= max(np.abs(v).max(), 1e-9)
|
|
t = np.arange(n) / fs
|
|
return v * (0.4 + 0.6 * np.abs(np.sin(2 * np.pi * 1.7 * t)))
|
|
|
|
|
|
def make(kind, n=64000, fs=FS, snr_db=30.0, seed=3):
|
|
rng = np.random.default_rng(seed)
|
|
t = np.arange(n) / fs
|
|
v = voice(n, fs, seed)
|
|
|
|
if kind == "nfm":
|
|
msg = v + 0.15 * np.sin(2 * np.pi * 100.0 * t)
|
|
x = np.exp(1j * np.cumsum(2 * np.pi * 2500 * msg / fs))
|
|
elif kind == "wfm":
|
|
x = np.exp(1j * np.cumsum(2 * np.pi * 3000 * v / fs))
|
|
elif kind == "am":
|
|
x = ((1 + 0.6 * v) * np.exp(2j * np.pi * 30 * t))
|
|
elif kind == "usb":
|
|
x = 0.5 * hilbert(v)
|
|
elif kind == "lsb":
|
|
x = 0.5 * np.conj(hilbert(v))
|
|
elif kind == "carrier":
|
|
x = np.exp(2j * np.pi * 137 * t)
|
|
elif kind == "cw":
|
|
dot = 0.08
|
|
pat = [1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0]
|
|
key = np.zeros(n)
|
|
i = 0
|
|
while i < n:
|
|
for b in pat:
|
|
m = int(dot * fs)
|
|
if i + m > n:
|
|
break
|
|
key[i:i + m] = b
|
|
i += m
|
|
env = np.convolve(key, np.hanning(int(0.005 * fs)), "same")
|
|
env /= max(env.max(), 1e-9)
|
|
x = env * np.exp(2j * np.pi * 300 * t)
|
|
elif kind.startswith("fsk"):
|
|
levels = int(kind[3])
|
|
baud, dev = (1200.0, 2400.0) if levels == 2 else (4800.0, 1800.0)
|
|
sp = int(fs / baud)
|
|
sym = rng.integers(0, levels, n // sp + 1)
|
|
lv = (sym - (levels - 1) / 2) / max(1, (levels - 1) / 2)
|
|
f = np.resize(np.repeat(lv, sp) * dev, n)
|
|
x = np.exp(1j * np.cumsum(2 * np.pi * f / fs))
|
|
elif kind.startswith("psk"):
|
|
m = int(kind[3])
|
|
sp = int(fs / 4800.0)
|
|
sym = rng.integers(0, m, n // sp + 1)
|
|
x = np.exp(1j * np.resize(np.repeat(2 * np.pi * sym / m, sp), n))
|
|
elif kind == "noise":
|
|
return (0.01 * (rng.standard_normal(n)
|
|
+ 1j * rng.standard_normal(n))).astype(np.complex64)
|
|
else:
|
|
raise ValueError(kind)
|
|
return _noise(np.asarray(x, dtype=np.complex128), snr_db, rng)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Packets. What the decoder is for: signals that carry something, rather
|
|
# than random keying that only exercises the classifier.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def _rf(env, fs, snr_db=30.0, seed=1):
|
|
"""An envelope on a carrier, with noise."""
|
|
x = np.asarray(env, dtype=np.float64).astype(np.complex128)
|
|
rng = np.random.default_rng(seed)
|
|
p = float(np.mean(np.abs(x) ** 2)) or 1e-9
|
|
n = np.sqrt(p / (2 * 10 ** (snr_db / 10.0)))
|
|
return (x + n * (rng.standard_normal(x.size)
|
|
+ 1j * rng.standard_normal(x.size))).astype(np.complex64)
|
|
|
|
|
|
def ook_pwm(bits, fs=50_000.0, alpha=350e-6, repeats=4, gap=10e-3,
|
|
sync=True, snr_db=30.0, seed=1, fixed_gap=None):
|
|
"""Pulse-width keying, as an EV1527 or PT2262 remote sends it.
|
|
|
|
With ``fixed_gap`` the gap is constant and only the pulse varies; without
|
|
it the two are complementary and the bit period is constant, which is
|
|
what the common parts actually do.
|
|
"""
|
|
a = int(round(alpha * fs))
|
|
env = []
|
|
for _ in range(repeats):
|
|
if sync:
|
|
env += [1] * a + [0] * (31 * a) # the sync pulse and its gap
|
|
for b in bits:
|
|
if fixed_gap is None:
|
|
high, low = (3 * a, a) if b == "1" else (a, 3 * a)
|
|
else:
|
|
high = 3 * a if b == "1" else a
|
|
low = int(round(fixed_gap * fs))
|
|
env += [1] * high + [0] * low
|
|
env += [0] * int(gap * fs)
|
|
return _rf(env, fs, snr_db, seed)
|
|
|
|
|
|
def ook_ppm(bits, fs=50_000.0, alpha=400e-6, repeats=4, gap=10e-3,
|
|
snr_db=30.0, seed=2):
|
|
"""Pulse-distance keying: the pulse is constant, the gap carries the bit."""
|
|
a = int(round(alpha * fs))
|
|
env = []
|
|
for _ in range(repeats):
|
|
for b in bits:
|
|
env += [1] * a + [0] * (3 * a if b == "1" else a)
|
|
env += [0] * int(gap * fs)
|
|
return _rf(env, fs, snr_db, seed)
|
|
|
|
|
|
def ook_manchester(bits, fs=50_000.0, baud=2000.0, repeats=3, gap=10e-3,
|
|
snr_db=30.0, seed=3, preamble="10101010"):
|
|
half = int(round(fs / baud / 2))
|
|
env = []
|
|
for _ in range(repeats):
|
|
for b in preamble + bits:
|
|
first, second = (1, 0) if b == "1" else (0, 1) # IEEE 802.3
|
|
env += [first] * half + [second] * half
|
|
env += [0] * int(gap * fs)
|
|
return _rf(env, fs, snr_db, seed)
|
|
|
|
|
|
def fsk_nrz(bits, fs=48_000.0, baud=1200.0, dev=4500.0, snr_db=30.0, seed=4):
|
|
"""Two-level FSK holding each bit for one symbol period."""
|
|
sp = fs / baud
|
|
n = int(round(len(bits) * sp))
|
|
idx = np.clip((np.arange(n) / sp).astype(int), 0, len(bits) - 1)
|
|
level = np.array([1.0 if c == "1" else -1.0 for c in bits])[idx]
|
|
rng = np.random.default_rng(seed)
|
|
x = np.exp(1j * np.cumsum(2 * np.pi * level * dev / fs))
|
|
nz = np.sqrt(1.0 / (2 * 10 ** (snr_db / 10.0)))
|
|
return (x + nz * (rng.standard_normal(n)
|
|
+ 1j * rng.standard_normal(n))).astype(np.complex64)
|
|
|
|
|
|
_C4FM = {"01": 3, "00": 1, "10": -1, "11": -3}
|
|
|
|
|
|
def c4fm(bits, fs=48_000.0, baud=4800.0, dev=1800.0, snr_db=30.0, seed=8):
|
|
"""Four-level FSK, mapped the way P25 and DMR map it."""
|
|
bits = bits[:len(bits) - len(bits) % 2]
|
|
level = np.array([_C4FM[bits[i:i + 2]] / 3.0
|
|
for i in range(0, len(bits), 2)])
|
|
sp = fs / baud
|
|
n = int(level.size * sp)
|
|
idx = np.clip((np.arange(n) / sp).astype(int), 0, level.size - 1)
|
|
rng = np.random.default_rng(seed)
|
|
x = np.exp(1j * np.cumsum(2 * np.pi * level[idx] * dev / fs))
|
|
nz = np.sqrt(1.0 / (2 * 10 ** (snr_db / 10.0)))
|
|
return (x + nz * (rng.standard_normal(n)
|
|
+ 1j * rng.standard_normal(n))).astype(np.complex64)
|
|
|
|
|
|
def ax25_address(call, ssid, last=False):
|
|
call = (call.upper() + " ")[:6]
|
|
return bytes((ord(c) << 1) & 0xFE for c in call) + \
|
|
bytes([0x60 | ((ssid & 0x0F) << 1) | (1 if last else 0)])
|
|
|
|
|
|
def ax25_frame(source, dest, info, path=()):
|
|
"""A complete AX.25 frame, with its frame-check sequence."""
|
|
body = ax25_address(*dest) + ax25_address(*source, last=not path)
|
|
for i, hop in enumerate(path):
|
|
body += ax25_address(*hop, last=(i == len(path) - 1))
|
|
body += bytes([0x03, 0xF0]) + info.encode()
|
|
crc = 0xFFFF
|
|
for byte in body:
|
|
crc ^= byte
|
|
for _ in range(8):
|
|
crc = (crc >> 1) ^ 0x8408 if crc & 1 else crc >> 1
|
|
crc ^= 0xFFFF
|
|
return body + bytes([crc & 0xFF, (crc >> 8) & 0xFF])
|
|
|
|
|
|
def aprs_afsk(frames, fs=48_000.0, baud=1200.0, snr_db=30.0, seed=6,
|
|
mark=1200.0, space=2200.0, dev=3000.0, flags=8):
|
|
"""AX.25 over Bell 202 AFSK inside an FM carrier, as APRS is sent."""
|
|
stream = ""
|
|
for frame in frames:
|
|
body = "01111110" * flags
|
|
ones = 0
|
|
for byte in frame:
|
|
for k in range(8): # least significant first
|
|
bit = (byte >> k) & 1
|
|
body += "1" if bit else "0"
|
|
if bit:
|
|
ones += 1
|
|
if ones == 5: # stuff a zero after five
|
|
body += "0"
|
|
ones = 0
|
|
else:
|
|
ones = 0
|
|
stream += body + "01111110" * flags
|
|
level, nrzi = 1, []
|
|
for bit in stream:
|
|
if bit == "0":
|
|
level ^= 1 # NRZI: a zero is a change
|
|
nrzi.append(level)
|
|
sp = fs / baud
|
|
n = int(len(nrzi) * sp)
|
|
idx = np.clip((np.arange(n) / sp).astype(int), 0, len(nrzi) - 1)
|
|
tone = np.where(np.array(nrzi)[idx] == 1, mark, space)
|
|
audio = np.sin(np.cumsum(2 * np.pi * tone / fs))
|
|
rng = np.random.default_rng(seed)
|
|
x = np.exp(1j * np.cumsum(2 * np.pi * dev * audio / fs))
|
|
nz = np.sqrt(1.0 / (2 * 10 ** (snr_db / 10.0)))
|
|
return (x + nz * (rng.standard_normal(n)
|
|
+ 1j * rng.standard_normal(n))).astype(np.complex64)
|