diff --git a/INSTALL.md b/INSTALL.md index eef7b3f..3ef29d1 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -216,6 +216,13 @@ aerial: a quarter-wave whip is 17 cm, which the stock telescopic aerial does if it is collapsed to about that. `bandsaunter weather --simulate` runs the whole thing without one. +If sensors you know are in range are not appearing, `bandsaunter weather +--diagnose` prints each second of band taken apart stage by stage and says +which of the four possible faults it is — nothing arriving, nothing above the +noise, a burst that sliced into the wrong shape, or bits that came out and +failed their checksums. The README section on it explains how to read the +output. + **Aircraft and callsign lookups need no installation**, only a network. They ask public registers about a callsign or a 24-bit address and cache the answers for a month; `--no-lookup` turns them off, and what the address and diff --git a/README.md b/README.md index 0e759ef..dead4f3 100644 --- a/README.md +++ b/README.md @@ -1959,15 +1959,42 @@ Only then is the magnitude taken. Filtering before detection rather than after is what keeps the neighbours — a doorbell, a tyre sensor, a car key — from adding themselves to the envelope of the sensor. -Slicing the envelope into bits never measures anything against a clock. The -newer sensors vary the length of the pulse and keep the gaps even; the two -older ones keep the pulse even and vary the gap. Both readings of the same -pulses are tried and the checksums say which it was — rather than deciding -from the timings, which is guessing, and wrong on a weak burst where the edges -have moved. A transmitter running ten per cent fast is read correctly and -never noticed, which matters: these transmitters are unlocked and drift tens -of kilohertz and a few per cent of rate with the temperature. An outdoor -sensor in January is not the one that was on the fence in July. +**Finding the bursts is done in two passes, and the reason is having more than +one sensor.** The first pass only asks where anything is happening at all, and +asks it against the noise — the bottom fifth of the second, which is noise +however busy the rest was. Whatever clears that is grouped into regions, and +the second pass re-thresholds each region against its own high and low, so +every sensor is sliced at its own amplitude. + +The obvious way to write this is one threshold per second, set halfway between +the noise floor and the loudest thing in it. That is wrong, and wrong in a way +worth describing because the symptom is so odd: a sensor on the windowsill and +a sensor at the end of the garden differ by forty decibels, so a threshold +halfway to the near one sits above everything the far one ever does — and the +far ones disappear, completely, and only while the near one is transmitting. A +block with one loud sensor in it yields exactly one sensor however many are out +there. + +**Slicing the envelope into bits never measures anything against a clock**, +and assumes as little as it can about how a bit is drawn. Three things are not +assumed. Which of the pulse and the gap carries the bit — the newer sensors +vary the pulse, the older two vary the gap. Whether the gap is the complement +of the pulse, so that every bit takes the same time, or just a fixed spacer: +judged against a fixed 200 µs spacer a short pulse of 220 µs is longer than its +own gap and reads as a one, which is the wrong bit, and every message then +fails its checksum saying nothing about why. And which of long and short means +one. + +So the same burst is read half a dozen ways — the pulse against its own gap, +the pulse against each threshold the pulse lengths themselves suggest, the +same for the gaps, and each of those inverted — and the checksums say which +reading it was. At most one of them can satisfy a checksum. It costs a few +microseconds per burst. + +A transmitter running ten per cent fast is therefore read correctly and never +noticed, which matters: these are unlocked and drift tens of kilohertz and a +few per cent of rate with the temperature. An outdoor sensor in January is not +the one that was on the fence in July. ### Afterwards @@ -2031,6 +2058,7 @@ only what is shown and can be changed afterwards on an old log. | Listen for | `--seconds` | until stopped | how long before stopping | | Write a log | `--log` / `--no-log` | yes | one line of JSON per message | | Print every message | `--messages` / `--no-messages` | no | a stream of lines instead of a table | +| Say what is arriving | `--diagnose` / `--no-diagnose` | no | each second taken apart stage by stage | | Keep on screen for | `--hold` | 1800 s | how long a sensor stays after its last message | | Show readings in | `--units` | metric | metric or imperial, for the display and the export | | Only named sensors | `--only-named` / `--all-sensors` | no | ignore anything without a name | @@ -2039,6 +2067,7 @@ only what is shown and can be changed afterwards on an old log. | Also write a spreadsheet | `--csv` / `--no-csv` | no | CSV beside the log | `--name ID=NAME` is not a setting: it names a sensor and is repeatable. +Neither is `--save-iq FILE`, which is a one-off capture of the raw samples. Saved in `weather.yaml` beside the other settings, from the menu's **s** or by hand. `bandsaunter readings` also takes `--sensor NAME` to narrow a log to one @@ -2061,12 +2090,56 @@ a program that cannot read a thermometer. ### If nothing is heard -These are a few milliwatts. A quarter-wave whip for 433.92 MHz is 17 cm of -wire, and the stock telescopic aerial set to about that length works well. -Indoors, behind a wall, with the dongle in the back of a machine, is usually -the problem. Try `--gain 40` if the automatic gain control is not finding -them, and `--messages` to watch individual receptions arrive while moving the -aerial about. +**`bandsaunter weather --diagnose` is the answer to this**, because "nothing +was heard" is four different faults wearing the same coat and they want four +different answers. It prints each second of band taken apart stage by stage: + +``` + 12.4s noise 0.0219 gate 0.0450 peak 1.0179 (46.4x the noise) 3 bursts + 60 pulses pulses 219×29 399×27 597×4 gaps 221×26 402×29 600×4 + Tower 592TXR 1A2B ch A temperature 8.4 C humidity 88% +``` + +Read it from the left. + +**`peak` is barely above `noise`, no bursts.** Nothing is arriving. That is an +aerial. These are a few milliwatts: a quarter-wave whip for 433.92 MHz is 17 cm +of wire, which is the stock telescopic aerial collapsed to about that, and +indoors behind a wall with the dongle in the back of a machine is usually the +problem. Try `--gain 40` if the automatic gain control is not finding them. + +**`peak` is well above `noise` and there are no bursts.** Something is there +and did not group — usually a continuous transmitter rather than a keyed one, +which is not one of these. + +**Bursts, but the pulse lengths are not two or three clean groups.** The +receiver is hearing it and the slicing is wrong. A real message shows two or +three lengths with nothing in between, like the `219×29 399×27 597×4` above. +A smear of lengths means noise is being sliced as signal, or two sensors are +transmitting over each other. + +**Bursts with clean pulse lengths and `nothing framed`.** The radio is fine +and the message is from a model this does not read, or reads differently. +That line of pulse lengths is exactly what is needed to add it. + +**`framed, but this model needs the same message twice`.** It was read +correctly and arrived once. The two older models are only believed on a second +copy; a stronger signal fixes it. + +When the listening stops it says which of those it was, once: + +``` +╭───────────────────────── what that came to ──────────────────────────╮ +│ 40 bursts were received and sliced, and not one of them framed as a │ +│ message. The radio end is working: what is arriving is a model this │ +│ cannot read, or reads differently. The pulse lengths printed above │ +│ are exactly what is needed to add it. │ +╰──────────────────────────────────────────────────────────────────────╯ +``` + +`--save-iq FILE` writes the raw samples alongside, for working out anything the +diagnosis cannot. It is 8 MB a second at the default rate, so bound it with +`--seconds 60`. ## Meters on 900 MHz diff --git a/bandsaunter/__init__.py b/bandsaunter/__init__.py index d916c88..35965a4 100755 --- a/bandsaunter/__init__.py +++ b/bandsaunter/__init__.py @@ -9,7 +9,7 @@ and transcribing speech. # 2026-08-21_02 is the second build made on the 21st. The revision is padded # to two digits so versions sort as text. VERSION_DATE = "2026-09-07" -VERSION_REVISION = 1 +VERSION_REVISION = 2 __version__ = f"{VERSION_DATE}_{VERSION_REVISION:02d}" diff --git a/bandsaunter/acurite.py b/bandsaunter/acurite.py index ce33a5c..3904404 100644 --- a/bandsaunter/acurite.py +++ b/bandsaunter/acurite.py @@ -76,9 +76,10 @@ __all__ = ["Reading", "Measure", "format_measure", "readings_from", "MODELS", "MODEL_NAMES", "CHANNELS", "ACURITE_HZ", "tower_frame", "five_in_one_wind_rain", "five_in_one_weather", "lightning_frame", "frame_609", "frame_606", - "bursts", "Burst", "bits_pwm", "bits_ppm", "pulse_train", + "bursts", "Burst", "bits_pwm", "bits_ppm", "slicings", "pulse_train", "modulate", "baseband", "WIND_POINTS", "compass", "parity8", - "crc8", "VirtualSensor", "SimulatedSensors", "default_sensors"] + "crc8", "VirtualSensor", "SimulatedSensors", "default_sensors", + "survey", "Survey", "timings"] # Where every one of these sensors transmits. Nominally 433.92 MHz; the @@ -632,7 +633,10 @@ def decode_burst(burst: "Burst") -> Reading | None: burst where the edges have moved -- both readings of the same pulses are tried, and the checksums say which one it was. """ - return _best(candidates(bits_pwm(burst)) + candidates(bits_ppm(burst))) + found = [] + for bits in slicings(burst): + found += candidates(bits) + return _best(found) @@ -708,68 +712,182 @@ def bursts(envelope: np.ndarray, rate: float, gap_us: float = 3_000.0, min_level: float = 1.8) -> list[Burst]: """Split an envelope into the runs of pulses worth trying to read. - The threshold is set between the noise floor and the peak rather than at - a fixed level, because a sensor on the fence and a sensor three gardens - away differ by forty decibels and both should be read. The floor is the - median of the whole block, which is honest here in a way it usually is - not: a block is a second long, a message is a fiftieth of that, and so - the median of a block genuinely is the sound of nothing happening. + This is done in two passes, and the reason is a garden with more than one + sensor in it. + + The first pass only asks where anything is happening at all, and asks it + against the noise rather than against the loudest thing in the block. + That distinction is the whole point. A single threshold set halfway + between the noise floor and the peak of a whole second sounds reasonable + and is not: a sensor on the windowsill and a sensor at the end of the + garden differ by forty decibels, so a threshold set halfway to the near + one sits far above everything the far one ever does, and the far one + disappears -- not weakly, but completely, and only while the near one is + transmitting. A block with one loud sensor in it would yield exactly one + sensor however many were out there. + + So the gate here is the noise floor plus a few times its own scatter, + which is a level the quietest readable burst still clears and noise does + not. Whatever clears it is grouped into regions, and the second pass + then re-thresholds each region against its own high and low. Every + sensor is sliced at its own amplitude, and a burst forty decibels down on + its neighbour is read exactly as well as the neighbour is. """ if envelope.size < 4: return [] - floor = float(np.median(envelope)) - peak = float(np.percentile(envelope, 99.95)) + smooth = _smoothed(envelope, rate) + floor = float(np.median(smooth)) + peak = float(np.percentile(smooth, 99.99)) if peak <= floor * min_level or peak <= 0.0: return [] # nothing above the noise worth slicing - # Smoothed over a fraction of the shortest pulse these sensors send, so - # that a sample of noise on the wrong side of the threshold does not - # become a pulse, and the shortest real pulse still survives it. - span = max(1, int(round(rate * 30e-6))) - if span > 1: - pad = np.concatenate(([0.0], np.cumsum(envelope, dtype=np.float64))) - smooth = ((pad[span:] - pad[:-span]) / span).astype(np.float32) - else: - smooth = envelope - on = smooth > (floor + 0.5 * (peak - floor)) - if not on.any(): + hot = smooth > _noise_gate(smooth, floor, peak) + if not hot.any(): return [] - per_us = rate / 1e6 + out: list[Burst] = [] + for lo, hi in _regions(hot, int(round(gap_us * per_us))): + out += _slice(smooth, lo, hi, rate, gap_us, min_run_us, floor) + return [b for b in out if b.pulses >= min_pulses] + + +def _smoothed(envelope: np.ndarray, rate: float) -> np.ndarray: + """A running mean over a fraction of the shortest pulse these send. + + Enough that a sample of noise on the wrong side of a threshold cannot + become a pulse, and little enough that the shortest real pulse -- about + two hundred microseconds -- survives it with its edges where they were. + """ + span = max(1, int(round(rate * 30e-6))) + if span <= 1: + return envelope + pad = np.concatenate(([0.0], np.cumsum(envelope, dtype=np.float64))) + return ((pad[span:] - pad[:-span]) / span).astype(np.float32) + + +def _noise_gate(smooth: np.ndarray, floor: float, peak: float) -> float: + """The level below which nothing is worth looking at. + + This only has to find where something happened; how loud it was is + settled afterwards, region by region. So it wants an estimate of the + noise and nothing else, and in particular it must not move when + something loud happens -- that is exactly the case where a sensor at the + end of the garden is about to be lost. + + Everything here is read off the bottom of the distribution rather than + the middle of it. A second of band holds a few hundred milliseconds of + sensor at the very most, so the lowest fifth of it is noise however busy + the rest was. The middle is not safe in the same way: the median and the + deviation about it both climb once a fair fraction of the block is + signal, and a gate built on them can end up above the peak and pass + nothing at all. + + The other two terms are floors under the gate, for a signal so clean that + the noise has almost no scatter to measure -- a synthetic envelope, or a + receiver with nothing connected -- where the first term alone would be + zero and everything would be above it. + + Note what none of the three terms is: a fraction of the way to the peak. + That is the obvious way to write this and it is the bug it replaced. Any + gate proportional to the loudest thing in the block rises when a sensor + close to the aerial transmits, and rises past every quieter sensor out + there -- so the far ones vanish, and vanish only while the near one is + talking, which is as confusing a symptom as radio produces. + """ + quiet = float(np.percentile(smooth, 20)) + scatter = float(np.percentile(smooth, 40) - np.percentile(smooth, 10)) + return quiet + max(6.0 * scatter, 0.5 * quiet, 0.001 * (peak - quiet)) + + +def _regions(hot: np.ndarray, gap_samples: int) -> list[tuple[int, int]]: + """Spans of activity, with short silences inside them kept. + + The silences between the pulses of one message are part of the message, + so they must not end a region; only a silence longer than any gap within + a message does that. + """ + lengths, values, starts = _runs(hot) + spans: list[list[int]] = [] + for length, live, start in zip(lengths, values, starts): + if not live: + continue + if spans and start - spans[-1][1] <= gap_samples: + spans[-1][1] = start + length + else: + spans.append([start, start + length]) + return [(lo, hi) for lo, hi in spans] + + +def _slice(smooth: np.ndarray, lo: int, hi: int, rate: float, gap_us: float, + min_run_us: float, block_floor: float) -> list[Burst]: + """Cut one region into marks and spaces, at its own amplitude. + + The threshold is halfway between what this burst does when it is on and + what it does when it is off, both taken from the region itself. A little + of the silence either side is included so that there is an "off" to + measure even when the region is nearly all pulses. + """ + per_us = rate / 1e6 + margin = int(round(400 * per_us)) + lo = max(0, lo - margin) + hi = min(smooth.size, hi + margin) + part = smooth[lo:hi] + if part.size < 4: + return [] + off = float(np.percentile(part, 5)) + on_level = float(np.percentile(part, 98)) + if on_level <= off: + return [] + on = part > (off + on_level) / 2.0 + lengths, values, starts = _runs(on) lengths, values, starts = _merge_short(lengths, values, starts, min_run_us * per_us) - out: list[Burst] = [] marks: list[float] = [] spaces: list[float] = [] began = 0 + # A silence only becomes a bit's gap once another pulse follows it. The + # silence at the end of a burst is not part of the last bit -- it is the + # wait until the next copy, and it is as long as that wait happens to be + # -- so measuring the last bit against it reads the last bit wrongly, + # which is the last bit of the checksum, which is the whole message. + # Held back rather than appended, and dropped if nothing follows. + waiting: float | None = None for length, high, start in zip(lengths, values, starts): micro = length / per_us if high: if not marks: began = start + elif waiting is not None: + spaces.append(waiting) + waiting = None marks.append(micro) elif marks: if micro > gap_us: - out.append(_burst_of(marks, spaces, began, rate, floor, peak)) - marks, spaces = [], [] + out.append(_burst_of(marks, spaces, lo + began, rate, + block_floor, on_level)) + marks, spaces, waiting = [], [], None else: - spaces.append(micro) + waiting = micro if marks: - out.append(_burst_of(marks, spaces, began, rate, floor, peak)) - return [b for b in out if b.pulses >= min_pulses] + out.append(_burst_of(marks, spaces, lo + began, rate, block_floor, + on_level)) + return out def _burst_of(marks, spaces, began, rate, floor, peak) -> Burst: - # One more mark than spaces, always: the silence that ended the burst - # belongs to what came after it. - # - # Everything is cast to an ordinary float on the way out. These numbers - # came from numpy and end up as a timestamp in a log and in a file of - # names, and neither JSON nor YAML will write a numpy float -- which is - # the sort of thing that only shows up when a sensor is finally named at - # two in the morning. + """One burst, from the marks and spaces it was cut into. + + There is always one more mark than there are spaces: the silence that + ended the burst belongs to whatever comes after it, not to this. + + Everything is cast to an ordinary float on the way out. These numbers + came from numpy and end up as a timestamp in a log and in a file of + names, and neither JSON nor YAML will write a numpy float -- which is the + sort of thing that only shows up when a sensor is finally named at two in + the morning. + """ marks = [float(m) for m in marks][:len(spaces) + 1] return Burst(at=float(began) / float(rate), marks=tuple(marks), spaces=tuple(float(s) for s in spaces), @@ -860,6 +978,82 @@ def bits_ppm(burst: Burst) -> str: return "".join("1" if space > middle else "0" for space in burst.spaces) +def _levels(values, most: int = 3) -> list[float]: + """Thresholds that might separate short from long, best division first. + + The lengths in a burst fall into clusters -- two of them for the bits, + and often a third for the sync, which is longer than either. Rather than + decide which cluster boundary is the one that means something, this + returns a few of the widest gaps in the sorted lengths and lets the + checksums say. Guessing here is what a fixed threshold does, and a fixed + threshold is wrong the moment a transmitter warms up. + """ + ordered = sorted(float(v) for v in values) + if len(ordered) < 4: + return [] + span = ordered[-1] - ordered[0] + if span <= 0: + return [] + steps = sorted(((ordered[i + 1] - ordered[i], i) + for i in range(len(ordered) - 1)), reverse=True) + out = [] + for step, i in steps[:most]: + if step < 0.08 * span: + break # not a division, just jitter + out.append((ordered[i] + ordered[i + 1]) / 2.0) + return sorted(out) + + +_FLIP = str.maketrans("01", "10") + + +def _by_width(values, level: float) -> str: + return "".join("1" if value > level else "0" for value in values) + + +def slicings(burst: Burst) -> list[str]: + """Every way this burst might be read, for the checksums to choose from. + + There are two things not to assume about a burst, and both of them are + assumptions that hold on the sensor in front of you and fail on the next + one. + + The first is which of the pulse and the gap carries the bit. The newer + models vary the pulse; the two older ones vary the gap. + + The second is subtler and is the one that had this reading nothing at + all. Where the pulse carries the bit, the gap may be the complement of + it -- so that every bit takes the same time, and a pulse can be judged + against the gap that follows it -- or the gap may simply be a fixed + spacer. Judged against a fixed spacer of two hundred microseconds, a + short pulse of two hundred and twenty is longer than its gap and reads as + a one, which is the wrong bit, and every message fails its checksum with + nothing to say why. + + So nothing is assumed. The pulse against its own gap, the pulse against + each threshold the pulse lengths themselves suggest, and the same for the + gaps: half a dozen readings of the same burst, of which at most one will + satisfy a checksum. It costs a few microseconds per burst and it is the + difference between working on the sensors this was written against and + working on the ones in somebody's garden. + """ + out = [bits_pwm(burst), bits_ppm(burst)] + for values in (burst.marks, burst.spaces): + for level in _levels(values): + # And the other way up. Which of long and short means one is a + # third thing not to assume, and complementing a bit string is + # free. + reading = _by_width(values, level) + out += [reading, reading.translate(_FLIP)] + seen: set = set() + keep = [] + for bits in out: + if bits and bits not in seen: + seen.add(bits) + keep.append(bits) + return keep + + def readings_from(iq: np.ndarray, sample_rate: float, offset: float = 0.0, when: float = 0.0) -> list[Reading]: """Every distinct sensor message in one block of samples, in order. @@ -879,7 +1073,17 @@ def readings_from(iq: np.ndarray, sample_rate: float, offset: float = 0.0, envelope, rate = baseband(iq, sample_rate, offset) found = [] for burst in bursts(envelope, rate): - for reading in candidates(bits_pwm(burst)) + candidates(bits_ppm(burst)): + # One burst is one copy of one message, however many ways of reading + # it happened to produce it. Counted per reading rather than per + # slicing, or two readings of the same burst would corroborate each + # other -- and corroboration is the only thing standing between the + # two thinly-checked models and a display full of sensors that are + # not there. + here: dict = {} + for bits in slicings(burst): + for reading in candidates(bits): + here.setdefault((reading.family, reading.bits), reading) + for reading in here.values(): reading.at = when + burst.at found.append(reading) return confirmed(found) @@ -905,6 +1109,82 @@ def confirmed(found: list[Reading]) -> list[Reading]: return sorted(out, key=lambda r: (r.at, r.family, r.sensor)) +# --------------------------------------------------------------------------- +# Looking at what actually arrived +# --------------------------------------------------------------------------- + +def timings(values, tolerance: float = 0.25) -> list[tuple[float, int]]: + """The distinct lengths in a burst, each with how many there were. + + A message drawn as pulses has two or three lengths in it and nothing in + between, so this is the shape of the protocol written out: (220, 28) + (402, 28) (598, 4) is a burst of fifty-six bits and four sync pulses, + and anyone who knows the sensor can see that at a glance. It is the + single most useful thing to print when nothing is decoding, because it + says whether the trouble is the radio or the arithmetic. + """ + ordered = sorted(float(v) for v in values) + groups: list[list[float]] = [] + for value in ordered: + if groups and value <= groups[-1][-1] * (1.0 + tolerance): + groups[-1].append(value) + else: + groups.append([value]) + return [(sum(g) / len(g), len(g)) for g in groups] + + +@dataclass +class Survey: + """What one block of samples looked like at every stage. + + Only ever built to be printed. It exists because "nothing was heard" is + four different faults wearing the same coat -- no signal at all, a signal + below the gate, a burst that sliced into the wrong number of pulses, or + bits that came out and failed their checksums -- and they want four + different answers. Each stage here separates one of them from the next. + """ + + quiet: float = 0.0 # the noise, as the gate measures it + gate: float = 0.0 # what a burst has to clear + peak: float = 0.0 # the loudest thing in the block + rate: float = 0.0 # what the envelope was sliced at + seen: list = field(default_factory=list) # (burst, slicings, framed) + readings: list = field(default_factory=list) + + @property + def loudest(self) -> float: + return self.peak / self.quiet if self.quiet > 0 else float("inf") + + +def survey(iq: np.ndarray, sample_rate: float, offset: float = 0.0, + when: float = 0.0) -> Survey: + """One block, taken apart stage by stage, for when nothing decodes. + + The readings are obtained by running the ordinary path rather than by + repeating it here, so that what this reports and what the program does + cannot come apart -- a diagnostic that disagrees with the thing it is + diagnosing is worse than none. + """ + envelope, rate = baseband(iq, sample_rate, offset) + smooth = _smoothed(envelope, rate) + floor = float(np.median(smooth)) + peak = float(np.percentile(smooth, 99.99)) if smooth.size else 0.0 + quiet = float(np.percentile(smooth, 20)) if smooth.size else 0.0 + seen = [] + for burst in bursts(envelope, rate): + tries = slicings(burst) + # Without the corroboration rule, so that a message which framed + # correctly and merely arrived once is reported as what it is, + # rather than as silence. + framed = _best([r for bits in tries for r in candidates(bits)], + confirm=False) + seen.append((burst, tries, framed)) + return Survey(quiet=quiet, + gate=_noise_gate(smooth, floor, peak) if smooth.size else 0.0, + peak=peak, rate=rate, seen=seen, + readings=readings_from(iq, sample_rate, offset, when)) + + # --------------------------------------------------------------------------- # Building the same messages, so the decoder can be held to them # --------------------------------------------------------------------------- diff --git a/bandsaunter/cli.py b/bandsaunter/cli.py index 0150878..728c6b3 100755 --- a/bandsaunter/cli.py +++ b/bandsaunter/cli.py @@ -325,6 +325,17 @@ examples: help="print every message as it arrives, not a table") we.add_argument("--no-messages", dest="messages", action="store_false", default=None, help="show the table that updates in place") + we.add_argument("--diagnose", dest="diagnose", action="store_true", + default=None, + help="say what each second of band looked like at every " + "stage, for when sensors you know are there are not " + "appearing") + we.add_argument("--no-diagnose", dest="diagnose", action="store_false", + default=None, help="the ordinary display") + we.add_argument("--save-iq", default=None, metavar="FILE", + help="also write the raw samples, for working out why " + "something will not decode (8 MB a second at the " + "default rate, so bound it with --seconds)") we.add_argument("--hold", type=float, default=None, metavar="SECONDS", help="how long a sensor stays on the display after its " "last message") @@ -1404,7 +1415,8 @@ def _weather_options(args, options): ("gain", "gain"), ("device", "device"), ("frequency", "frequency"), ("offset", "offset"), ("simulate", "simulate"), ("log_messages", "log"), - ("messages", "messages"), ("hold", "hold"), + ("messages", "messages"), ("diagnose", "diagnose"), + ("hold", "hold"), ("units", "units"), ("only_named", "only_named"), ("unknown", "unknown"), ("report", "report"), ("csv", "csv")): @@ -1478,7 +1490,10 @@ def cmd_weather(args) -> int: book = SensorBook() _name_sensors(book, args.name) heard = wx.listen(console, options, cfg.output_dir, log_path=args.log, - book=book) + book=book, save_iq=args.save_iq) + if not heard.sensors and not options.diagnose: + console.print("[grey62]nothing decoded — `bandsaunter weather " + "--diagnose` says which stage it stops at[/grey62]") return 0 if heard.sensors else 1 diff --git a/bandsaunter/weather.py b/bandsaunter/weather.py index c954b6c..0c27158 100644 --- a/bandsaunter/weather.py +++ b/bandsaunter/weather.py @@ -99,6 +99,7 @@ class WeatherOptions: seconds: float = 0.0 log: bool = True messages: bool = False + diagnose: bool = False hold: float = 1800.0 # -- sensors -------------------------------------------------------- @@ -217,6 +218,22 @@ OPTIONS: tuple[Setting, ...] = ( "or to watch individual receptions arrive while working out whether " "an aerial is any good.", flags=("--messages",), off_flags=("--no-messages",)), + O("diagnose", "Say what is arriving", "Listening", "bool", + "print what each second of band looked like at every stage", + "For when nothing is being heard, which is four different faults " + "wearing the same coat. Each second is reported as the noise level, " + "the level a burst has to clear, the loudest thing in the block, and " + "then every burst found with the lengths of its pulses and gaps and " + "whatever was made of them. Between them those say whether the " + "trouble is that nothing is arriving, that it is too quiet to slice, " + "that it sliced into the wrong shape, or that the bits came out and " + "failed their checksums. Turns the table off, because the two cannot " + "share a screen.", + flags=("--diagnose",), off_flags=("--no-diagnose",), + guidance="Turn this on when sensors you know are there are not " + "appearing. The line of pulse lengths is the useful part: a " + "real message has two or three lengths in it and nothing in " + "between."), O("hold", "Keep on screen for", "Listening", "float", "how long a sensor stays on the display after its last message", "Half an hour by default, which is long compared with the sixteen " @@ -492,6 +509,11 @@ class Heard: book: object = None log_path: Path | None = None csv_path: Path | None = None + iq_path: Path | None = None + # What the diagnosis saw, if it was asked for: how many blocks, how many + # of them held anything, how many bursts came out and how far each got. + survey: dict = field(default_factory=lambda: dict( + blocks=0, dead=0, loud=0, bursts=0, framed=0, reported=0)) named: int = 0 # how many were given a name while listening @property @@ -547,7 +569,7 @@ def open_log(console, options: WeatherOptions, output_dir: str, def pump(device, options: WeatherOptions, garden: Garden, book, log, - started: float, on_block=None, on_reading=None, + started: float, on_block=None, on_reading=None, on_samples=None, stopping=None) -> int: """Read the receiver until it stops, or until told to. @@ -566,6 +588,8 @@ def pump(device, options: WeatherOptions, garden: Garden, book, log, samples = device.read_samples(block) if samples is None or samples.size == 0: break + if on_samples is not None: + on_samples(samples, at) for reading in readings_from(samples, options.rate, options.offset, when=at): if not reading.measures and not options.unknown: @@ -586,8 +610,68 @@ def pump(device, options: WeatherOptions, garden: Garden, book, log, return total +def _survey_line(console, options: WeatherOptions, samples, at: float, + started: float, tally: dict) -> None: + """One second of band, taken apart stage by stage. + + Printed rather than summarised because the person reading it is trying to + find out which stage is the one that fails, and a summary would be this + with the answer removed. + """ + from .acurite import survey, timings + + look = survey(samples, options.rate, options.offset, when=at) + console.print( + f"[grey62]{at - started:7.1f}s noise {look.quiet:.4f} " + f"gate {look.gate:.4f} peak {look.peak:.4f} " + f"({look.loudest:.1f}x the noise) " + f"{len(look.seen)} burst{'' if len(look.seen) == 1 else 's'}[/grey62]", + highlight=False) + for burst, tries, framed in look.seen: + marks = " ".join(f"{v:.0f}×{n}" for v, n in timings(burst.marks)) + gaps = " ".join(f"{v:.0f}×{n}" for v, n in timings(burst.spaces)) + console.print(f" [cyan]{burst.pulses:3d} pulses[/cyan] " + f"pulses {marks} gaps {gaps}", highlight=False) + if framed is None: + console.print(f" [yellow]nothing framed[/yellow] " + f"[grey62]{len(tries)} ways of reading it tried; " + f"the pulse lengths above are what to look at" + f"[/grey62]", highlight=False) + elif not look.readings: + console.print(f" [yellow]{framed.describe(options.imperial)}" + f"[/yellow] [grey62]framed, but this model needs " + f"the same message twice and it came once[/grey62]", + highlight=False) + else: + console.print(f" [green]{framed.describe(options.imperial)}" + f"[/green]", highlight=False) + tally["blocks"] += 1 + tally["dead"] += look.peak <= 0.0 + tally["loud"] += look.loudest > 3.0 + tally["bursts"] += len(look.seen) + tally["framed"] += sum(1 for _b, _t, f in look.seen if f is not None) + tally["reported"] += len(look.readings) + + +def _open_capture(console, path, options: WeatherOptions): + """The raw-sample file, or None. Says how fast it will fill.""" + if not path: + return None + where = Path(path).expanduser() + try: + where.parent.mkdir(parents=True, exist_ok=True) + handle = open(where, "wb") + except OSError as exc: + console.print(f"[red]cannot write {where}: {exc}[/red]") + return None + console.print(f"[yellow]writing raw samples to {where} — " + f"{options.rate * 8 / 1e6:.0f} MB a second, so bound it " + f"with --seconds[/yellow]") + return handle + + def listen(console, options: WeatherOptions, output_dir: str, - log_path=None, book=None) -> Heard: + log_path=None, book=None, save_iq=None) -> Heard: """Park on 433.92 MHz and write down what the neighbourhood says. Everything heard goes into the log as it arrives, and the names go into @@ -612,10 +696,20 @@ def listen(console, options: WeatherOptions, output_dir: str, if log is not None: console.print(f"[grey62]writing {log.path}[/grey62]") + capture = _open_capture(console, save_iq, options) display, live = _open_display(console, options, book, started) if live is not None: console.print() + def on_samples(samples, at: float) -> None: + if capture is not None: + try: + samples.astype("complex64").tofile(capture) + except OSError as exc: + console.print(f"[red]cannot write the samples: {exc}[/red]") + if options.diagnose: + _survey_line(console, options, samples, at, started, heard.survey) + def on_block(total: int) -> None: if live is None: return @@ -623,6 +717,8 @@ def listen(console, options: WeatherOptions, output_dir: str, live.update(display.render()) def on_reading(reading, sensor) -> None: + if options.diagnose: + return # the survey already said so if options.messages or live is None: label = sensor.name or reading.sensor console.print(f"[cyan]{label}[/cyan] " @@ -632,7 +728,7 @@ def listen(console, options: WeatherOptions, output_dir: str, total = 0 try: total = _run(console, device, options, garden, book, log, started, - display, live, on_block, on_reading, heard) + display, live, on_block, on_reading, on_samples, heard) except KeyboardInterrupt: pass finally: @@ -640,6 +736,9 @@ def listen(console, options: WeatherOptions, output_dir: str, live.stop() console.print("[grey62]stopped listening[/grey62]") device.close() + if capture is not None: + capture.close() + heard.iq_path = Path(save_iq).expanduser() if log is not None: log.close() heard.log_path = log.path @@ -648,7 +747,7 @@ def listen(console, options: WeatherOptions, output_dir: str, def _run(console, device, options, garden, book, log, started, - display, live, on_block, on_reading, heard) -> int: + display, live, on_block, on_reading, on_samples, heard) -> int: """The listening loop, with the keyboard live where there is one. Naming happens here rather than afterwards because that is the moment it @@ -660,7 +759,8 @@ def _run(console, device, options, garden, book, log, started, if live is None: return pump(device, options, garden, book, log, started, - on_block=on_block, on_reading=on_reading) + on_block=on_block, on_reading=on_reading, + on_samples=on_samples) with KeyReader() as keys: pressed = {"key": ""} @@ -685,7 +785,8 @@ def _run(console, device, options, garden, book, log, started, return pressed["key"] == "stop" return pump(device, options, garden, book, log, started, - on_block=block, on_reading=on_reading, stopping=stopping) + on_block=block, on_reading=on_reading, + on_samples=on_samples, stopping=stopping) def _name_one(console, keys, live, display, book, garden, @@ -745,7 +846,8 @@ def _open_display(console, options: WeatherOptions, book, started: float): display that redraws itself twice a second is unreadable as a stream of text, and worse than useless in a file. """ - if options.messages or not getattr(console, "is_terminal", False): + if options.messages or options.diagnose \ + or not getattr(console, "is_terminal", False): return None, None from rich.live import Live @@ -776,11 +878,22 @@ def finish(console, options: WeatherOptions, output_dir: str, heard: Heard, book.flush() except OSError as exc: console.print(f"[red]cannot save {book.path}: {exc}[/red]") + # Before the early return below, because the run that heard nothing is + # exactly the run whose verdict is worth reading. + if options.diagnose: + _verdict(console, heard) + if heard.iq_path is not None: + console.print(f"[grey62]raw samples in {heard.iq_path} " + f"({heard.iq_path.stat().st_size / 1e6:.0f} MB of " + f"complex64 at {options.rate / 1e6:g} MS/s, tuned " + f"{options.offset / 1e3:g} kHz below " + f"{options.frequency / 1e6:g} MHz)[/grey62]") if garden is None or not len(garden): - console.print("[yellow]nothing heard. These sensors are a few " - "milliwatts at 433.92 MHz: a quarter-wave whip is " - "17 cm, and indoors is usually the problem." - "[/yellow]") + if not options.diagnose: + console.print("[yellow]nothing heard. These sensors are a few " + "milliwatts at 433.92 MHz: a quarter-wave whip is " + "17 cm, and indoors is usually the problem." + "[/yellow]") return heard if options.report: @@ -802,6 +915,53 @@ def finish(console, options: WeatherOptions, output_dir: str, heard: Heard, return heard +def _verdict(console, heard: Heard) -> None: + """Which of the four faults it looks like, said once at the end. + + Said at the end rather than on every quiet second, because a line an hour + is a diagnosis and a line a second is weather of its own. + """ + from rich.panel import Panel + from rich.text import Text + + tally = heard.survey + if not tally["blocks"]: + return + if tally["dead"] == tally["blocks"]: + verdict = ("The receiver returned empty samples for every second of " + "this. Nothing was decoded because nothing arrived at all " + "\u2014 check `bandsaunter devices`.") + elif not tally["bursts"] and not tally["loud"]: + verdict = ("Nothing ever came above the noise. That is an aerial " + "rather than a decoder: these sensors are a few " + "milliwatts, a quarter-wave whip for 433.92 MHz is 17 cm " + "of wire, and indoors behind a wall is usually the " + "problem. Try --gain 40 as well.") + elif not tally["bursts"]: + verdict = (f"Something was above the noise in {tally['loud']} of " + f"{tally['blocks']} seconds and never grouped into a " + "burst, which usually means a transmitter that is on " + "continuously rather than one keying a carrier on and " + "off. Not one of these sensors.") + elif not tally["framed"]: + verdict = (f"{tally['bursts']} bursts were received and sliced, and " + "not one of them framed as a message. The radio end is " + "working: what is arriving is a model this cannot read, " + "or reads differently. The pulse lengths printed above " + "are exactly what is needed to add it.") + elif not tally["reported"]: + verdict = (f"{tally['framed']} of {tally['bursts']} bursts framed " + "correctly and none was reported, which means each " + "arrived once. The two older models are believed only on " + "a second copy of the same message, so this wants a " + "stronger signal rather than a different decoder.") + else: + verdict = (f"{tally['reported']} readings out of {tally['bursts']} " + f"bursts over {tally['blocks']} seconds. Working.") + console.print(Panel(Text(verdict), title="[bold]what that came to", + border_style="blue", padding=(0, 1))) + + def _write_csv(console, heard: Heard, book, options: WeatherOptions): from .weatherlog import read_logs, write_csv diff --git a/packaging/bandsaunter.1 b/packaging/bandsaunter.1 index 795aa4d..ed0f50c 100644 --- a/packaging/bandsaunter.1 +++ b/packaging/bandsaunter.1 @@ -1,5 +1,5 @@ .\" Generated by packaging/make-man.py -- do not edit by hand. -.TH BANDSAUNTER 1 "2026-09-07" "bandsaunter 2026-09-07_01" "User Commands" +.TH BANDSAUNTER 1 "2026-09-07" "bandsaunter 2026-09-07_02" "User Commands" .SH NAME bandsaunter \- scan, record and identify radio signals with an RTL-SDR .SH SYNOPSIS @@ -2277,13 +2277,28 @@ A running average of the complex samples then rejects the spike, and only after that is the magnitude taken \[em] filtering before detection rather than after is what keeps the neighbours out of the envelope of the sensor. .PP -Slicing the envelope into bits never measures anything against a clock. The -newer sensors vary the length of the pulse and keep the gaps even; the two -older ones keep the pulse even and vary the gap. Both readings of the same -pulses are tried and the checksums say which it was. A transmitter running ten -per cent fast is read correctly and never noticed, which matters: these are -unlocked and drift with the temperature, and an outdoor sensor in January is -not the one that was on the fence in July. +Finding the bursts is done in two passes, and the reason is having more than +one sensor. The first pass asks only where anything is happening at all, and +asks it against the noise: the bottom fifth of a second, which is noise +however busy the rest was. Whatever clears that is grouped into regions, and +the second pass re-thresholds each region against its own high and low, so +every sensor is sliced at its own amplitude. One threshold per second, set +halfway between the noise and the loudest thing in it, is the obvious way to +write this and is wrong: a sensor on the windowsill and a sensor at the end of +the garden differ by forty decibels, so the far ones fall below it and vanish, +and vanish only while the near one is transmitting. +.PP +Slicing the envelope into bits never measures anything against a clock, and +assumes as little as it can about how a bit is drawn. Not which of the pulse +and the gap carries the bit; not whether the gap is the complement of the +pulse or a fixed spacer, since a 220 microsecond pulse against a 200 +microsecond spacer is the longer of the two and reads as the wrong bit; and +not which of long and short means one. The same burst is read half a dozen +ways and the checksums say which reading it was, at most one of them being +able to satisfy one. A transmitter running ten per cent fast is therefore read +correctly and never noticed, which matters: these are unlocked and drift with +the temperature, and an outdoor sensor in January is not the one that was on +the fence in July. .SS Afterwards When the listening stops, two tables. The first is about reception \[em] who, how often, how well \[em] and is the one to look at when something is missing: @@ -2320,14 +2335,45 @@ transmitting real messages with real checksums, keyed on and off as a real one does, through the real filter, the real slicer and the real decoders. Nothing touches the receiver. .SS If nothing is heard -These are a few milliwatts. A quarter-wave whip for 433.92 MHz is 17 cm of -wire, and the stock telescopic aerial set to about that length works well; -indoors, behind a wall, with the dongle in the back of a machine, is usually -the problem. Try +.B "bandsaunter weather \-\-diagnose" +is the answer to this, because "nothing was heard" is four different faults +wearing the same coat and they want four different answers. It prints each +second of band taken apart stage by stage: the noise level, the level a burst +has to clear, the loudest thing in the block, and then every burst found with +the lengths of its pulses and gaps and whatever was made of them. +.TP +.B "peak barely above noise, no bursts" +Nothing is arriving, which is an aerial. These are a few milliwatts; a +quarter-wave whip for 433.92 MHz is 17 cm of wire, which is the stock +telescopic aerial collapsed to about that, and indoors behind a wall with the +dongle in the back of a machine is usually the problem. Try .B \-\-gain 40 -if the automatic gain control is not finding them, and -.B \-\-messages -to watch individual receptions arrive while moving the aerial about. +if the automatic gain control is not finding them. +.TP +.B "peak well above noise, no bursts" +Something is there and did not group into a burst, usually a transmitter that +is on continuously rather than keyed. Not one of these. +.TP +.B "bursts whose pulse lengths are not two or three clean groups" +The receiver is hearing it and the slicing is wrong. A real message shows two +or three lengths with nothing in between; a smear means noise is being sliced +as signal, or two sensors are transmitting over each other. +.TP +.B "clean pulse lengths, nothing framed" +The radio is fine and the message is from a model this does not read. The line +of pulse lengths is what is needed to add it. +.TP +.B "framed, but needs the same message twice" +It was read correctly and arrived once. The two older models are believed only +on a second copy, so this wants a stronger signal. +.PP +When the listening stops it says which of those five it was, once, rather than +on every quiet second. +.PP +.BI \-\-save\-iq " FILE" +writes the raw samples alongside, for anything the diagnosis cannot settle. It +is 8 MB a second at the default rate, so bound it with +.BR \-\-seconds . .SH WEATHER OPTIONS Every option the weather side takes, in the four groups the menu shows them in. Each is a flag here and a line in the menu, and both come from one table @@ -2403,6 +2449,15 @@ Print every message \[em] one line per message instead of a table that updates i .br Setting name \fBmessages\fR, default \fBno\fR. .TP +.B --diagnose / --no-diagnose +Say what is arriving \[em] print what each second of band looked like at every stage. +.br +Setting name \fBdiagnose\fR, default \fBno\fR. +.RS +.PP +Turn this on when sensors you know are there are not appearing. The line of pulse lengths is the useful part: a real message has two or three lengths in it and nothing in between. +.RE +.TP .B --hold Keep on screen for \[em] how long a sensor stays on the display after its last message (s). .br diff --git a/packaging/make-man.py b/packaging/make-man.py index 1e1ba4d..deb2b13 100755 --- a/packaging/make-man.py +++ b/packaging/make-man.py @@ -1389,13 +1389,28 @@ A running average of the complex samples then rejects the spike, and only after that is the magnitude taken \[em] filtering before detection rather than after is what keeps the neighbours out of the envelope of the sensor. .PP -Slicing the envelope into bits never measures anything against a clock. The -newer sensors vary the length of the pulse and keep the gaps even; the two -older ones keep the pulse even and vary the gap. Both readings of the same -pulses are tried and the checksums say which it was. A transmitter running ten -per cent fast is read correctly and never noticed, which matters: these are -unlocked and drift with the temperature, and an outdoor sensor in January is -not the one that was on the fence in July. +Finding the bursts is done in two passes, and the reason is having more than +one sensor. The first pass asks only where anything is happening at all, and +asks it against the noise: the bottom fifth of a second, which is noise +however busy the rest was. Whatever clears that is grouped into regions, and +the second pass re-thresholds each region against its own high and low, so +every sensor is sliced at its own amplitude. One threshold per second, set +halfway between the noise and the loudest thing in it, is the obvious way to +write this and is wrong: a sensor on the windowsill and a sensor at the end of +the garden differ by forty decibels, so the far ones fall below it and vanish, +and vanish only while the near one is transmitting. +.PP +Slicing the envelope into bits never measures anything against a clock, and +assumes as little as it can about how a bit is drawn. Not which of the pulse +and the gap carries the bit; not whether the gap is the complement of the +pulse or a fixed spacer, since a 220 microsecond pulse against a 200 +microsecond spacer is the longer of the two and reads as the wrong bit; and +not which of long and short means one. The same burst is read half a dozen +ways and the checksums say which reading it was, at most one of them being +able to satisfy one. A transmitter running ten per cent fast is therefore read +correctly and never noticed, which matters: these are unlocked and drift with +the temperature, and an outdoor sensor in January is not the one that was on +the fence in July. .SS Afterwards When the listening stops, two tables. The first is about reception \[em] who, how often, how well \[em] and is the one to look at when something is missing: @@ -1432,14 +1447,45 @@ transmitting real messages with real checksums, keyed on and off as a real one does, through the real filter, the real slicer and the real decoders. Nothing touches the receiver. .SS If nothing is heard -These are a few milliwatts. A quarter-wave whip for 433.92 MHz is 17 cm of -wire, and the stock telescopic aerial set to about that length works well; -indoors, behind a wall, with the dongle in the back of a machine, is usually -the problem. Try +.B "bandsaunter weather \-\-diagnose" +is the answer to this, because "nothing was heard" is four different faults +wearing the same coat and they want four different answers. It prints each +second of band taken apart stage by stage: the noise level, the level a burst +has to clear, the loudest thing in the block, and then every burst found with +the lengths of its pulses and gaps and whatever was made of them. +.TP +.B "peak barely above noise, no bursts" +Nothing is arriving, which is an aerial. These are a few milliwatts; a +quarter-wave whip for 433.92 MHz is 17 cm of wire, which is the stock +telescopic aerial collapsed to about that, and indoors behind a wall with the +dongle in the back of a machine is usually the problem. Try .B \-\-gain 40 -if the automatic gain control is not finding them, and -.B \-\-messages -to watch individual receptions arrive while moving the aerial about. +if the automatic gain control is not finding them. +.TP +.B "peak well above noise, no bursts" +Something is there and did not group into a burst, usually a transmitter that +is on continuously rather than keyed. Not one of these. +.TP +.B "bursts whose pulse lengths are not two or three clean groups" +The receiver is hearing it and the slicing is wrong. A real message shows two +or three lengths with nothing in between; a smear means noise is being sliced +as signal, or two sensors are transmitting over each other. +.TP +.B "clean pulse lengths, nothing framed" +The radio is fine and the message is from a model this does not read. The line +of pulse lengths is what is needed to add it. +.TP +.B "framed, but needs the same message twice" +It was read correctly and arrived once. The two older models are believed only +on a second copy, so this wants a stronger signal. +.PP +When the listening stops it says which of those five it was, once, rather than +on every quiet second. +.PP +.BI \-\-save\-iq " FILE" +writes the raw samples alongside, for anything the diagnosis cannot settle. It +is 8 MB a second at the default rate, so bound it with +.BR \-\-seconds . .SH WEATHER OPTIONS Every option the weather side takes, in the four groups the menu shows them in. Each is a flag here and a line in the menu, and both come from one table diff --git a/tests/test_acurite.py b/tests/test_acurite.py index 13c590e..0fa06bf 100644 --- a/tests/test_acurite.py +++ b/tests/test_acurite.py @@ -540,3 +540,205 @@ def test_a_nine_byte_sensor_that_is_not_a_lightning_detector_is_not_read_as_one( assert got.measures == () assert got.sensor == "0011" assert got.value("temperature") is None + + +# --------------------------------------------------------------------------- +# A garden with more than one sensor in it +# --------------------------------------------------------------------------- + +def block_of(*bursts_in, seconds: float = 1.0, noise: float = 0.02, + rate: float = RATE, offset: float = OFFSET, seed: int = 0): + """One second of band with whatever was handed in placed about in it.""" + rng = np.random.default_rng(seed) + n = int(rate * seconds) + block = ((rng.standard_normal(n) + 1j * rng.standard_normal(n)) + * noise).astype(np.complex64) + for at, part in bursts_in: + start = int(at * rate) + room = min(part.size, max(0, n - start)) + block[start:start + room] += part[:room] + return block + + +def keyed(bits, amplitude: float = 1.0, rate: float = RATE, + offset: float = OFFSET): + return a.modulate(bits, rate, offset=offset, amplitude=amplitude, + lead_us=0.0, noise=0.0) + + +def test_a_sensor_by_the_aerial_does_not_hide_the_rest_of_the_garden(): + """The fault that had this reading one sensor out of six. + + A threshold set halfway between the noise and the loudest thing in the + block is halfway to whichever sensor happens to be nearest, and every + quieter sensor is then below it -- so they disappear, and disappear only + while the near one is transmitting, which is as confusing a symptom as + radio produces. Thirty-six decibels between these two. + """ + loud = keyed(a.tower_frame(0x1A2B, 21.5, 48, "A"), amplitude=8.0) + faint = keyed(a.tower_frame(0x0C41, 3.2, 91, "B"), amplitude=0.12) + block = block_of((0.05, loud), (0.5, faint)) + heard_now = {r.sensor for r in a.readings_from(block, RATE, offset=OFFSET)} + assert heard_now == {"1A2B", "0C41"} + + +@pytest.mark.parametrize("apart", [4.0, 20.0, 80.0]) +def test_two_sensors_are_both_read_however_far_apart_in_strength(apart): + loud = keyed(a.tower_frame(0x1A2B, 21.5, 48, "A"), amplitude=0.9) + faint = keyed(a.tower_frame(0x0C41, 3.2, 91, "B"), amplitude=0.9 / apart) + block = block_of((0.05, loud), (0.5, faint), noise=0.9 / apart / 12.0) + got = {r.sensor for r in a.readings_from(block, RATE, offset=OFFSET)} + assert got == {"1A2B", "0C41"}, f"{apart:g}x apart: heard {got}" + + +def test_six_sensors_in_one_second_all_come_back(): + parts = [(0.02 + i * 0.14, + keyed(a.tower_frame(0x100 + i, 10.0 + i, 50, "A"), + amplitude=0.15 * (i + 1))) + for i in range(6)] + got = {r.sensor for r in a.readings_from(block_of(*parts), RATE, + offset=OFFSET)} + assert got == {f"{0x100 + i:04X}" for i in range(6)} + + +# --------------------------------------------------------------------------- +# Not assuming how a bit is drawn +# --------------------------------------------------------------------------- + +def transmitted(bits, one_mark, zero_mark, gap, sync_mark=600.0, + sync_gap=600.0, syncs=4, copies=3, amplitude=1.0): + """A sensor keyed with whatever timings, rather than with mine.""" + per_us = RATE / 1e6 + parts = [np.zeros(int(3_000 * per_us), dtype=np.float32)] + + def push(mark, space): + parts.append(np.full(int(round(mark * per_us)), amplitude, + dtype=np.float32)) + parts.append(np.zeros(int(round(space * per_us)), dtype=np.float32)) + + for _ in range(copies): + for _ in range(syncs): + push(sync_mark, sync_gap) + for bit in bits: + push(one_mark if bit == "1" else zero_mark, gap) + parts.append(np.zeros(int(9_000 * per_us), dtype=np.float32)) + return a._to_air(np.concatenate(parts), RATE, OFFSET, 0.02, 0) + + +@pytest.mark.parametrize("name,one,zero,gap", [ + # The gap is the complement of the pulse, so every bit takes the same + # time. This is the one it was written against. + ("complementary gap", 408.0, 220.0, None), + # The gap is a fixed spacer. Judged against a 200 us spacer a 220 us + # pulse is the longer of the two and reads as a one, which is the wrong + # bit, and every message fails its checksum saying nothing about why. + ("short fixed gap", 408.0, 220.0, 200.0), + ("long fixed gap", 408.0, 220.0, 500.0), + # And the other way up: the short pulse is the one. + ("inverted", 220.0, 408.0, 200.0), +]) +def test_a_burst_is_read_whichever_way_the_bits_are_drawn(name, one, zero, + gap): + bits = a.tower_frame(0x1A2B, 21.5, 48, "A") + if gap is None: + # complementary: build it a bit at a time so each gap completes its + # own bit period + per_us = RATE / 1e6 + parts = [np.zeros(int(3_000 * per_us), dtype=np.float32)] + for _ in range(3): + for mark, space in zip(*a.pulse_train(bits, "pwm")): + parts.append(np.full(int(round(mark * per_us)), 1.0, + dtype=np.float32)) + parts.append(np.zeros(int(round(space * per_us)), + dtype=np.float32)) + parts.append(np.zeros(int(9_000 * per_us), dtype=np.float32)) + iq = a._to_air(np.concatenate(parts), RATE, OFFSET, 0.02, 0) + else: + iq = transmitted(bits, one, zero, gap) + got = a.readings_from(iq, RATE, offset=OFFSET) + assert [r.sensor for r in got] == ["1A2B"], f"{name}: heard {got}" + + +def test_the_readings_of_a_burst_are_all_different_from_each_other(): + """Half a dozen ways of reading it, and no duplicates among them.""" + burst = a.bursts(*a.baseband( + a.modulate(a.tower_frame(0x1A2B, 21.5, 48, "A"), RATE, offset=OFFSET, + noise=0.02), RATE, OFFSET))[0] + tries = a.slicings(burst) + assert len(tries) >= 4 + assert len(set(tries)) == len(tries) + assert a.tower_frame(0x1A2B, 21.5, 48, "A") in "".join(tries) + + +def test_reading_a_burst_several_ways_is_not_the_same_as_hearing_it_twice(): + """Corroboration counts messages, not readings of one message. + + The two thinly-checked models are believed when the same message arrives + twice. If two ways of reading one burst each produced it, that would + look like two arrivals and the rule would protect nothing. + """ + iq = a.modulate(a.frame_609(0x5C, 4.2, 80), RATE, coding="ppm", + offset=OFFSET, repeats=1, noise=0.02) + burst = a.bursts(*a.baseband(iq, RATE, OFFSET))[0] + ways = [r for bits in a.slicings(burst) for r in a.candidates(bits)] + assert any(r.family == "609" for r in ways), "it did frame" + assert a.readings_from(iq, RATE, offset=OFFSET) == [] + + +# --------------------------------------------------------------------------- +# Saying what arrived, when nothing decodes +# --------------------------------------------------------------------------- + +def test_the_survey_reports_the_same_readings_the_program_acts_on(): + """A diagnostic that disagrees with the thing it diagnoses is worse than + none, so it runs the ordinary path rather than repeating it.""" + iq = a.modulate(a.tower_frame(0x1A2B, 21.5, 48, "A"), RATE, offset=OFFSET, + noise=0.05) + look = a.survey(iq, RATE, OFFSET, when=1_000.0) + assert [r.describe() for r in look.readings] == \ + [r.describe() for r in a.readings_from(iq, RATE, offset=OFFSET, + when=1_000.0)] + + +def test_the_survey_separates_nothing_arriving_from_nothing_decoding(): + rng = np.random.default_rng(2) + n = int(RATE) + quiet = ((rng.standard_normal(n) + 1j * rng.standard_normal(n)) + * 0.02).astype(np.complex64) + nothing = a.survey(quiet, RATE, OFFSET) + assert nothing.seen == [] and nothing.readings == [] + assert nothing.loudest < 3.0 # and it says the band was quiet + + # A burst of the right shape whose bits are nonsense: it groups, it + # slices, and it frames nothing. A different fault, and it looks it. + rubbish = transmitted("01" * 28, 408.0, 220.0, 220.0) + junk = a.survey(rubbish, RATE, OFFSET) + assert junk.seen and junk.readings == [] + assert all(framed is None for _b, _t, framed in junk.seen) + assert junk.loudest > 3.0 + + +def test_the_survey_shows_a_message_that_framed_and_was_not_corroborated(): + """Which is a third fault again, and the one hardest to guess at.""" + iq = a.modulate(a.frame_609(0x5C, 4.2, 80), RATE, coding="ppm", + offset=OFFSET, repeats=1, noise=0.02) + look = a.survey(iq, RATE, OFFSET) + assert look.readings == [] + assert any(framed is not None and framed.family == "609" + for _b, _t, framed in look.seen) + + +def test_the_pulse_lengths_of_a_burst_are_reported_as_the_protocol_shape(): + burst = a.bursts(*a.baseband( + a.modulate(a.tower_frame(0x1A2B, 21.5, 48, "A"), RATE, offset=OFFSET, + noise=0.02), RATE, OFFSET))[0] + marks = a.timings(burst.marks) + assert len(marks) == 3 # short, long, sync + assert [n for _v, n in marks] == [28, 28, 4] + assert [round(v / 10) * 10 for v, _n in marks] == [220, 400, 600] + assert sum(n for _v, n in marks) == burst.pulses + + +def test_a_burst_of_one_length_is_reported_as_one_length(): + assert a.timings([400.0] * 12) == [(400.0, 12)] + assert len(a.timings([200.0] * 6 + [400.0] * 6)) == 2 diff --git a/tests/test_weather.py b/tests/test_weather.py index 1a3f808..9ca53e1 100644 --- a/tests/test_weather.py +++ b/tests/test_weather.py @@ -1073,3 +1073,174 @@ def test_a_waiting_name_does_not_get_handed_to_the_wrong_sensor(book): book.heard(reading(sensor=0x0C41, at=1_000.0)) assert book.name_for("tower/0C41") == "" assert book.name_for("?/1A2B") == "back fence" + + +# --------------------------------------------------------------------------- +# Saying what is arriving, for when nothing is +# --------------------------------------------------------------------------- + +def test_the_diagnosis_says_what_each_second_of_band_looked_like(tmp_path, + monkeypatch): + options = wx.WeatherOptions(rate=RATE, offset=OFFSET, diagnose=True, + log=False) + monkeypatch.setattr(wx, "open_device", lambda console, opts: Garden6(30)) + console = Console(width=140, force_terminal=False) + with console.capture() as cap: + wx.listen(console, options, str(tmp_path), + book=SensorBook(path=tmp_path / "s.yaml")) + out = cap.get() + assert "noise" in out and "gate" in out and "the noise" in out + assert "pulses" in out and "gaps" in out + # The pulse lengths are the useful part: two or three, and nothing + # in between. + assert "×" in out + assert "Tower 592TXR" in out + + +def test_the_diagnosis_turns_the_live_table_off(tmp_path): + """The two cannot share a screen: one redraws in place, one scrolls.""" + console = Console(width=120, force_terminal=True) + display, live = wx._open_display( + console, wx.WeatherOptions(diagnose=True), None, time.time()) + assert (display, live) == (None, None) + + +def test_raw_samples_can_be_captured_for_working_out_why(tmp_path, + monkeypatch): + options = wx.WeatherOptions(rate=RATE, offset=OFFSET, messages=True, + log=False) + monkeypatch.setattr(wx, "open_device", lambda console, opts: Garden6(3)) + where = tmp_path / "band.cf32" + console = Console(width=120, force_terminal=False) + with console.capture() as cap: + heard = wx.listen(console, options, str(tmp_path), + book=SensorBook(path=tmp_path / "s.yaml"), + save_iq=str(where)) + assert heard.iq_path == where and where.exists() + kept = np.fromfile(where, dtype=np.complex64) + assert kept.size == 3 * int(RATE) + # And it says how fast that fills, because it fills fast. + assert "MB a second" in cap.get() + + +def test_capturing_to_somewhere_unwritable_is_a_message_not_a_crash( + tmp_path, monkeypatch): + options = wx.WeatherOptions(rate=RATE, offset=OFFSET, messages=True, + log=False) + monkeypatch.setattr(wx, "open_device", lambda console, opts: Silence(2)) + console = Console(width=120, force_terminal=False) + with console.capture() as cap: + heard = wx.listen(console, options, str(tmp_path), + book=SensorBook(path=tmp_path / "s.yaml"), + save_iq="/proc/nowhere/band.cf32") + assert heard.iq_path is None + assert "cannot write" in cap.get() + + +def test_an_offset_too_big_for_the_sample_rate_is_refused_with_a_reason(): + """Half the sample rate is all the band there is to move a signal within.""" + errs = wx.WeatherOptions(rate=400_000.0, offset=250_000.0).validate() + assert any("offset" in e and "sample rate" in e for e in errs) + assert wx.WeatherOptions(rate=400_000.0, offset=100_000.0).validate() == [] + + +def test_hearing_nothing_points_at_the_diagnosis(tmp_path, monkeypatch): + from bandsaunter.cli import build_parser, cmd_weather + import bandsaunter.cli as cli + + monkeypatch.setattr(wx, "open_device", lambda console, opts: Silence(2)) + monkeypatch.setattr(wx, "load_options", + lambda *a, **kw: wx.WeatherOptions(rate=RATE, + offset=OFFSET, + messages=True, + log=False)) + console = Console(width=120, force_terminal=False) + monkeypatch.setattr(cli, "console", console) + args = build_parser().parse_args(["weather"]) + with console.capture() as cap: + assert cmd_weather(args) == 1 + assert "--diagnose" in cap.get() + + +@pytest.mark.parametrize("flags,key,value", [ + (["--diagnose"], "diagnose", True), + (["--no-diagnose"], "diagnose", False), +]) +def test_the_diagnosis_flags_reach_the_option(flags, key, value): + from bandsaunter.cli import _weather_options, build_parser + + args = build_parser().parse_args(["weather"] + flags) + assert getattr(_weather_options(args, wx.WeatherOptions()), key) == value + + +def test_save_iq_is_a_path_on_the_command_line_and_not_a_saved_setting(): + """It is a one-off capture, not something to carry between runs.""" + from bandsaunter.cli import build_parser + + args = build_parser().parse_args(["weather", "--save-iq", "/tmp/x.cf32"]) + assert args.save_iq == "/tmp/x.cf32" + assert not hasattr(wx.WeatherOptions(), "save_iq") + + +# --------------------------------------------------------------------------- +# The verdict: which of the four faults it was +# --------------------------------------------------------------------------- + +def verdict_of(**tally): + heard = wx.Heard() + heard.survey.update(tally) + console = Console(width=120, force_terminal=False) + with console.capture() as cap: + wx._verdict(console, heard) + return " ".join(cap.get().split()) + + +@pytest.mark.parametrize("tally,says", [ + # Nothing arrived at all: not the decoder's fault, and not the aerial's. + (dict(blocks=30, dead=30), "empty samples"), + # Nothing above the noise: the aerial. + (dict(blocks=30), "aerial"), + # Something there, never keyed: not one of these sensors. + (dict(blocks=30, loud=12), "continuously"), + # Sliced and never framed: the radio works, the protocol is not one of + # the five. This is the one where the pulse lengths matter. + (dict(blocks=30, loud=12, bursts=40), "cannot read"), + # Framed and never corroborated: a stronger signal, not a new decoder. + (dict(blocks=30, loud=12, bursts=40, framed=9), "second copy"), + (dict(blocks=30, loud=12, bursts=40, framed=9, reported=6), "Working"), +]) +def test_the_verdict_tells_the_four_faults_apart(tally, says): + assert says in verdict_of(**tally) + + +def test_the_verdict_says_nothing_when_there_was_nothing_to_judge(): + assert verdict_of() == "" + + +def test_the_verdict_is_given_even_when_nothing_was_heard(tmp_path, + monkeypatch): + """Which is exactly the run whose verdict is worth reading.""" + monkeypatch.setattr(wx, "open_device", lambda console, opts: Silence(3)) + console = Console(width=120, force_terminal=False) + with console.capture() as cap: + heard = wx.listen(console, wx.WeatherOptions(rate=RATE, offset=OFFSET, + diagnose=True, log=False), + str(tmp_path), + book=SensorBook(path=tmp_path / "s.yaml")) + assert heard.sensors == 0 + assert "what that came to" in cap.get() + assert "empty samples" in cap.get() + + +def test_a_working_run_is_counted_as_one(tmp_path, monkeypatch): + monkeypatch.setattr(wx, "open_device", lambda console, opts: Garden6(30)) + console = Console(width=120, force_terminal=False) + with console.capture() as cap: + heard = wx.listen(console, wx.WeatherOptions(rate=RATE, offset=OFFSET, + diagnose=True, log=False, + report=False), + str(tmp_path), + book=SensorBook(path=tmp_path / "s.yaml")) + assert "Working" in cap.get() + assert heard.survey["reported"] == heard.messages + assert heard.survey["blocks"] == 30