diff --git a/.gitignore b/.gitignore index e5a6c1a..d85aae5 100644 --- a/.gitignore +++ b/.gitignore @@ -16,3 +16,6 @@ recordings/ # local environments .venv/ venv/ + +# a personal helper, not part of the program +resume.sh diff --git a/INSTALL.md b/INSTALL.md index 2062e0f..baa9e40 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -59,6 +59,13 @@ model into a small apt repository under `dist/repo`, so that `apt install bandsaunter` on every other machine brings transcription with it. See **Install → From your own apt repository** in the README. +It needs two more tools than `build-deb.sh` does, neither of which is on a +minimal system: + +```bash +sudo apt install dpkg-dev apt-utils # dpkg-scanpackages and apt-ftparchive +``` + --- ## B. Anywhere else: a virtual environment @@ -80,8 +87,10 @@ python3 -m venv .venv pip install . # a few wheels; under a minute ``` -That installs four wheels — numpy, scipy, rich, PyYAML — and puts two commands -on the path, `bandsaunter` and `saunterbrowse`. Nothing is built from source. +That installs seven wheels — numpy, scipy, rich and PyYAML, plus the three +Rich brings with it (Pygments, markdown-it-py, mdurl) — and puts two commands +on the path, `bandsaunter` and `saunterbrowse`. Nothing is built from source, +and it takes under a minute. Use `pip install -e .` instead if you intend to change the code. @@ -123,26 +132,120 @@ if not. --- -## Optional dependencies +## Every dependency, in one table -Everything below is genuinely optional. The program starts, scans, records, -identifies, decodes Morse and draws waterfalls with none of it, and says -plainly when a feature is unavailable rather than failing. +**Required.** Four Python packages and one system library. The `.deb` and +`pip install` both pull the Python ones in; the system library is the only +thing either way that has to come from your distribution. + +| | Debian/Ubuntu/Mint | Fedora | Arch | what needs it | +|---|---|---|---|---| +| Python 3.10+ | `python3` | `python3` | `python` | everything | +| NumPy | `python3-numpy` | `python3-numpy` | `python-numpy` | every signal path, every picture | +| SciPy | `python3-scipy` | `python3-scipy` | `python-scipy` | filtering, demodulation, classifying | +| Rich | `python3-rich` | `python3-rich` | `python-rich` | the menus and the live display | +| PyYAML | `python3-yaml` | `python3-pyyaml` | `python-yaml` | the settings file | +| **librtlsdr** | `librtlsdr0` | `rtl-sdr` | `rtl-sdr` | **talking to the dongle** | + +**`librtlsdr` is the one manual dependency that matters.** It is a C library, +not a Python package, so `pip` cannot install it and no virtual environment +brings it with it. bandsaunter opens it with `ctypes` at runtime, trying +`librtlsdr.so.2`, `.so.0`, `.so`, `librtlsdr.dylib`, `rtlsdr.dll` and +`librtlsdr.dll` in that order. +Without it everything that does not touch the hardware still works — reading +logs, drawing maps, decoding recordings, the simulated sky — and anything that +does says so plainly instead of failing: + +```sh +sudo apt install librtlsdr0 # Debian, Ubuntu, Mint +sudo dnf install rtl-sdr # Fedora +sudo pacman -S rtl-sdr # Arch +brew install librtlsdr # macOS +``` + +**Optional.** Everything below is genuinely optional. The program starts, +scans, records, identifies, decodes Morse, draws waterfalls and maps with none +of it, and says plainly when a feature is unavailable rather than failing. | Install | Gives you | Without it | |---|---|---| -| `sudo apt install espeak-ng` | clearer spoken timestamps on combined recordings, rendered about three times faster | a built-in formant synthesiser does the same job, less clearly | +| `sudo apt install python3-pyqt6` | the realtime aircraft window (`bandsaunter adsb --window`) | the terminal board still shows every aircraft, and the maps are still drawn afterwards | | `sudo apt install ffmpeg` | `bandsaunter flights --out sky.mp4` writes a video | a GIF is written instead, and it says so | -| `sudo apt install rtl-sdr` | `rtl_test`, `rtl_sdr` and friends, for diagnosing hardware | nothing missing from bandsaunter itself | +| `sudo apt install espeak-ng` | clearer spoken timestamps on combined recordings, rendered about three times faster | a built-in formant synthesiser does the same job, less clearly | +| `sudo apt install rtl-sdr` | `rtl_test`, `rtl_sdr` and friends, for diagnosing the hardware | nothing missing from bandsaunter itself | | `pip install faster-whisper` | speech transcription of recorded voice — see [the step-by-step below](#speech-transcription-step-by-step) (~250 MB installed, plus a 148 MB model) | transcription is off; scans report that no recogniser is installed | | `pip install vosk` | a smaller, weaker recogniser (~10 MB plus a 40 MB model) | as above | +| `pip install openai-whisper` | the reference Whisper, slower and heavier than faster-whisper | as above | +| `pip install pocketsphinx` | a tiny recogniser, poor on radio audio | as above | +| whisper.cpp (`whisper-cli` on the PATH) | transcription with no Python dependencies at all | as above | | `pip install pyte` | the terminal-resize tests | those tests skip | +| `pip install pytest` | running the test suite | you cannot run the tests | + +`bandsaunter transcribe --list` says which recognisers it can actually see, +and which one it would use. + +**Nothing is downloaded behind your back.** The things that reach a network do +so only when you ask, and each is cached on disk afterwards: + +| What | Where from | When | Kept in | +|---|---|---|---| +| aircraft and route lookups | `api.adsbdb.com`, `hexdb.io` | while listening, unless `--no-lookup` | `~/.cache/bandsaunter/flights.json`, a month | +| amateur callsign lookups | `callook.info`, `api.hamdb.org` | when a scan hears a callsign, unless turned off | `~/.cache/bandsaunter/callsigns.json`, a month | +| map tiles | `tile.openstreetmap.org`, or `--tiles URL` | when a map is drawn, unless `--no-basemap` | `~/.cache/bandsaunter/tiles`, for ever | +| aerodrome positions | `overpass-api.de` | when a map is drawn, unless `--no-airports` | `~/.cache/bandsaunter/airports`, a month | +| flight schedules | the four paid services below | only if you have set a key | a month | + +Every request identifies itself as `bandsaunter` and carries nothing but the +question — a callsign, a 24-bit address, or a box of the world. No identity, +no position, no key. + +**The aircraft window needs Qt**, and any of four bindings will do — PyQt6, +PyQt5, PySide6 or PySide2 — because distributions disagree about which they +package. `python3-pyqt6` on Debian and Fedora, `python-pyqt6` on Arch, or +`pip install PyQt6` in the environment bandsaunter runs from. Without it the +window is the only thing missing, and the program says so rather than failing. **Aircraft and callsign lookups need no installation**, only a network. They ask public registers about a callsign or a 24-bit address and cache the answers for a month; `--no-lookup` turns them off, and what the address and the callsign say on their own is worked out offline either way. +### Optional: a paid schedule service + +Nothing here needs installing either — these are accounts, not packages, and +all four are optional. They answer the one question the free registers cannot: +an airline runs the same flight number over several legs in a day, and a free +register holds one route per number, so it will often name somebody else's +leg. A schedule service holds the day's actual movements. + +| Service | Sign up at | Set | +|---|---|---| +| FlightAware AeroAPI | | `BANDSAUNTER_AEROAPI_KEY` | +| Flightradar24 | | `BANDSAUNTER_FR24_TOKEN` | +| OAG Flight Info | | `BANDSAUNTER_OAG_KEY` | +| Cirium (FlightStats) | | `BANDSAUNTER_CIRIUM_APP_ID` and `BANDSAUNTER_CIRIUM_APP_KEY` | + +Put the ones you have in your shell profile: + +```sh +echo 'export BANDSAUNTER_AEROAPI_KEY=your-key-here' >> ~/.bashrc +. ~/.bashrc +``` + +**Keys are read from the environment and never written to the settings file**, +on purpose: a settings file gets copied between machines and pasted into +messages asking for help, and an API key should not travel that way. + +Any service whose key is set is asked; one whose key is not set is skipped +silently, and the free registers answer exactly as they did before. +`--schedules flightaware,oag` picks which to ask and in what order. Only the +callsign and the time are ever sent. + +These readers were written from each service's published response format and +tested against it, but none has been run against a live service, because each +one needs a paid account. Each is written to return nothing rather than guess, +so a service that has changed its format costs you a route, not a scan. + --- ## Speech transcription, step by step diff --git a/README.md b/README.md index 3b6c3e5..deb7624 100644 --- a/README.md +++ b/README.md @@ -111,9 +111,16 @@ be built. | **required** | Rich | `python3-rich` | `python3-rich` | `python-rich` | menus and the live display | | **required** | PyYAML | `python3-yaml` | `python3-pyyaml` | `python-yaml` | settings file and profiles | | *recommended* | eSpeak NG | `espeak-ng` | `espeak-ng` | `espeak-ng` | clearer spoken timestamps | +| *optional* | Qt | `python3-pyqt6` | `python3-pyqt6` | `python-pyqt6` | the realtime aircraft window | +| *optional* | ffmpeg | `ffmpeg` | `ffmpeg` | `ffmpeg` | writing `.mp4` instead of `.gif` | | *optional* | rtl-sdr tools | `rtl-sdr` | `rtl-sdr` | `rtl-sdr` | `rtl_test` and friends for diagnosis | | *optional* | a speech recogniser | **pip only** | **pip only** | AUR | transcribing speech to text | -| *optional* | Matplotlib | `python3-matplotlib` | `python3-matplotlib` | `python-matplotlib` | nothing yet; reserved for plots | + +**Only `librtlsdr` cannot come from pip.** It is a C library, so no virtual +environment brings it with it, and it is the one thing you have to install +from your distribution by hand. Everything that does not touch the hardware +works without it — reading logs, drawing maps, decoding recordings, the +simulated sky — and anything that does says so plainly rather than failing. Two notes on the optional ones: @@ -949,6 +956,229 @@ bandsaunter adsb --simulate # invent a sky, for a receiver with no aerial bandsaunter flights # read the log back: report, map, animation ``` +While it listens, the screen is a live board of what is overhead: + +``` +╭─────────────────────────────────────────────────────────────────────────────╮ +│ 1090 MHz 6 overhead 9 seen 1,284 frames 19/s 0:04:31 control-C │ +╰─────────────────────────────────────────────────────────────────────────────╯ +callsign ICAO aircraft altitude speed kt track position frames last +BAW49 4008F6 B744 G-VROS 33,025↑ 480 300° WNW 48.2775,-121.8050 1,204 0s +ASA412 A24C71 B738 N625AS 12,400↓ 310 155° SSE 47.3323,-122.7387 412 1s +N517HP A6F109 R44 N517HP 1,200 95 020° NNE 47.6485,-122.2647 88 2s +``` + +One line per aircraft, in the order they were first heard. **The counter +climbs as frames arrive**, altitude is coloured low-warm to high-cold with an +arrow for climb or descent, and the age of the last frame goes green → yellow +→ red. When nothing has been heard from an aircraft for `hold` seconds +(45 by default) its line is removed and everything below moves up — the board +is the sky now, not a list of everything ever heard. Nothing is lost by it: +the log has every frame and the report at the end lists every aircraft. + +The registers are asked *while* it listens, so the registration, type, +operator and route fill themselves in on the line as the answers arrive. A +narrow terminal drops the columns a website supplied and keeps the ones only +the aircraft can give. `--frames` prints the raw stream instead, and a pipe +or a log file gets a plain running count rather than a display that redraws +four times a second. + +**Speeds in whatever you read in.** `--speed-unit knots|mph|kph` (or the +option in the menu) changes the column heading on the live display, the speed +written beside every aircraft on the map, and the speeds in the report — and +it moves the distances with them, so a map labelled in mph has a scale bar in +statute miles and one in kph has kilometres, rather than two different miles +on one picture. **The log always keeps knots**, because that is what the +aircraft broadcast: the recording stays the thing that arrived, and the +conversion happens at the moment of showing it to somebody. + +### A window, while it happens + +```bash +bandsaunter adsb --window # or the menus: 5, then r +``` + +The terminal board says what is overhead; this says **where**. A real map, the +aircraft moving on it as the frames arrive, and beside each one a box with +everything known about the flight — type and registration, who operates it, +where it came from and where it is going — each end with its country's flag — +altitude with a climb or descent rate, speed and heading, how far away and on +what bearing, its position, how many frames it has sent and how long since the +last one. + +![the realtime window](docs/realtime.png) + +The boxes are placed so they cover neither each other nor another aircraft's +symbol: the eight spots beside the aircraft are tried first, then rings +outward, and a leader line runs to the near edge of the box rather than +through it. Altitude is the colour, low warm to high cold, the same ramp the +GIFs use. + +Long names are folded rather than allowed to stretch the box — a route +between two airports with their full names runs to sixty characters, which +would otherwise make one box wider than the map under it. A route breaks at +the arrow first, so the two ends of the flight stay whole and sit under one +another where they read as a pair. + +**A box that has to move swings there rather than jumping.** Two things make +one move: an aircraft flies into the space its neighbour's box was using, or +a new one arrives and claims it. Recomputing the layout every frame and +drawing the answer means boxes teleport, and the eye reads a thing that +teleports as a different thing — on a busy screen several do it at once. + +So a box keeps the spot it has for as long as that spot still works, and is +only moved when something genuinely takes it. On a real evening's log that is +about a third as many moves as placing each box afresh every frame. The moves +that are left are eased over about half a second, and the picture redraws at +thirty frames a second for as long as anything is actually moving, dropping +back to its usual five the moment everything has settled. + +The spot a box is *heading for* is what the next box is laid out against, not +the place it has got to so far — laying out against a box in mid-swing would +move its neighbours too, and move them back when it arrived, and the screen +would never settle. A box on the move is drawn last, over the ones standing +still, so it stays readable while it crosses them. The box is remembered as an +offset from its aircraft, so one crossing the window carries its box along +without that counting as a move at all. + +Thirty frames a second is affordable because **the ground is dimmed once and +kept**. Cutting the view out of the fetched map, dimming it and looking every +level up in the palette is about 70 ms over two megapixels, and none of it +changes between one frame and the next unless the view, the window, the +brightness or the map itself has. Doing it every frame put a ceiling of a +dozen frames a second on the window at 1920×1080 and spent a whole core +holding it there; keeping it takes the same window from 87 ms a frame to 14. + +**The line from a box to its aircraft is dashed, and is its own colour**, on +the animated pictures as well as here. It used to be drawn in the aircraft's +own colour, which made it the same colour as that aircraft's trail — and a straight solid line running out of an +aeroplane, in the colour of the path behind the aeroplane, reads as more path. +On a busy picture that is a heading nobody flew. Dashes and a neutral colour +say "this box belongs to that aeroplane" instead, which is all it was ever +meant to say. Qt measures a dash pattern in multiples of the pen's width, so +each pass of the glow divides the pattern by its own width; without that the +halo's dashes are three times the core's and the line comes out as beads. + +**The animation had no such line at all** until now — a label pushed out into +one of the outward rings by a crowd had nothing tying it to the aeroplane it +was about. It has one now, dashed and in the same colour, walked along the +line's own length rather than along whichever axis is longer so that a nearly +horizontal leader and a nearly vertical one get dashes of the same length +instead of one of them turning into a dotted line. + +**A red flag stands where the receiver is**, on the window and on the animated +pictures alike, from the coordinates in the settings (`--at`, or **Receiver +position** in the menu). The foot of the pole is the position and the pennant +flies up and to the right of it, so nothing the flag is made of covers the +place it points at. It is pure red in every theme — "you are here" is the one +mark whose meaning must not change with the colours, and pure red is both the +brightest red there is and the one furthest from every altitude colour in +every theme. A softer red sat close enough to a low aeroplane on the default +map, and to a mid-altitude one on the red theme, to be taken for one. + +**Nothing is ever drawn over the flag.** It goes down after everything else on +both pictures — after the aircraft, their trails and their boxes — because it +says where the receiver is standing, and that is the one mark that must not +end up behind an aeroplane that happened to fly over it. The halo of a vector +theme cannot cover it either: a halo only ever goes on the ground, the grid +and the empty background. + +The flag is drawn **only where the receiver was actually told where it is**. +Without a position the middle of the picture is worked out from whatever flew +past, which is not a place anybody is standing, and a flag on it would say +that somebody is. + +**`--box-opacity PERCENT`** (85 by default) is how solid the card behind each +information box is. The words beside an aircraft are readable over water and +not over a city, so a card goes behind them; at 0 they sit straight on the map +and at 100 the map does not show through at all. An indexed picture cannot +blend, so in the animation this darkens the ground under the box instead — +which leaves the coastline faintly visible through it, where a flat wash would +not. The animated pictures had **no card at all** before this, so `0` is what +they used to look like. + +**Range rings** put faint discs at a quarter, a half and three quarters of the +radius, concentric on the receiver and each labelled with its distance. They +are translucent and they stack, so the ground inside the innermost is lifted +three times, the next twice, the outer once. What that gives is a sense of how +far away something is without measuring anything: an aircraft two shades in is +about halfway to the edge of what this receiver hears. + +An indexed picture cannot blend, so "translucent" in the animation means +moving the ground under the disc a step or two up its own ramp of shades — +which keeps the coastline and the roads visible through it, where a flat wash +of one colour would not. The window has real alpha and simply paints one. + +They need a receiver position and a radius and are not drawn without both, and +each is a separate setting: `--rings` / `--no-rings` for the pictures, +`--window-rings` / `--no-window-rings` for the window, both also in the ADS-B +options menu. A picture is studied and a window is glanced at, and the rings +help one more than the other depending which you are doing. + +**The aerodromes are marked here too**, in the same colour and with the same +square as the animation draws them. They are asked for once per area, on the +thread that fetches the tiles but not behind them — they used to be fetched +only in the same pass as a piece of map, which meant they queued behind a +hundred and twenty tiles coming off a network, and once the map was in hand +there were no more passes and they were never fetched at all. + +`d` cycles the detail — full box, just height and speed, or symbols alone — +for when the sky is busy. `t` toggles trails, `g` the map underneath, `[`/`]` +its brightness, `+`/`-` the range, `q` closes it. + +Aircraft fade out here too, over `--fade` seconds, keeping their symbol and +losing their box as they go — a box at a tenth of its colour is something in +the way of the aircraft still flying. One that has gone quiet stops being +counted as overhead, since saying it is would be saying more than was heard. + +The map holds still. It is fetched an eighth larger than the window in every +direction, and the middle of the view settles once and then stays put rather +than being recomputed as aircraft come and go — otherwise the view shifts by a +fraction of a mile every few seconds, throws away the tiles fetched for the +old one, and the ground blinks out while new ones arrive. + +**Closing the window leaves exactly the files a passive capture does**: the +same log, the same report, the same KML and animation, because it is the same +code with a different thing watching it. The receiver runs on its own thread, +so a slow repaint cannot cost a frame and a slow tile fetch cannot stop the +picture moving. + +Qt is asked for and not required — PyQt6, PyQt5, PySide6 and PySide2 are all +tried, since distributions disagree about which to package. Without any of +them you lose this window and nothing else, and the program says how to get +one rather than failing. + +**Or from the menus: `bandsaunter` → 5, Aircraft (ADS-B).** `p` starts a +passive capture, `r` opens the realtime window and `m` draws a map from a log +— no flags to remember, and the options can be saved as the default. + +The options are in six groups rather than one list, because thirty-three of +them on one screen is a wall rather than a menu: + +| | | | +| --- | --- | --- | +| **receiver** | the dongle, and where it is standing | device, gain, sample rate, position, the simulated sky | +| **listening** | what one session does | how long, the log, KML, how long an aircraft stays up, draw when finished | +| **aircraft** | who they are | the registers, the schedule services, rechecking positions | +| **animation** | the moving picture | kind, length, speed, frame rate, width, trails, fading | +| **the map** | what is under and around them | tiles, theme, brightness, radius, aerodromes, range rings | +| **labels** | what is written beside them | the labels themselves, and the unit | + +A number opens a group; inside it, a number changes an option and `?N` +explains any of them at length. The numbers are the option's place in the +whole list, so the same number means the same option wherever it is typed. + +**Or type the name.** `map brightness` at the top of the menu goes straight to +that option, and part of a name lists everything it could mean. A name that +matches exactly wins outright, so `speed` reaches the setting called speed +rather than that one and every other whose description mentions the word. + +> **This is not a scan, and the band plan's `adsb` preset will not do it.** +> Sweeping 1090 MHz records the bursts as clicks in a WAV file and decodes +> nothing: the signalling is a megabit a second and the scan path is 12.5 kHz +> wide. Both the scanner and the menus now say so when a sweep is pointed at +> 1090 MHz or 978 MHz, rather than letting it run silently. + Every airliner overhead broadcasts its address, callsign, altitude, position and speed twice a second, unencrypted, to nobody in particular. @@ -1014,6 +1244,29 @@ and `A835AF` is American with no network at all), and the first three letters of an airline callsign are its ICAO designator, so `RYR1234` is Ryanair. `--no-lookup` stops at that. +**A callsign is a flight number, not a leg.** An airline runs the same number +over several legs in a day — Southwest especially — and a register holds one +route for it, so an aircraft crossing Arizona is quite often handed a +half-hour hop between two airports in Texas. The two databases routinely +disagree with each other about the same flight number, and both are snapshots +years old. + +Nothing on the air settles it: **ADS-B carries no origin or destination.** An +aircraft broadcasts who and where it is, not where it is going. So what can be +done is checked rather than trusted: + +- A route the aircraft **cannot** be flying is left off the map and out of the + window. The two ends are known, and an aircraft on a route is never much + further along it than the route is long. It is still written in the report + with a note saying so, because it is what the register holds for that flight + number and worth having — it is just not a statement about where this + aeroplane was going. +- Where a source lists a **whole day's stops** rather than a leg — hexdb + answers `KORD-KEWR-KORD` for some flight numbers — the aircraft's own + position picks the leg out. Reading the ends off that string instead gives + Chicago to Chicago, which is not a flight. +- Where no leg fits, none is claimed. + ``` 4008F6 BAW49 registration: G-VROS @@ -1029,6 +1282,46 @@ of an airline callsign are its ICAO designator, so `RYR1234` is Ryanair. speed: up to 480 kt ``` +### Knowing the leg for certain + +That needs live schedule data, which none of the free sources carry. Four +commercial services are wired up, and all four are **optional**: + +| service | keys | +| --- | --- | +| [FlightAware AeroAPI](https://www.flightaware.com/commercial/aeroapi/) | `BANDSAUNTER_AEROAPI_KEY` | +| [Flightradar24](https://fr24api.flightradar24.com/) | `BANDSAUNTER_FR24_TOKEN` | +| [OAG Flight Info](https://developer.oag.com/) | `BANDSAUNTER_OAG_KEY` | +| [Cirium (FlightStats)](https://developer.cirium.com/) | `BANDSAUNTER_CIRIUM_APP_ID` and `BANDSAUNTER_CIRIUM_APP_KEY` | + +Each holds the timetable and the day's movements, so each can answer the +question the registers cannot: which leg of that flight number was in the air +at the moment the aircraft was overhead. A schedule service is asked first and +the free databases pick the question back up where it does not answer, so a +program with no keys set behaves exactly as it did before. + +**Keys are read from the environment, never the settings file** — a settings +file is meant to be copied between machines and pasted into a message asking +for help, and an API key is not. There is a test that enforces it. + +```sh +export BANDSAUNTER_AEROAPI_KEY=... +bandsaunter flights # ask every service with a key +bandsaunter flights --schedules flightaware # ask only that one +bandsaunter adsb --schedules oag,cirium # ask those two, in that order +``` + +A service with no key is skipped rather than asked and refused, since a +request is only a slow way of finding out there is no key. The callsign and +the moment are all that is sent. The same setting lives in the ADS-B options +menu under **Schedule services**. + +One caveat, stated plainly: **each reader was written from its service's +published response shape and tested against that shape; none has been run +against a live service**, because each wants a paid account. So each is +written to find what it recognises and return nothing at all otherwise — a +service that has changed since costs a route, not a scan. + ### The moving map ```bash @@ -1047,17 +1340,397 @@ the twenty minutes the data says it took. Nothing moves at a constant speed for the look of the thing, and an aircraft not heard from for five minutes stops being drawn rather than being flown on by guesswork. +**An aircraft that goes quiet fades rather than vanishing.** Taking it off the +picture between one frame and the next says it stopped existing; fading it +says it stopped talking, which is what happened. It fades where it was last +actually seen and never along a reckoned track — the reason for giving up on +it in the first place is that where it would be by now is a guess. `--fade +SECONDS` (20 by default, and in the options menu) sets how long that takes; +zero takes it away at once, as before. + Time runs at `--speed` seconds of flying per second of animation, or give `--seconds` and let it work the speed out. Altitude is the colour, low warm to high cold, with the key along the bottom; the trail behind each aircraft is the path it actually flew, in the colours of the heights it flew them at. +Beside each aircraft goes what is known about it — height and speed, what sort +of aircraft it is, type and registration, and the two ends of the route, each +with **a small flag of the country the airport is in**: + +``` +PRIME04 + 36,000 ft 493MPH + [US] MILITARY HEAVY + C17 90-0534 +``` +``` +BAW49 + 33,000 ft 552MPH + [GB] HEAVY + B744 G-VROS + [GB] EGLL + [US] KSEA +``` + +The height is in feet with the unit on it. It used to be the flight level — +`330`, hundreds of feet, the way it is said on the radio — which is shorter +and is what an aviator reads, but three digits beside an aircraft are only a +height to somebody who already knows they are one. It is rounded to the +nearest 25 ft, the step Mode S reports altitude in: a real reading is a +multiple of that and comes through untouched, while a moment between two +reports — which is most of the moments in an animation, since they are +interpolated — stops claiming to know the aircraft's height to the foot. + +The flags are twelve pixels by eight, drawn from a table in `flags.py` rather +than fetched: at that size a flag is not a rendering of the real thing but the +arrangement that makes one recognisable — the bands and where they run, the +canton, the disc. A country not in the table is named by its two letters +instead, because a flag that is nearly another country's is worse than no flag +at all. + +Where a route arrives as nothing but a pair of airport codes, the country +comes from the code itself: the first letter or two of an ICAO code is a +region, so `EGLL` is British and `KSEA` American with nothing else to go on. +The aircraft's own **country of registration** gets a flag too, beside the +type and registration — from the register where one answered, and otherwise +from the 24-bit address, which says by treaty who issued it. + +### What sort of aircraft it is + +`MILITARY HEAVY` above is two facts, joined only because both are known. + +**What it is** comes off the air. Every identification message carries three +bits under its type code saying what sort of thing is transmitting — light, +small, large, high vortex, heavy, high performance, rotorcraft, and under a +different type code glider, airship, parachutist, ultralight, drone, +spacecraft, or a vehicle on the ground. It is the only word about what an +aircraft *is* that needs no register at all. Which list the three bits index +depends on the type code, so the same value means "heavy" in one message and +"parachutist" in another. Zero means the aircraft declined to say, which is +common and is answered with nothing rather than a guess. + +**Whether it is military** comes off no air at all: no aircraft broadcasts it, +and a tanker calls itself `heavy` exactly as an airliner does. It is read from +the 24-bit address instead — states set aside blocks of their national range +for their armed forces, and those blocks are published. Two honest limits: a +state can fly a military aircraft on a civil address whenever it likes, so +this finds nobody who does not want to be found; and the table here holds the +allocations a receiver in the ordinary world actually hears rather than every +one that exists. + +Worth seeing the two agree. On one evening here, address `AE07D3` was in the +United States military block and broadcast the category `heavy`; the register, +asked separately and knowing nothing of either, came back with a C-17A +Globemaster III, tail 90-0534, operated by the United States Air Force. + +**Every aerodrome under the picture is marked**, not only the ones being flown +between: a receiver hears aircraft over its own county, and the county's +airports are what say where on the map you are looking. They come from the +same OpenStreetMap data the tiles are drawn from, asked as a question rather +than a picture, once per area and kept for a month — a runway does not move. +`--no-airports` turns it off. + +They are drawn **magenta**, which nothing else on the picture is. The old +amber sat 16 units of CIELAB from the altitude ramp's 12,000-foot yellow, +which is to say it was the same colour: an aeroplane low over a field was +drawn in the field's own colour and neither could be picked out from the +other. The ramp already spends red, amber, green, cyan and violet on height; +magenta is what it leaves free, it is 56 units away, and it is what an +aeronautical chart marks an aerodrome in anyway. There is a test that states +this as the property — how far the airport colour is from the nearest +altitude colour — rather than as the colour, so that changing the ramp cannot +quietly walk an aircraft back into the airports. + +### Labels that stay where they are + +The same two rules as the window, so the two look like the same program. + +**A label keeps its place** for as long as that place still works, and is +moved only when something genuinely takes it. Deciding afresh every frame +means a label hops from one side of its aircraft to the other and back +because the aeroplane two along moved a pixel — on a real evening's log that +is about twice as many moves as keeping it. The place is remembered as an +offset from the aircraft, so one crossing the picture carries its label along +and only a real clash asks for a new spot. + +**The moves that are left are eased** over about half a second of playback, +not jumped. What the next label is laid out against is where a moving one is +going, not where it has got to, since laying out against a label in mid-swing +would move its neighbours too and move them back when it arrived. A still +picture has no frame before it, so it places its labels exactly as it always +did. + +**The whole box fades with the aircraft.** The name always did, because it is +drawn in the aircraft's own colour; the rows and the flag were fixed colours +and stayed at full brightness on a label that was on its way out, which left +the brightest thing on that part of the picture being the one aeroplane +nothing had been heard from. An indexed picture cannot blend, so the row grey +and all twelve flag colours have their own dimmed copies at each of the four +fade steps. + The GIF is written here from first principles — a palette, an LZW stream, frame differencing with a transparent index — in the same spirit as the PNGs elsewhere, so nothing but numpy is needed to draw one. Where ffmpeg happens to be installed, `--out something.mp4` is smaller and smoother; where it is not, nothing breaks and a GIF is written instead. +### How far the map reaches + +```bash +bandsaunter flights --radius 100 # the default: a hundred miles round +bandsaunter flights --radius 0 # fit whatever turned up, warts and all +bandsaunter flights --at 32.54,-111.17 # say where the receiver is +``` + +**The map is framed on the receiver, not on whatever was heard.** An aerial +reaches a hundred miles on a good day, and a position that decoded wrongly can +land anywhere on Earth — so a map drawn to fit everything is drawn to fit the +mistakes, and the aircraft come out a pixel wide in the middle of an empty +continent. One real night's recording spanned 240°N to 20°S before this. + +`--radius` is in the same unit as the speeds, so it is nautical miles with +knots and statute miles with mph. Left alone, the centre is the *median* of +everything heard — a receiver hears aircraft all round it, and a median cannot +be dragged anywhere by a handful of bad positions — or `--at LAT,LON` fixes it, +which is worth doing if you want the same frame every night. Fixes outside the +radius are dropped from the drawing, one fix at a time rather than one aircraft +at a time, so a single bad position in the middle of a real flight does not +take the whole flight off the map with it. Nothing is dropped from the log. + +### Positions that never happened + +```bash +bandsaunter flights --recheck # throw out the impossible ones +``` + +A position is sent as *half* a position — an even frame and an odd one — and +the pair only means anything while the aircraft has not moved between them. +Logs written before this version paired them however old they were, so an even +frame kept from ten minutes ago decoded against a fresh odd one to a place on +the wrong side of the world, written down as confidently as a real position. +On one night's recording that was **two aircraft in three**, with positions out +to 7,378 nautical miles and one latitude of 239°. + +`--recheck` reads a log back and keeps, for each aircraft, the longest run of +positions that could describe one aeroplane. It is deliberately not a forward +walk that drops whatever disagrees with the last position kept: one bad fix +then becomes the reference, and it is the truth that gets thrown away — on the +same recording that discarded a fifth of everything, most of it real. Nothing +is changed in the log; the frames stay exactly as they arrived. + +Two things it will not do, on purpose. An aircraft that goes quiet for five +minutes and is heard again a long way off is an aeroplane, not an error, and +nothing after that gap is second-guessed — the radius is what keeps those off +the picture. And where two positions contradict each other and nothing else +has an opinion, one of them is wrong and there is no saying which, so the +later one goes. + +**New logs need none of this**: the decoder now refuses a pair more than ten +seconds apart, refuses a position that is not on Earth, and refuses one the +aircraft could not have reached, as the frames arrive. + +### The ground under it + +**There is a real map under the aircraft.** A flight path over a black +rectangle says how the aircraft moved and nothing about where it was; over a +coastline it says which airport it left. + +Standard `{z}/{x}/{y}` raster tiles are fetched the first time an area is +drawn — OpenStreetMap by default — reprojected from Web Mercator onto the +picture pixel by pixel, inverted and dimmed so that the map is the ground and +the aircraft stay the brightest thing on it. The PNG tiles are decoded here, +by the same reasoning the PNGs are written here: zlib, numpy and the five +row filters from the specification, and no imaging library. + +Using somebody else's tile server carries three obligations, and all three +are met rather than assumed: + +- **tiles are cached** in `~/.cache/bandsaunter/tiles` and never fetched + twice, so redrawing an evening costs nothing and works with no network; +- **every request says who is asking**, in the User-Agent; +- **the attribution is drawn onto the picture**, because a GIF travels + without the readme that would otherwise carry it. + +`--no-basemap` draws the tracks on their own, `--tiles URL` points at another +server (your own, if you run one), and when there is no network and nothing +cached the picture falls back to the plain grid it drew before. + +**The zoom is chosen from how wide the picture is, not from the area alone.** +A map asked for at 1920 pixels fetches finer tiles than the same map asked for +at 960, and about 1.4× the width is fetched deliberately and then averaged +down — a downscaled tile is sharp and an upscaled one is not, so it is better +to fetch too much and shrink it than to fetch too little and stretch it. + +The window fetches a little more world than it shows, so that panning does not +leave the ground blank, and it fetches that bigger piece **at the bigger +piece's own size**: what the window then shows comes out pixel for pixel with +the screen. Rendering the wider piece into the window's own pixels and +stretching it back is an upscale of a fifth applied to the whole map, which is +what a sharp map looks like when it looks blurred. + +Measured, at a 100-mile radius: + +``` +animation at 960 px zoom 9 28 tiles 1792 px 1.9x oversampled +animation at 1920 px zoom 10 91 tiles 3328 px 1.7x oversampled +window at 1920x1080 zoom 10 135 tiles 3840 px 1.6x oversampled +window at 3840x2160 zoom 10 135 tiles 3840 px 0.8x — upscaled +``` + +A drawing is still capped, at a couple of hundred tiles, and the last line is +what that cap looks like: at 4K the zoom has already stopped climbing and the +map is enlarged after all. Somebody else's tile server is not a thing to fetch +a thousand tiles from for one picture. A smaller `--radius` buys the detail +back, since the same budget then covers less ground. + +`--map-brightness PERCENT` (70 by default) is how far up its range the map is +drawn, and **the vector themes bend the middle of that range down hard** — +because a tinted photograph of a county behind the vectors is the one thing +that stops a vector display looking like one. It is a curve, not a ceiling: +the top of the setting is a full-brightness map on every theme. It used to be +a plain multiplier, which meant that on a phosphor theme the setting could not +reach a visible map at all — turned the whole way up it still came out at a +tenth of what the default theme gives, which is to say invisible. So if the +ground is too faint to make out, this is the setting that fixes it, and on a +vector theme it takes rather more turning up than on the default one. The ground has to stay dark enough that the aircraft are the brightest +thing on the picture and light enough that a coastline can be made out at all, +and which way to err depends on the screen you are looking at. In the window, +`[` and `]` change it while it runs. + +## Themes + +`--theme NAME`, on `bandsaunter adsb` and `bandsaunter flights` alike, and in +the ADS-B options menu. It changes the window and the animated pictures +together, because both read their colours out of the same palette. + +| theme | | +| --- | --- | +| `night` | the default: a night-blue ground, height as colour | +| `digital` | blue phosphor, cyan vectors on black, amber aerodromes | +| `phosphor` | green P1 phosphor, height as brightness | +| `amber` | amber phosphor, warm vectors on black | +| `red` | red phosphor, for a room that wants its night vision | + +The four vector themes are the screens the phrase "air defence display" +actually calls to mind: a black tube, one phosphor, and thin bright lines +with a halo around them. Three things follow from that, and they are +constraints rather than decoration. + +**Height becomes brightness.** The default map spends the whole spectrum on +altitude — low warm, high cold — which is why nothing else on it can be amber +or green. A phosphor screen has one colour, so on those themes low is dim and +high burns. That is the same trade the real displays made. + +**The ground goes well back.** A tinted photograph of a county behind the +vectors is the one thing that stops a vector display looking like one, so the +map underneath is drawn at about two-fifths of the brightness asked for and +the lines carry the picture. `--map-brightness` still moves it. + +**Countries are named, not flown.** A flag is half a dozen colours and a +phosphor has one, so those themes write the two letters instead — which is +what a display of the period would have done anyway. + +### The glow + +A vector display draws by holding a beam on the phosphor, and the phosphor +spreads the light a little and keeps glowing after the beam has gone. So a +line on one of those screens is not one pixel wide with a hard edge; it is a +bright core inside a halo. Both drawings do that, by different means, because +they are different kinds of picture: + +- **the window** lays the same line down two or three times, wider and fainter + each pass, and then the core on top — trails, aircraft, leader lines, box + borders and the aerodrome squares; +- **the animation** cannot blend at all, because a GIF is indexed colour. So + it dilates what it has drawn and fills the halo with the *dimmed copy* of + the colour underneath it. The aircraft colours already have dimmed copies — + those are the trail shades — so an aeroplane glows into the colour its own + trail is drawn in, which is the colour a phosphor would have spread into. + The fixed colours have two rings each added to the palette for the purpose. + +The halo goes over the map, the grid and the empty background and nothing +else: a halo is what light does to the dark around a line, and painting it +over another line would be light doing something light does not do. Where two +rings meet the nearer wins, which is what happens on the tube as well. It +costs about 55 ms a frame at 1400×1258, and the default theme skips the pass +entirely. + +## Every ADS-B option + +Thirty-four of them, in the six groups the menu shows. Each is a flag on the +command line and a line in `bandsaunter` → 5, and both are generated from one +table in the source, so they cannot disagree. `man bandsaunter` has the long +form of every one. + +**receiver** + +| option | flag | default | what it does | +| --- | --- | --- | --- | +| Receiver | `--device` | `0` | which receiver to use, when more than one is plugged in | +| Gain | `--gain` | `auto` | tuner gain in dB, or automatic | +| Sample rate | `--rate` | `2 MHz` | how fast to sample; two megasamples a second is the minimum | +| Receiver at | `--at` | — | where the receiver is, as latitude,longitude (blank = work it out) | +| Invent a sky | `--simulate` | `no` | fly imaginary aircraft past an imaginary receiver | +| Imaginary sky near | `--near` | `47.55,-122.30` | where the simulated aircraft are flying | + +**listening** + +| option | flag | default | what it does | +| --- | --- | --- | --- | +| Listen for | `--seconds` | `until stopped` | how long to listen before stopping (0 = until interrupted) | +| Show every frame | `--frames` | `no` | print each frame as it arrives, rather than a running count | +| Write the log | `--log` `--no-log` | `yes` | write every frame to a file as it arrives | +| Also write a KML | `--kml` | `no` | write the flight paths for Google Earth as well | +| Keep on screen for | `--hold` | `45 s` | how long an aircraft stays on the display after its last frame | +| Draw when finished | `--map` | `no` | draw the map as soon as the listening stops | + +**aircraft** + +| option | flag | default | what it does | +| --- | --- | --- | --- | +| Look the aircraft up | `--lookup` `--no-lookup` | `yes` | ask the public registers who each aircraft is | +| Schedule services | `--schedules` | — | which paid schedule services to ask, in order (blank = all with keys) | +| Check the positions | `--recheck` | `no` | throw out positions the aircraft could not have been in | + +**animation** + +| option | flag | default | what it does | +| --- | --- | --- | --- | +| Picture | `--out` | `gif` | what kind of picture to draw | +| Animation length | `--seconds` | `30 s` | how long the animation should run for | +| Speed | `--speed` | `fit to the length` | seconds of flying per second of animation (0 = fit to the length) | +| Frames a second | `--fps` | `12` | how many frames of animation each second holds | +| Picture width | `--width` | `960 px` | how many pixels across the picture is | +| Trail | `--trail` | `the whole path` | how much of the path to leave behind each aircraft (0 = all of it) | +| Fade out over | `--fade` | `20 s` | how long an aircraft takes to fade away once it has gone quiet | +| Forget after | `--stale` | `300 s` | stop drawing an aircraft this long after its last report | + +**the map** + +| option | flag | default | what it does | +| --- | --- | --- | --- | +| Map underneath | `--basemap` `--no-basemap` | `yes` | draw a real map under the flight paths | +| Tile server | `--tiles` | — | where the map tiles come from | +| Colour theme | `--theme` | `night` | how the map looks: the colours, and whether the lines glow | +| Map brightness | `--map-brightness` | `70 %` | how bright the map under the aircraft is drawn, as a percentage | +| Map radius | `--radius` | `100` | how far around the receiver the map reaches (0 = fit whatever was heard) | +| Mark the airports | `--airports` `--no-airports` | `yes` | mark every aerodrome on the map, not only the ones flown between | +| Range rings on the pictures | `--rings` `--no-rings` | `yes` | faint discs at a quarter, a half and three quarters of the radius | +| Range rings in the window | `--window-rings` `--no-window-rings` | `yes` | the same discs on the realtime display | + +**labels** + +| option | flag | default | what it does | +| --- | --- | --- | --- | +| Box translucency | `--box-opacity` | `85 %` | how solid the card behind each information box is, as a percentage | +| Label the aircraft | `--labels` `--no-labels` | `yes` | write the callsign, height and speed beside each aircraft | +| Speed in | `--speed-unit` | `knots` | what to show speeds and distances in | + +**Where they live.** Saved with `s` in the menu to +`~/.config/bandsaunter/aircraft.yaml`, which is plain YAML you can edit or +copy between machines. API keys are deliberately **not** kept there — see +[Knowing the leg for certain](#knowing-the-leg-for-certain). + ## Meters and weather sensors Two things on the ISM bands are worth naming rather than reporting as hex. @@ -1739,6 +2412,12 @@ deleted. ## Built-in help +Menu **5, Aircraft (ADS-B)**, is the whole of the aircraft mode without a +command line: nineteen options on one screen, each with a line saying what it +does, `?N` for the long version with the flag it corresponds to, `l` to listen +and `m` to draw a map from any log in the recordings directory. `s` saves the +options to `~/.config/bandsaunter/aircraft.yaml`. + Press `h` in the menus for topics covering setup, how the sweep works, why nothing (or too much) is being recorded, capturing conversations, where files go, trunked systems, HF reception and the keys available during a scan. Typing a setting name diff --git a/bandsaunter/__init__.py b/bandsaunter/__init__.py index 81a1c99..2f8d679 100755 --- a/bandsaunter/__init__.py +++ b/bandsaunter/__init__.py @@ -8,8 +8,8 @@ and transcribing speech. # Versions are the release date and a revision within that day, so # 2026-08-21_02 is the second build made on the 21st. The revision is padded # to two digits so versions sort as text. -VERSION_DATE = "2026-09-03" -VERSION_REVISION = 5 +VERSION_DATE = "2026-09-06" +VERSION_REVISION = 3 __version__ = f"{VERSION_DATE}_{VERSION_REVISION:02d}" diff --git a/bandsaunter/adsb.py b/bandsaunter/adsb.py index e368035..e3bca5c 100644 --- a/bandsaunter/adsb.py +++ b/bandsaunter/adsb.py @@ -30,7 +30,8 @@ import numpy as np __all__ = ["decode_adsb", "Frame", "Aircraft", "AircraftRegistry", "crc24", "ADSB_HZ", "SAMPLE_RATE", "PREAMBLE_US", "encode_identification", "encode_position", "encode_velocity", "modulate", "SimulatedSky", - "VirtualAircraft", "default_sky"] + "VirtualAircraft", "default_sky", "category_name", + "EMITTER_CATEGORIES"] ADSB_HZ = 1_090_000_000.0 @@ -38,6 +39,19 @@ ADSB_HZ = 1_090_000_000.0 SAMPLE_RATE = 2_000_000 PREAMBLE_US = (0.0, 1.0, 3.5, 4.5) + +# A position takes an even frame and an odd one, and the pair is only good +# for as long as the aircraft has not meaningfully moved between them. Ten +# seconds is what the standard allows; past that the two halves describe +# different places and the answer is not a position at all. +CPR_PAIR_SECONDS = 10.0 + +# How long a previous position stays worth checking a new one against. +CPR_TRUST_SECONDS = 300.0 + +# Faster than anything with a transponder on it, so that a real aircraft is +# never called an error -- Concorde cruised at 1150 kt. +MAX_GROUND_SPEED_KT = 2000.0 SHORT_BITS = 56 LONG_BITS = 112 @@ -55,6 +69,35 @@ TYPE_NAMES = { } +# The three bits under an identification message's type code say what sort +# of thing is transmitting. Which list they index depends on the type code +# -- the same three bits mean "heavy" under type 4 and "glider" under type 3 +# -- which is why this is a table of tables and not a table. +# +# This is the only word about what an aircraft *is* that comes off the air. +# Everything else -- the model, the operator, the registration -- is a +# lookup in somebody's database against the address. There is no category +# for "military": that has to be read off the address block instead. +EMITTER_CATEGORIES = { + 4: {1: "light", 2: "small", 3: "large", 4: "high vortex", + 5: "heavy", 6: "high performance", 7: "rotorcraft"}, + 3: {1: "glider", 2: "airship", 3: "parachutist", 4: "ultralight", + 6: "drone", 7: "spacecraft"}, + 2: {1: "emergency vehicle", 2: "service vehicle", 3: "obstacle", + 4: "obstacle", 5: "obstacle"}, + 1: {}, # reserved, and nothing transmits it +} + + +def category_name(type_code: int, category: int) -> str: + """What an identification message says the transmitter is, or "". + + Zero means the aircraft declined to say, which is common and is not an + error: it is answered with nothing rather than with a guess. + """ + return EMITTER_CATEGORIES.get(type_code, {}).get(category, "") + + def crc24(data: bytes) -> int: """The Mode S parity, polynomial 0xFFF409. @@ -86,6 +129,7 @@ class Frame: df: int = 0 # downlink format icao: str = "" # the aircraft's permanent 24-bit address type_code: int = 0 + category: int = 0 # the emitter category, with the type code at_sample: int = 0 callsign: str = "" altitude_ft: int = 0 @@ -264,6 +308,10 @@ def _read(bits: str, data: bytes) -> Frame: frame.type_code = int(me[:5], 2) if 1 <= frame.type_code <= 4: frame.callsign = _callsign(me) + # The three bits under the type code say what sort of thing this is + # -- heavy, rotorcraft, glider, ground vehicle. It comes off the + # air with the callsign and costs nothing to keep. + frame.category = int(me[5:8], 2) elif 9 <= frame.type_code <= 18 or 20 <= frame.type_code <= 22: frame.altitude_ft = _altitude(me) frame.cpr_odd = me[21] == "1" @@ -379,10 +427,12 @@ class Aircraft: track_deg: float = 0.0 vertical_rate_fpm: int = 0 messages: int = 0 + category: str = "" # what it said it was: heavy, rotorcraft... first_seen: float = 0.0 last_seen: float = 0.0 _even: Frame | None = field(default=None, repr=False) _odd: Frame | None = field(default=None, repr=False) + _placed_at: float = field(default=0.0, repr=False) @property def located(self) -> bool: @@ -426,6 +476,8 @@ class AircraftRegistry: frame.received_at = when if frame.callsign: seen.callsign = frame.callsign + if frame.category and not seen.category: + seen.category = category_name(frame.type_code, frame.category) if frame.altitude_ft: seen.altitude_ft = frame.altitude_ft if frame.ground_speed_kt: @@ -438,19 +490,73 @@ class AircraftRegistry: seen._odd = frame else: seen._even = frame - if seen._even is not None and seen._odd is not None: - # Whichever of the pair arrived later is the one the position - # is reported at. Compared by arrival rather than by sample - # offset: the offset restarts at zero every block, so a pair - # that straddles two blocks would otherwise be read backwards - # and put the aircraft in the wrong zone. - even_first = (seen._even.received_at, seen._even.at_sample) > \ - (seen._odd.received_at, seen._odd.at_sample) - found = global_position(seen._even, seen._odd, even_first) - if found is not None: - seen.latitude, seen.longitude = found + self._place(seen) return seen + def _place(self, seen: Aircraft) -> None: + """Work out where an aircraft is, from the last even and odd frames. + + The pair has to be recent, and the answer has to be reachable. Both + checks are the difference between a map and a scatter of nonsense: + an unpaired frame kept from ten minutes ago decodes against a fresh + one to a position on the wrong side of the world, because compact + position reporting sends a fraction of a zone and the two fractions + are then read as though the aircraft had not moved between them. + Measured against one night's recording, that produced positions up + to seven thousand miles out, on two aircraft in three. + """ + even, odd = seen._even, seen._odd + if even is None or odd is None: + return + if abs(even.received_at - odd.received_at) > CPR_PAIR_SECONDS: + return + # Whichever of the pair arrived later is the one the position is + # reported at. Compared by arrival rather than by sample offset: the + # offset restarts at zero every block, so a pair that straddles two + # blocks would otherwise be read backwards and put the aircraft in + # the wrong zone. + even_first = (even.received_at, even.at_sample) > \ + (odd.received_at, odd.at_sample) + found = global_position(even, odd, even_first) + if found is None or not _on_earth(*found): + return + when = max(even.received_at, odd.received_at) + if not seen.located or self._reachable(seen, found, when): + seen.latitude, seen.longitude = found + seen._placed_at = when + + @staticmethod + def _reachable(seen: Aircraft, found: tuple[float, float], + when: float) -> bool: + """Whether an aircraft could have got there from where it was. + + A position that would need eight hundred knots in the second since + the last one is not a position, whatever the checksum said about the + frames it came from. The bar is set well above anything that flies + so that a genuinely fast aircraft, or a gap in reception, is never + mistaken for an error. + """ + gap = when - seen._placed_at + if gap <= 0 or gap > CPR_TRUST_SECONDS: + return True # too long ago to argue with + miles = _distance_nm(seen.latitude, seen.longitude, *found) + return miles <= MAX_GROUND_SPEED_KT * gap / 3600.0 + + +def _on_earth(lat: float, lon: float) -> bool: + """Whether a decoded position is a place at all.""" + return -90.0 <= lat <= 90.0 and -180.0 <= lon <= 180.0 + + +def _distance_nm(lat1: float, lon1: float, lat2: float, lon2: float) -> float: + """Great-circle distance, in nautical miles.""" + p1, p2 = math.radians(lat1), math.radians(lat2) + dp = p2 - p1 + dl = math.radians(lon2 - lon1) + a = math.sin(dp / 2) ** 2 + \ + math.cos(p1) * math.cos(p2) * math.sin(dl / 2) ** 2 + return 2 * 3440.065 * math.asin(min(1.0, math.sqrt(a))) + def described(self) -> list[str]: return [craft.describe() for craft in sorted(self.aircraft.values(), key=lambda a: a.icao)] diff --git a/bandsaunter/aircraft.py b/bandsaunter/aircraft.py new file mode 100644 index 0000000..e233f83 --- /dev/null +++ b/bandsaunter/aircraft.py @@ -0,0 +1,1099 @@ +"""Listening to aircraft, and drawing where they went, from either front end. + +The command line and the menus do the same two things here -- park on 1090 MHz +and write down what arrives, then turn a log of that into a map -- so the doing +of it lives here and both front ends call it. Neither imports the other, and +the options are described once, in the same shape as every other setting in the +program, so the menu can print help for each one without knowing what any of +them mean. + +ADS-B does not go through the scanner and cannot be made to: it is a megabit a +second, which needs at least two megasamples a second of raw receiver output, +and the scan path decimates everything to a channel twelve and a half kilohertz +wide long before a decoder sees it. A scan of 1090 MHz records the envelope of +the bursts as clicks in a WAV file and decodes nothing, which is the mistake +:func:`scanning_aircraft_band` exists to catch. +""" + +from __future__ import annotations + +import time +from dataclasses import asdict, dataclass, field +from datetime import datetime +from pathlib import Path + +import yaml + +from .adsb import ADSB_HZ, SAMPLE_RATE +from .flightlog import read_position +from .settings import Setting, format_value + +__all__ = ["AircraftOptions", "OPTIONS", "listen", "watch", "draw", + "logs_in", "open_device", "open_log", "pump", "finish", + "windowed", + "format_option", + "load_options", "save_options", "options_path", + "scanning_aircraft_band", "AIRCRAFT_BANDS", "SCAN_WARNING"] + + +# --------------------------------------------------------------------------- +# The bands that look like a scan and are not one +# --------------------------------------------------------------------------- + +# (name, low, high, what to do instead) +AIRCRAFT_BANDS = ( + ("ADS-B", 1_089_000_000.0, 1_091_000_000.0, "bandsaunter adsb"), + ("UAT / ADS-B 978", 977_000_000.0, 979_000_000.0, None), +) + +SCAN_WARNING = ( + "These ranges cover a band the scanner cannot decode: {bands}. " + "The signalling is a megabit a second and the scan path is 12.5 kHz " + "wide, so a scan of it records clicks and finds no aircraft." +) + + +def scanning_aircraft_band(ranges) -> str: + """Warn when a sweep covers a band that needs the ADS-B mode instead. + + Returns the warning to print, or an empty string. The band plan lists + 1090 MHz because that is where ADS-B is, and choosing it from the band + plan is the obvious thing to do and the wrong one; saying so before the + sweep starts costs a line and saves an evening. + """ + hit = [] + for name, low, high, _ in AIRCRAFT_BANDS: + for r in ranges or (): + start = float(getattr(r, "start", 0.0) or 0.0) + stop = float(getattr(r, "stop", start) or start) + if start <= high and stop >= low: + hit.append(name) + break + if not hit: + return "" + return SCAN_WARNING.format(bands=", ".join(hit)) + + +# --------------------------------------------------------------------------- +# The options, described the way every other setting is +# --------------------------------------------------------------------------- + +@dataclass +class AircraftOptions: + """Everything the aircraft mode can be told, in one place. + + Kept apart from :class:`~bandsaunter.config.ScanConfig` because none of it + controls a scan: a sweep has no sample rate this high, no animation and no + aircraft. Saved in its own small file for the same reason. + """ + + # -- listening ------------------------------------------------------ + seconds: float = 0.0 + rate: float = float(SAMPLE_RATE) + gain: str = "auto" + device: int = 0 + frames: bool = False + log: bool = True + lookup: bool = True + schedules: str = "" + kml: bool = False + hold: float = 45.0 + speed_unit: str = "knots" + draw_after: bool = False + simulate: bool = False + near: str = "47.55,-122.30" + + # -- drawing -------------------------------------------------------- + picture: str = "gif" + length: float = 30.0 + speed: float = 0.0 + fps: float = 12.0 + width: int = 960 + trail: float = 0.0 + stale: float = 300.0 + fade: float = 20.0 + labels: bool = True + radius: float = 100.0 + location: str = "" + recheck: bool = False + basemap: bool = True + airports: bool = True + rings: bool = True + window_rings: bool = True + theme: str = "night" + box_opacity: int = 85 + map_brightness: int = 70 + tile_url: str = "" + + def validate(self) -> list[str]: + out = [] + if self.rate < SAMPLE_RATE: + out.append(f"sample rate must be at least {SAMPLE_RATE/1e6:g} MS/s " + "or a bit is too narrow to see") + if self.fps <= 0: + out.append("frames a second must be more than zero") + if self.width < 160: + out.append("the picture must be at least 160 pixels across") + if self.seconds < 0 or self.length <= 0: + out.append("times cannot be negative") + return out + + def to_dict(self) -> dict: + return asdict(self) + + +O = Setting + +OPTIONS: tuple[Setting, ...] = ( O("device", "Receiver", "Receiver", "int", + "which receiver to use, when more than one is plugged in", + "The index shown by `bandsaunter devices`. Zero unless you have " + "several dongles.", + minimum=0, flags=("--device",), example="0"), + O("gain", "Gain", "Receiver", "gain", + "tuner gain in dB, or automatic", + "ADS-B is a weak burst from a long way off, and the automatic gain " + "control usually does well enough. A fixed high gain can hear more " + "where there is nothing strong nearby to overload the front end.", + flags=("--gain",), example="auto"), + O("rate", "Sample rate", "Receiver", "float", + "how fast to sample; two megasamples a second is the minimum", + "One microsecond per bit means two samples per bit at 2 MS/s, which is " + "the least that can read one. Higher rates decode a little more of the " + "weak traffic and cost proportionally more processing.", + unit="Hz", minimum=float(SAMPLE_RATE), flags=("--rate",), + example="2000000"), + O("location", "Receiver at", "Receiver", "text", + "where the receiver is, as latitude,longitude (blank = work it out)", + "The centre of the map. Left blank it is taken from the middle of " + "everything heard, which is very close to right: a receiver hears " + "aircraft all around it, and the middle is a median rather than an " + "average, so a handful of wrong positions cannot drag it anywhere. " + "Setting it explicitly is worth doing if you want the same frame every " + "night regardless of which way the traffic went.", + flags=("--at",), example="32.54,-111.17", metavar="LAT,LON"), + O("simulate", "Invent a sky", "Receiver", "bool", + "fly imaginary aircraft past an imaginary receiver", + "Six aircraft that are not there, broadcasting real frames with real " + "checksums through the real decoder. Nothing touches the receiver, so " + "the log, the lookups, the report and the map can all be tried before " + "an aerial exists.", + flags=("--simulate",), + guidance="Turn this on to see what the whole thing does without " + "hardware. Turn it off to hear real aircraft."), + O("near", "Imaginary sky near", "Receiver", "text", + "where the simulated aircraft are flying", + "Latitude and longitude, as two numbers. Only used when the sky is " + "invented; it decides where the map ends up centred.", + flags=("--near",), example="47.55,-122.30", metavar="LAT,LON"), + # -- listening ------------------------------------------------------ + O("seconds", "Listen for", "Listening", "float", + "how long to listen before stopping (0 = until interrupted)", + "A whole evening is 0: listening runs until control-C, and everything " + "heard is on the disk as it arrives, so stopping never loses anything. " + "A number is useful for a quick look at whether the aerial hears " + "anything at all.", + unit="s", minimum=0.0, flags=("--seconds",), example="600", + guidance="Sixty seconds is enough to know whether aircraft are being " + "heard. An evening of traffic wants no limit."), + O("frames", "Show every frame", "Listening", "bool", + "print each frame as it arrives, rather than a running count", + "Every frame, with what was read out of it. Useful once, to see that " + "it is working; unreadable for an evening.", + flags=("--frames",)), + O("log", "Write the log", "Listening", "bool", + "write every frame to a file as it arrives", + "The JSON Lines log is what the report and the map are made from, and " + "it holds the raw hexadecimal of every frame beside what was decoded " + "from it. Turning this off leaves nothing behind but the screen.", + flags=("--log",), off_flags=("--no-log",)), + O("kml", "Also write a KML", "Listening", "bool", + "write the flight paths for Google Earth as well", + "One line per aircraft on the globe, with a pin where it was last " + "heard, in a .kml beside the log.", + flags=("--kml",)), + O("hold", "Keep on screen for", "Listening", "float", + "how long an aircraft stays on the display after its last frame", + "An aircraft that has gone out of range stops sending, and its line " + "would otherwise sit there for the rest of the evening saying the same " + "thing. When nothing has been heard from it for this long the line is " + "removed and everything below it moves up. Nothing is lost by it: the " + "log holds every frame, and the report at the end lists every aircraft " + "heard.", + unit="s", minimum=1.0, example="45", + guidance="Long enough that a gap in reception does not make rows jump " + "about; short enough that the screen is the sky now.", + flags=("--hold",), metavar="SECONDS"), + O("draw_after", "Draw when finished", "Listening", "bool", + "draw the map as soon as the listening stops", + "Saves running the map separately. It uses the drawing options below.", + flags=("--map",)), + O("lookup", "Look the aircraft up", "Aircraft", "bool", + "ask the public registers who each aircraft is", + "Two registers are asked -- adsbdb, then hexdb -- for the " + "registration, the type, the operator and the route, and the answers " + "are cached for a month. Only the address and callsign heard on the " + "air are ever sent. What the address block and the callsign say on " + "their own is worked out offline either way.", + flags=("--lookup",), off_flags=("--no-lookup",)), + O("schedules", "Schedule services", "Aircraft", "text", + "which paid schedule services to ask, in order (blank = all with keys)", + "A callsign is a flight number and an airline runs the same number over " + "several legs in a day, so the free route databases -- which hold one " + "route per number -- often name somebody else's leg. A commercial " + "schedule service holds the timetable and the day's movements and can " + "say which leg is in the air now. Four are wired up: flightaware, " + "flightradar24, oag and cirium. Each wants a key, none is required, " + "and a service with no key is skipped in silence. Keys are read from " + "the environment rather than kept here, because a settings file gets " + "copied between machines and pasted into messages asking for help: " + "BANDSAUNTER_AEROAPI_KEY, BANDSAUNTER_FR24_TOKEN, BANDSAUNTER_OAG_KEY, " + "and BANDSAUNTER_CIRIUM_APP_ID with BANDSAUNTER_CIRIUM_APP_KEY.", + flags=("--schedules",), metavar="NAMES", + example="flightaware,cirium", + guidance="Leave it blank unless you want one service tried before " + "another. With no keys set, nothing changes."), + O("recheck", "Check the positions", "Aircraft", "bool", + "throw out positions the aircraft could not have been in", + "For logs recorded before the decoder checked how old the two halves " + "of a position were. A compact-position report is half a position: an " + "even frame and an odd one, and the pair only means anything while the " + "aircraft has not moved between them. An even frame kept from ten " + "minutes ago decodes against a fresh odd one to a place on the wrong " + "side of the world, and that gets written down as confidently as a " + "real position. This reads the log back and keeps, for each aircraft, " + "the longest run of positions that could describe one aeroplane. " + "Nothing is changed in the log itself.", + flags=("--recheck",), + guidance="Worth turning on for anything recorded before this version. " + "Newer logs have the check applied as they are written, so it " + "finds almost nothing."), + # -- drawing -------------------------------------------------------- + O("picture", "Picture", "Animation", "choice", + "what kind of picture to draw", + "gif is an animation that plays anywhere and needs nothing installed. " + "mp4 is smaller and smoother but needs ffmpeg. png is one still " + "picture of the whole session, every path drawn at once.", + choices=("gif", "mp4", "png"), flags=("--out",), + guidance="Start with gif. Use png when you want one picture to look at " + "or send."), + O("length", "Animation length", "Animation", "float", + "how long the animation should run for", + "The whole session is fitted into this many seconds, so an evening of " + "flying plays in half a minute. Ignored when a speed is given.", + unit="s", minimum=1.0, flags=("--seconds",), example="30"), + O("speed", "Speed", "Animation", "float", + "seconds of flying per second of animation (0 = fit to the length)", + "60 means a minute of real flying every second. Setting this overrides " + "the length above: a long session simply makes a longer animation.", + unit="x", minimum=0.0, flags=("--speed",), example="60"), + O("fps", "Frames a second", "Animation", "float", + "how many frames of animation each second holds", + "Twelve is smooth enough for aircraft, which do not move quickly on a " + "map. A GIF can only hold whole hundredths of a second per frame, so " + "the real rate is rounded to the nearest one it can express.", + minimum=1.0, flags=("--fps",), example="12"), + O("width", "Picture width", "Animation", "int", + "how many pixels across the picture is", + "The height follows from the shape of the area the aircraft covered, " + "so that a mile across looks like a mile up the picture.", + unit="px", minimum=160, flags=("--width",), example="960"), + O("trail", "Trail", "Animation", "float", + "how much of the path to leave behind each aircraft (0 = all of it)", + "The whole flight is drawn by default, which is what makes the picture " + "a map of the evening rather than a snapshot. A number of seconds " + "leaves a comet tail instead, which is easier to follow when many " + "aircraft cross the same piece of sky.", + unit="s", minimum=0.0, flags=("--trail",), example="0"), + O("fade", "Fade out over", "Animation", "float", + "how long an aircraft takes to fade away once it has gone quiet", + "An aircraft that stops transmitting has not stopped existing, and " + "taking it off the picture between one frame and the next says that it " + "did. Instead it is left where it was last actually seen and fades from " + "there, which reads as an aircraft going quiet rather than as a blink. " + "Nothing is invented by it: the fading happens at the last known " + "position, never at a reckoned one, because the reason for giving up on " + "an aircraft in the first place is that where it would be by now is a " + "guess. Zero takes it away the moment it is given up on.", + unit="s", minimum=0.0, flags=("--fade",), example="20", + guidance="Long enough to notice, short enough that a busy sky is not " + "half ghosts."), + O("stale", "Forget after", "Animation", "float", + "stop drawing an aircraft this long after its last report", + "Between reports an aircraft is dead-reckoned from the speed and " + "heading it last gave. After a few minutes of that it has flown fifty " + "miles on a guess, so it is dropped instead of invented.", + unit="s", minimum=1.0, flags=("--stale",), example="300"), + O("basemap", "Map underneath", "The map", "bool", + "draw a real map under the flight paths", + "A flight path over a black rectangle says how the aircraft moved and " + "nothing about where it was; over a coastline it says which airport it " + "left. The map is fetched from a standard tile server the first time an " + "area is drawn and kept on the disk afterwards, so drawing the same " + "evening again costs nothing and needs no network. A few dozen tiles " + "at most, dimmed so the aircraft stay the brightest thing on the " + "picture, and the credit the tiles require is written on it.", + flags=("--basemap",), off_flags=("--no-basemap",), + guidance="Turn it off for a picture with nothing but the tracks on it, " + "or where there is no network and no cached tiles."), + O("tile_url", "Tile server", "The map", "text", + "where the map tiles come from", + "Any server that serves 256-pixel tiles as {z}/{x}/{y}.png will do, " + "including one of your own. The default is the standard " + "OpenStreetMap one, whose tiles are free to use within its usage " + "policy: identify yourself, cache what you fetch, and do not bulk " + "download. This program does all three.", + flags=("--tiles",), metavar="URL", + example="https://tile.openstreetmap.org/{z}/{x}/{y}.png"), + O("theme", "Colour theme", "The map", "choice", + "how the map looks: the colours, and whether the lines glow", + "The default draws a night-blue ground with height as colour, low " + "warm to high cold, which is what every other aircraft map does and " + "is the easiest to read. The rest are the screens the phrase 'air " + "defence display' calls to mind: a black tube, one phosphor, and thin " + "bright vector lines with a halo round them. Those have one colour to " + "spend, so height is brightness instead -- low is dim, high burns -- " + "the ground underneath is pushed well back so the lines carry the " + "picture, and a country is named in two letters rather than drawn as " + "a flag, because a flag needs half a dozen colours and a phosphor has " + "one. It applies to the window and to the animated pictures alike.", + choices=("night", "digital", "phosphor", "amber", "red"), + flags=("--theme",), metavar="NAME", example="phosphor", + guidance="night to read it, the others to look at it."), + O("map_brightness", "Map brightness", "The map", "int", + "how bright the map under the aircraft is drawn, as a percentage", + "The map is the ground, not the subject, so it is drawn dark enough " + "that the aircraft and their trails stay the brightest things on the " + "picture. Too dark and a coastline cannot be made out at all; too " + "bright and a city washes out the aircraft crossing it. This is which " + "way to err on the screen you are actually looking at. The vector " + "themes bend the middle of this range down hard, because a tinted " + "photograph of a county behind the vectors is the one thing that " + "stops a vector display looking like one -- but the top of the range " + "is a full-brightness map on every theme, so if the ground is barely " + "visible this is the setting that fixes it.", + unit="%", minimum=10, maximum=100, flags=("--map-brightness",), + metavar="PERCENT", example="70", + guidance="Turn it up until the coast and the roads are readable, and " + "no further. On a vector theme it takes rather more turning " + "up than on the default one."), + O("radius", "Map radius", "The map", "float", + "how far around the receiver the map reaches (0 = fit whatever was heard)", + "An aerial hears a hundred miles on a good day, and a position that " + "decoded wrongly can land anywhere on Earth. A map drawn to fit " + "everything heard is therefore drawn to fit the mistakes: the aircraft " + "come out a pixel wide in the middle of an empty continent. This frames " + "the picture on the receiver instead, so the scale stays the same from " + "one evening to the next and anything further out is left off the edge. " + "In the same unit as the speeds -- nautical miles with knots, statute " + "miles with mph, kilometres with kph.", + minimum=0.0, flags=("--radius",), example="100", + guidance="Set it to what your aerial can really hear. Zero goes back " + "to fitting whatever turned up, mistakes and all."), + O("airports", "Mark the airports", "The map", "bool", + "mark every aerodrome on the map, not only the ones flown between", + "A route names the two airports its aircraft is flying between, and " + "those are almost never the ones underneath: a receiver hears aircraft " + "over its own county, and the county's airports are what say where on " + "the map you are looking. They are asked for once per area from the " + "same map data the tiles are drawn from, and kept on disk for a month " + "afterwards, since a runway does not move.", + flags=("--airports",), off_flags=("--no-airports",), + guidance="Turn it off for a picture with nothing but the aircraft on " + "it, or where there is no network and nothing cached."), + O("rings", "Range rings on the pictures", "The map", "bool", + "faint discs at a quarter, a half and three quarters of the radius", + "Concentric on the receiver and translucent, so that they stack: the " + "ground inside the innermost is lifted three times, the next twice, " + "the outer once. What that gives is a sense of how far away a thing " + "is without measuring anything -- an aircraft two shades in is about " + "halfway to the edge of what this receiver hears. Each is labelled " + "with its distance. They need a receiver position and a radius, and " + "are not drawn without both.", + flags=("--rings",), off_flags=("--no-rings",), + guidance="Turn it off for a picture with nothing on it but the " + "aircraft and the ground."), + O("window_rings", "Range rings in the window", "The map", "bool", + "the same discs on the realtime display", + "The same rings as the pictures get, on the window instead. They are " + "separate settings because the two are looked at differently: a " + "picture is studied and a window is glanced at, and the rings help " + "one more than the other depending on which you are doing.", + flags=("--window-rings",), off_flags=("--no-window-rings",), + guidance="Turn it off if the window is busy enough already."), + O("box_opacity", "Box translucency", "Labels", "int", + "how solid the card behind each information box is, as a percentage", + "The words beside an aircraft are readable over water and not over a " + "city, so a card goes behind them. At nothing they sit straight on " + "the map as they used to; at the whole way the map does not show " + "through at all and the box is a solid panel. In between it darkens " + "the ground under the box, so the coastline still shows faintly " + "through it. It applies to the window and to the animated pictures " + "alike -- the pictures had no card at all before, and setting this " + "to nothing is what they used to look like.", + unit="%", minimum=0, maximum=100, flags=("--box-opacity",), + metavar="PERCENT", example="85", + guidance="Turn it up over a busy map and down over an empty one."), + O("labels", "Label the aircraft", "Labels", "bool", + "write the callsign, height and speed beside each aircraft", + "Height is the flight level -- hundreds of feet -- the way it is said " + "on the radio. Turning labels off leaves the shapes of the traffic, " + "which is worth seeing on a busy evening.", + flags=("--labels",), off_flags=("--no-labels",)), + O("speed_unit", "Speed in", "Labels", "choice", + "what to show speeds and distances in", + "Aircraft broadcast knots and the log keeps knots, because that is " + "what the standard sends; this is the unit they are shown in. It " + "changes the heading of the live display, the speeds written beside " + "each aircraft on the map and in the report, and the distance unit " + "that goes with them -- nautical miles with knots, statute miles with " + "miles an hour, kilometres with km/h, so that one picture never " + "carries two different miles.", + choices=("knots", "mph", "kph"), + flags=("--speed-unit",), metavar="UNIT", + guidance="knots is what aviation uses and what the aircraft actually " + "said. mph or kph if that is what means something to you."), +) + +OPTION_GROUPS = ("Receiver", "Listening", "Aircraft", "Animation", "The map", "Labels") + + +def in_group(group: str) -> list[Setting]: + return [o for o in OPTIONS if o.group == group] + + +def by_key(key: str) -> Setting | None: + return next((o for o in OPTIONS if o.key == key), None) + + +# What a zero means, per option: "0 s" is true and unhelpful. +_ZERO_MEANS = {"seconds": "until stopped", "speed": "fit to the length", + "trail": "the whole path"} + + +def format_option(option: Setting, value) -> str: + """Render an option the way the menu should show it.""" + if not value and option.key in _ZERO_MEANS: + return _ZERO_MEANS[option.key] + return format_value(option, value) + + +def describe(options: AircraftOptions) -> str: + """One line for a menu: what listening and drawing would do now.""" + how_long = ("until stopped" if not options.seconds + else f"{options.seconds:g} s") + where = "simulated" if options.simulate else "1090 MHz" + return (f"{where}, {how_long}, " + f"{'looked up' if options.lookup else 'no lookups'}, " + f"{options.picture}") + + +# --------------------------------------------------------------------------- +# Where the options are kept +# --------------------------------------------------------------------------- + +def options_path(directory=None) -> Path: + from .config import DEFAULT_CONFIG_DIR + + return Path(directory or DEFAULT_CONFIG_DIR) / "aircraft.yaml" + + +def load_options(directory=None) -> AircraftOptions: + """The saved options, or the defaults. A broken file is not an error.""" + options = AircraftOptions() + try: + body = yaml.safe_load(options_path(directory).read_text()) or {} + except (OSError, ValueError, yaml.YAMLError): + # A hand-edited file with a typo in it should cost the defaults, not + # the menu it is read from. + return options + if not isinstance(body, dict): + return options + known = set(options.__dict__) + for key, value in body.items(): + if key in known and value is not None: + try: + setattr(options, key, type(getattr(options, key))(value)) + except (TypeError, ValueError): + pass + return options + + +def save_options(options: AircraftOptions, directory=None) -> Path: + path = options_path(directory) + path.parent.mkdir(parents=True, exist_ok=True) + with open(path, "w") as fh: + yaml.safe_dump(options.to_dict(), fh, sort_keys=False, + default_flow_style=False) + return path + + +def logs_in(directory) -> list[Path]: + """Every ADS-B log in a directory, newest first.""" + try: + found = list(Path(directory).expanduser().glob("adsb_*.jsonl")) + except OSError: + return [] + return sorted(found, key=lambda p: p.stat().st_mtime, reverse=True) + + +def coordinates(text: str) -> tuple[float, float]: + """A LAT,LON pair, falling back to the default sky.""" + try: + lat, lon = (float(x) for x in str(text).split(",", 1)) + return lat, lon + except (TypeError, ValueError): + return 47.55, -122.30 + + +# --------------------------------------------------------------------------- +# Listening +# --------------------------------------------------------------------------- + +@dataclass +class Heard: + """What one listening session came to.""" + + frames: int = 0 + registry: object = None + log_path: Path | None = None + report_path: Path | None = None + kml_path: Path | None = None + picture: object = None + tracks: list = field(default_factory=list) + + @property + def aircraft(self) -> int: + return len(self.registry) if self.registry is not None else 0 + + +def windowed() -> bool: + """Whether the realtime window can be opened on this machine.""" + from . import livemap + + return livemap.available() + + +def open_device(console, options: AircraftOptions): + """The receiver, or an invented sky, or None if neither can be had.""" + from rich.panel import Panel + from rich.text import Text + + from .adsb import SimulatedSky, default_sky + from .device import RtlSdrDevice, RtlSdrError + + if options.simulate: + console.print("[yellow]simulated: these aircraft are not there." + "[/yellow]") + return SimulatedSky(default_sky(*coordinates(options.near)), + sample_rate=options.rate, realtime=True).open() + try: + device = RtlSdrDevice(index=options.device, + sample_rate=int(options.rate), + gain=options.gain, + agc=options.gain == "auto") + device.open() + except RtlSdrError as exc: + console.print(Panel(Text(str(exc)), + title="[red]cannot open the receiver", + border_style="red")) + return None + return device + + +def open_log(console, options: AircraftOptions, output_dir: str, + started: float, log_path=None): + """The frame log, or None if it was not wanted or cannot be written.""" + from .flightlog import FlightLog + + if not options.log: + return None + stamp = datetime.fromtimestamp(started).strftime("%Y-%m-%d_%H_%M_%S") + where = Path(log_path).expanduser() if log_path else \ + Path(output_dir).expanduser() / f"adsb_{stamp}.jsonl" + try: + return FlightLog(where, frequency=ADSB_HZ, sample_rate=options.rate, + receiver="simulated" if options.simulate else + f"device {options.device}", started=started) + except OSError as exc: + console.print(f"[red]cannot write {where}: {exc}[/red]") + return None + + +def pump(device, options: AircraftOptions, registry, log, book, + started: float, on_block=None, on_frame=None, stopping=None) -> int: + """Read the receiver until it stops, or until told to. + + The one loop both the terminal board and the window are driven from, so + that what is written to the log cannot depend on which one you happened + to be looking at. + """ + from .adsb import decode_frames + + total = 0 + device.tune(ADSB_HZ) + block = int(options.rate) # a second at a time + while stopping is None or not stopping(): + at = time.time() + samples = device.read_samples(block) + if samples is None or samples.size == 0: + break + for frame in decode_frames(samples, options.rate): + # The real time the frame arrived, not its offset in the block: + # everything downstream is a clock, and a log that started again + # from zero every second would be unusable. + when = at + frame.at_sample / options.rate + craft = registry.add(frame, when=when) + total += 1 + if log is not None: + log.append(frame, craft, when=when) + if options.lookup: + book.get(frame.icao, craft.callsign) + if on_frame is not None: + on_frame(frame, craft) + if on_block is not None: + on_block(total) + if options.seconds and time.time() - started >= options.seconds: + break + return total + + +def listen(console, options: AircraftOptions, output_dir: str, + log_path=None) -> Heard: + """Park on 1090 MHz and write down the aircraft overhead. + + Everything heard goes into the log as it arrives, because an aircraft is + overhead for four minutes and then gone: the screen is for the person + watching, and the log is for the report, the map and everything + afterwards. + """ + from .adsb import AircraftRegistry + from .flights import FlightBook + + heard = Heard() + device = open_device(console, options) + if device is None: + return heard + + started = time.time() + log = open_log(console, options, output_dir, started, log_path) + registry = AircraftRegistry() + console.print(f"[grey62]listening on {ADSB_HZ/1e6:g} MHz at " + f"{options.rate/1e6:g} MS/s — control-C to stop[/grey62]") + if log is not None: + console.print(f"[grey62]writing {log.path}[/grey62]") + + # The registers are asked while the listening runs rather than after it, + # so a registration and a route appear on the line as they arrive. The + # book answers immediately with what it knows and fills itself in later. + book = FlightBook(online=options.lookup, + schedules=schedule_names(options)) + display, live = _open_display(console, options, book, started) + + def on_block(total: int) -> None: + if live is not None: + display.update(registry, total, + log.path if log is not None else None) + live.update(display.render()) + elif not options.frames and total: + console.print(f"[grey62]{len(registry)} aircraft, " + f"{total} frames[/grey62]", highlight=False) + + def on_frame(frame, craft) -> None: + if options.frames: + console.print(f"[cyan]{frame.icao}[/cyan] " + f"{frame.describe()}", highlight=False) + + total = 0 + try: + total = pump(device, options, registry, log, book, started, + on_block=on_block, on_frame=on_frame) + except KeyboardInterrupt: + pass + finally: + if live is not None: + live.stop() + console.print("[grey62]stopped listening[/grey62]") + device.close() + if log is not None: + log.close() + heard.log_path = log.path + return finish(console, options, output_dir, heard, registry, book, + started, total) + + +def watch(console, options: AircraftOptions, output_dir: str, + log_path=None) -> Heard: + """The same listening, in a window with a map in it. + + The receiver runs on its own thread and the window paints from a copy of + what it found, so that a slow repaint can never cost a frame and a slow + network can never stop the picture moving. Everything else -- the log, + the report, the lookups, the map drawn at the end -- is exactly what the + passive capture does, because it is the same code. + """ + import threading + + from rich.panel import Panel + from rich.text import Text + + from . import livemap + from .adsb import AircraftRegistry + from .flightlog import read_position + from .flights import FlightBook + + heard = Heard() + if not livemap.available(): + console.print(Panel(Text(livemap.MISSING_QT), + title="[yellow]no window to open", + border_style="yellow")) + return heard + device = open_device(console, options) + if device is None: + return heard + + started = time.time() + log = open_log(console, options, output_dir, started, log_path) + registry = AircraftRegistry() + book = FlightBook(online=options.lookup, + schedules=schedule_names(options)) + # Set before the window is built, because the window reads its colours + # out of the palette this writes. + from .flightmap import set_theme + + set_theme(options.theme) + sky = livemap.Sky(unit=options.speed_unit, hold=options.hold, + home=read_position(options.location), + radius_nm=radius_in_nm(options) or 100.0, + brightness=max(10, options.map_brightness) / 100.0, + fade=max(0.0, options.fade), + airports=options.airports, + rings=options.window_rings, + box_opacity=max(0, options.box_opacity) / 100.0) + sky.started = started + sky.log_name = log.path.name if log is not None else "" + if options.simulate: + sky.note = "simulated" + + def told_about(craft): + """What a register says about one aircraft, for the window.""" + if not options.lookup: + return None + entry = book.get(craft.icao, craft.callsign) + if craft.located: + # A source that listed a whole day's stops has said which leg + # this is, once there is a position to read it with. + entry = book.resolve(entry, craft.latitude, craft.longitude) + return entry + + def on_block(total: int) -> None: + sky.update([livemap.blip_for(craft, told_about(craft)) + for craft in registry.aircraft.values()], + total, len(registry)) + + counted = {"frames": 0} + + def listening() -> None: + try: + counted["frames"] = pump(device, options, registry, log, book, + started, on_block=on_block, + stopping=lambda: sky.stopping) + except Exception as exc: # a window must survive it + sky.note = str(exc)[:60] + finally: + # However it ended -- the time ran out, the receiver stopped, or + # it fell over -- the window is told, and shuts itself. + sky.finished = True + + threads = [threading.Thread(target=listening, daemon=True, + name="adsb-receiver")] + # One thread serves both the map and the aerodromes, and either on its + # own is reason enough to start it: the aerodromes are a separate + # question to a separate service, and asking for them with the tiles + # turned off is a perfectly ordinary thing to want. + if options.basemap or options.airports: + threads.append(threading.Thread( + target=livemap.fetch_ground, args=(sky, options.tile_url), + daemon=True, name="adsb-basemap")) + for thread in threads: + thread.start() + console.print(f"[grey62]listening on {ADSB_HZ/1e6:g} MHz — close the " + "window to stop[/grey62]") + if log is not None: + console.print(f"[grey62]writing {log.path}[/grey62]") + try: + livemap.show(sky, "bandsaunter — aircraft on 1090 MHz") + finally: + sky.stopping = True + for thread in threads: + thread.join(timeout=3.0) + device.close() + if log is not None: + log.close() + heard.log_path = log.path + return finish(console, options, output_dir, heard, registry, book, + started, counted["frames"]) + + +def finish(console, options: AircraftOptions, output_dir: str, heard: Heard, + registry, book, started: float, total: int) -> Heard: + """Everything that happens once the listening stops. + + Shared by the passive capture and the window, so that closing a window + leaves exactly the same files behind as pressing control-C does. + """ + from .flightlog import read_logs, report, write_kml + + heard.frames = total + heard.registry = registry + if not registry: + console.print("[yellow]nothing heard. ADS-B needs an aerial cut for " + "1090 MHz; the whip that came with the dongle will " + "hear the airport and not much else.[/yellow]") + return heard + + aircraft_table(console, registry, total, options.speed_unit) + if options.lookup: + book.wait(12.0) + book.save() + lookup_table(console, registry, book) + + heard.tracks = read_logs(heard.log_path) if heard.log_path \ + else tracks_from(registry) + if options.kml: + where = (heard.log_path.with_suffix(".kml") if heard.log_path + else Path(output_dir).expanduser() / "aircraft.kml") + written = write_kml(where, heard.tracks, + book if options.lookup else None, + unit=options.speed_unit) + heard.kml_path = written + console.print(f"[green]{written}[/green]" if written + else "[yellow]nothing was placed on the map[/yellow]") + if heard.log_path is not None: + told = heard.log_path.with_suffix(".txt") + try: + told.write_text("\n".join(report( + heard.tracks, book if options.lookup else None, + title=f"bandsaunter — aircraft heard " + f"{datetime.fromtimestamp(started):%Y-%m-%d %H:%M}", + unit=options.speed_unit))) + heard.report_path = told + console.print(f"[green]{told}[/green]") + except OSError as exc: + console.print(f"[red]cannot write {told}: {exc}[/red]") + if options.draw_after: + heard.picture = draw(console, options, heard.tracks, + out_path=_picture_path(heard.log_path, options, + output_dir), + book=book if options.lookup else None) + return heard + + +def _open_display(console, options: AircraftOptions, book, started: float): + """A live table where there is a terminal to draw it on, else nothing. + + Piped output, a test or a log file gets the running count it had before: + a display that redraws itself four times a second is unreadable as a + stream of text, and worse than useless in a file. + """ + if options.frames or not getattr(console, "is_terminal", False): + return None, None + from rich.live import Live + + from .ui import AircraftDisplay + + display = AircraftDisplay(console, book=book if options.lookup else None, + hold=options.hold, unit=options.speed_unit) + display.started = started + live = Live(display.render(), console=console, refresh_per_second=4, + screen=False, transient=False, vertical_overflow="crop") + live.start() + return display, live + + +def _picture_path(log_path, options: AircraftOptions, output_dir: str) -> Path: + suffix = "." + options.picture + if log_path is not None: + return Path(log_path).with_suffix(suffix) + return Path(output_dir).expanduser() / f"aircraft{suffix}" + + +# --------------------------------------------------------------------------- +# Drawing +# --------------------------------------------------------------------------- + +def checked(console, options: AircraftOptions, tracks): + """Throw out impossible positions, if asked, and say how many went.""" + if not options.recheck: + return tracks + from .flightlog import recheck as recheck_tracks + + before = sum(len(t.fixes) for t in tracks) + tracks, dropped = recheck_tracks(tracks) + if dropped: + console.print(f"[yellow]dropped {dropped:,} of {before:,} positions " + f"({100.0 * dropped / max(1, before):.1f}%) that no " + "aircraft could have been in[/yellow]") + else: + console.print("[grey62]every position checks out[/grey62]") + return tracks + + +def schedule_names(options: AircraftOptions) -> list[str]: + """The schedule services to ask, in the order given.""" + return [name.strip() for name in str(options.schedules or "").split(",") + if name.strip()] + + +def radius_in_nm(options: AircraftOptions) -> float: + """The map radius as the drawing wants it, in nautical miles. + + The option is in whatever the speeds are in, because a picture that + measured its speeds in one unit and its own extent in another would be + a puzzle rather than a map. + """ + from .flightlog import speed_unit + + return max(0.0, float(options.radius)) / speed_unit(options.speed_unit)[3] + + +def draw(console, options: AircraftOptions, tracks, out_path, book=None): + """Draw the map, saying what it is drawing and what came out of it.""" + from .flightmap import animate, ffmpeg_available, set_theme + + out_path = Path(out_path).expanduser() + if out_path.suffix.lower() in (".mp4", ".mov", ".m4v") and \ + not ffmpeg_available(): + console.print("[yellow]ffmpeg is not installed; writing a GIF " + "instead[/yellow]") + out_path = out_path.with_suffix(".gif") + console.print(f"[grey62]drawing {out_path.name}…[/grey62]") + try: + set_theme(options.theme) + drawn = animate(tracks, out_path, book=book, fps=options.fps, + seconds=options.length, speed=options.speed, + width=options.width, trail_seconds=options.trail, + stale=options.stale, labels=options.labels, + fade=max(0.0, options.fade), + unit=options.speed_unit, ground=options.basemap, + tile_url=options.tile_url, + brightness=max(10, options.map_brightness) / 100.0, + airports=options.airports, + rings=options.rings, + box_opacity=max(0, options.box_opacity) / 100.0, + radius_nm=radius_in_nm(options), + centre=read_position(options.location)) + except (OSError, RuntimeError, ValueError) as exc: + console.print(f"[red]{exc}[/red]") + return None + if drawn is None: + console.print("[yellow]nothing was placed on the map: no aircraft " + "reported a position[/yellow]") + return None + size = drawn.path.stat().st_size / 1e6 + console.print(f"[green]{drawn.path}[/green] " + f"[grey62]{drawn.summary()}, {size:.1f} MB[/grey62]") + return drawn + + +def draw_log(console, options: AircraftOptions, paths, out_path=None): + """Read logs back and draw them: the whole of what ``flights`` does.""" + from .flightlog import read_logs + from .flights import FlightBook + + paths = [Path(p).expanduser() for p in paths] + tracks = read_logs(paths) + if not tracks: + console.print(f"[yellow]{paths[0].name} holds no frames[/yellow]") + return None + tracks = checked(console, options, tracks) + book = FlightBook(online=options.lookup, + schedules=schedule_names(options)) + if options.lookup: + for track in tracks: + book.get(track.icao, track.callsign) + book.wait(20.0) + book.save() + if out_path is None: + out_path = paths[0].with_suffix("." + options.picture) + return draw(console, options, tracks, out_path, + book if options.lookup else None) + + +# --------------------------------------------------------------------------- +# What was heard, on the screen +# --------------------------------------------------------------------------- + +def aircraft_table(console, registry, total: int, + unit: str = "knots") -> None: + """What was heard, as it was heard: no register, only the air.""" + from rich.table import Table + + from .flightlog import in_speed, speed_label + + t = Table(title=f"{len(registry)} aircraft, {total} frames", box=None, + header_style="bold") + for column in ("ICAO", "callsign", "altitude", "position", + f"speed ({speed_label(unit)})", "frames"): + t.add_column(column) + for craft in sorted(registry.aircraft.values(), key=lambda a: a.icao): + t.add_row(craft.icao, craft.callsign or "", + f"{craft.altitude_ft:,} ft" if craft.altitude_ft else "", + (f"{craft.latitude:.4f}, {craft.longitude:.4f}" + if craft.located else ""), + (f"{in_speed(craft.ground_speed_kt, unit):.0f} " + f"{craft.track_deg:.0f}°" + if craft.ground_speed_kt else ""), + str(craft.messages)) + console.print(t) + + +def lookup_table(console, registry, book) -> None: + """And what the registers say about them, kept separate on purpose.""" + from rich.table import Table + + rows = [] + for craft in sorted(registry.aircraft.values(), key=lambda a: a.icao): + entry = book.get(craft.icao, craft.callsign) + told = entry.summary() + if told or entry.country: + rows.append((craft.icao, craft.callsign or "", entry.country, + told or "—")) + if not rows: + return + t = Table(title="what the registers say", box=None, header_style="bold") + for column in ("ICAO", "callsign", "registered", + "aircraft, operator, route"): + t.add_column(column, overflow="fold") + for row in rows: + t.add_row(*row) + console.print(t) + + +def tracks_from(registry): + """Tracks from a registry, for a session that wrote no log. + + One position each: what is on the screen is all there is, because nothing + kept the ones before it. + """ + from .flightlog import Fix, Track + + out = [] + for craft in sorted(registry.aircraft.values(), key=lambda a: a.icao): + track = Track(icao=craft.icao, callsign=craft.callsign, + frames=craft.messages, first_seen=craft.first_seen, + last_seen=craft.last_seen) + if craft.located: + track.fixes.append(Fix(at=craft.last_seen, latitude=craft.latitude, + longitude=craft.longitude, + altitude_ft=craft.altitude_ft, + ground_speed_kt=craft.ground_speed_kt, + track_deg=craft.track_deg, + vertical_rate_fpm=craft.vertical_rate_fpm)) + out.append(track) + return out + + +def summarise(options: AircraftOptions) -> str: + """The options as one line, for a menu row.""" + return ", ".join(f"{o.label.lower()} {format_option(o, getattr(options, o.key))}" + for o in OPTIONS[:3]) diff --git a/bandsaunter/basemap.py b/bandsaunter/basemap.py new file mode 100644 index 0000000..ede263e --- /dev/null +++ b/bandsaunter/basemap.py @@ -0,0 +1,538 @@ +"""The ground under the aircraft: map tiles, fetched, cached and dimmed. + +A flight path over a black rectangle says how the aircraft moved and nothing +about where it was. Over a coastline it says which airport it left. So the +map behind the animation is a real one: standard raster tiles, fetched once, +kept on disk, reprojected onto the picture and dimmed until the aircraft are +the brightest thing on it. + +Three things follow from using somebody else's tile server, and all three are +obligations rather than options. Tiles are **cached** and never fetched +twice. Every request identifies the program in its User-Agent. And the +attribution the licence requires is drawn onto the picture, not left to a +readme nobody ships with a GIF. A drawing is capped at a few dozen tiles: an +aircraft map is a hobby drawing, not a reason to hammer a volunteer-funded +service. + +The PNG decoding is here for the same reason the PNG writing is in +:mod:`bandsaunter.images`: a scanner that cannot draw a map because an +imaging library is missing is worse than one that draws it from zlib and +numpy, which is all this needs. +""" + +from __future__ import annotations + +import json +import math +import os +import struct +import time +import urllib.parse +import urllib.request +import zlib +from pathlib import Path + +import numpy as np + +from . import __version__ + +__all__ = ["decode_png", "tile_of", "choose_zoom", "fetch_tile", "mosaic", + "ground_under", "TILE_URL", "ATTRIBUTION", "MAX_TILES", "MAX_ZOOM", + "cache_dir", "PNGError", "airports_in", "AIRPORTS_URL", + "tile_span"] + +# The standard OpenStreetMap tiles. Any {z}/{x}/{y} server can be put here +# instead; nothing below knows anything about this one in particular. +TILE_URL = "https://tile.openstreetmap.org/{z}/{x}/{y}.png" + +# Drawn onto every picture that used the tiles. The licence requires it and +# a GIF travels without its readme. +ATTRIBUTION = "MAP DATA (C) OPENSTREETMAP CONTRIBUTORS" + +# The backstop on how many tiles one drawing may fetch. Reached only by a +# view so wide that no zoom covers it without fetching half a country; the +# usual limit is the size of the picture, which is asked for instead. A +# screenful at a hundred-mile radius comes to about a hundred and fifty, +# fetched once and kept. +MAX_TILES = 220 +MAX_ZOOM = 13 + +# How much more detail to fetch than the picture holds. Averaging several +# source pixels into each output one is what makes lettering and coastlines +# come out smooth; taking exactly one leaves them as hard as they were. +OVERSAMPLE = 1.4 +MIN_ZOOM = 2 +TILE_PIXELS = 256 + +USER_AGENT = (f"bandsaunter/{__version__} " + "(+https://github.com/topics/rtl-sdr; aircraft map drawing)") + +# Politeness between requests to a volunteer-funded service. Only paid on a +# tile that was not already on the disk. +FETCH_PAUSE = 0.12 + + +# Where to ask what aerodromes are in a piece of the world. The same OSM +# data the tiles are drawn from, asked as a question rather than a picture. +AIRPORTS_URL = "https://overpass-api.de/api/interpreter" + +# One question covers a whole view and is kept for a month, because runways +# do not move. The same politeness as the tiles: cache it, say who is +# asking, and do not ask twice for the same thing. +AIRPORT_CACHE_DAYS = 30 + +# Bumped when the question changes, so that answers to the old one are asked +# again rather than believed. +AIRPORT_CACHE_VERSION = 2 + +# Enough to name what is under an aircraft and not so many that the map is a +# list of airstrips. +MOST_AIRPORTS = 40 + + +class PNGError(ValueError): + """A PNG this decoder cannot read.""" + + +# --------------------------------------------------------------------------- +# Reading a PNG +# --------------------------------------------------------------------------- + +_CHANNELS = {0: 1, 2: 3, 3: 1, 4: 2, 6: 4} + + +def decode_png(data: bytes) -> np.ndarray: + """A PNG's pixels as an ``(h, w, 3)`` array of bytes. + + Eight bits a channel and no interlacing, which is what every tile server + sends; anything else raises rather than being guessed at. + """ + if data[:8] != b"\x89PNG\r\n\x1a\x08"[:8] and data[:8] != \ + b"\x89PNG\r\n\x1a\n": + raise PNGError("not a PNG") + width = height = depth = colour = interlace = 0 + palette = None + body = bytearray() + at = 8 + while at + 8 <= len(data): + length, tag = struct.unpack(">I4s", data[at:at + 8]) + chunk = data[at + 8:at + 8 + length] + at += 12 + length + if tag == b"IHDR": + (width, height, depth, colour, _compression, _filter, + interlace) = struct.unpack(">IIBBBBB", chunk) + elif tag == b"PLTE": + palette = np.frombuffer(chunk, dtype=np.uint8).reshape(-1, 3) + elif tag == b"IDAT": + body += chunk + elif tag == b"IEND": + break + if depth != 8: + raise PNGError(f"{depth}-bit PNG: only 8 bits a channel is read here") + if interlace: + raise PNGError("interlaced PNG") + if colour not in _CHANNELS: + raise PNGError(f"colour type {colour}") + if not width or not height: + raise PNGError("no image") + + channels = _CHANNELS[colour] + raw = _unfilter(zlib.decompress(bytes(body)), width, height, channels) + if colour == 3: + if palette is None: + raise PNGError("palette image with no palette") + return palette[np.clip(raw[:, :, 0], 0, len(palette) - 1)] + if colour == 0: + return np.repeat(raw, 3, axis=2) + if colour == 4: + return np.repeat(raw[:, :, :1], 3, axis=2) + return raw[:, :, :3] + + +def _unfilter(data: bytes, width: int, height: int, + channels: int) -> np.ndarray: + """Undo the per-row filters PNG applies before compressing. + + The five of them, from the specification. None and Up are whole-row + arithmetic; Sub is a running total along the row, which is a cumulative + sum once the bytes are grouped by which channel they belong to; Average + and Paeth each need the byte before them to have been worked out + already, so those two are the only ones that walk the row. + """ + stride = width * channels + if len(data) < height * (stride + 1): + raise PNGError("truncated image data") + out = np.zeros((height, stride), dtype=np.uint8) + previous = np.zeros(stride, dtype=np.uint8) + at = 0 + for row in range(height): + kind = data[at] + line = np.frombuffer(data, dtype=np.uint8, count=stride, + offset=at + 1).astype(np.uint16) + at += stride + 1 + if kind == 0: + current = line.astype(np.uint8) + elif kind == 1: + current = np.empty(stride, dtype=np.uint8) + for offset in range(channels): + current[offset::channels] = np.cumsum( + line[offset::channels], dtype=np.uint32) % 256 + elif kind == 2: + current = ((line + previous) % 256).astype(np.uint8) + elif kind in (3, 4): + current = _walk_row(bytes(line.astype(np.uint8)), previous, + channels, kind) + else: + raise PNGError(f"filter type {kind}") + out[row] = current + previous = current + return out.reshape(height, width, channels) + + +def _walk_row(line: bytes, previous: np.ndarray, channels: int, + kind: int) -> np.ndarray: + """Average and Paeth: each byte needs the one ``channels`` back.""" + up = previous.tolist() + out = [0] * len(line) + for i, value in enumerate(line): + left = out[i - channels] if i >= channels else 0 + above = up[i] + if kind == 3: + out[i] = (value + ((left + above) >> 1)) & 0xFF + else: + upleft = up[i - channels] if i >= channels else 0 + base = left + above - upleft + da, db, dc = abs(base - left), abs(base - above), abs(base - upleft) + near = left if (da <= db and da <= dc) else ( + above if db <= dc else upleft) + out[i] = (value + near) & 0xFF + return np.array(out, dtype=np.uint8) + + +# --------------------------------------------------------------------------- +# Which tiles +# --------------------------------------------------------------------------- + +def tile_of(lat: float, lon: float, zoom: int) -> tuple[float, float]: + """Where a coordinate falls in the tile grid, in fractional tiles. + + Web Mercator, which is what every {z}/{x}/{y} tile server serves and is + not what this program's maps are drawn in -- hence the resampling + further down rather than a straight paste. + """ + lat = max(-85.05112878, min(85.05112878, lat)) + n = float(2 ** zoom) + x = (lon + 180.0) / 360.0 * n + radians = math.radians(lat) + y = (1.0 - math.asinh(math.tan(radians)) / math.pi) / 2.0 * n + return x, y + + +def tile_span(south: float, west: float, north: float, east: float, + zoom: int) -> tuple[int, int]: + """How many tiles wide and tall a box is at one zoom.""" + x0, y0 = tile_of(north, west, zoom) + x1, y1 = tile_of(south, east, zoom) + return (int(math.floor(x1)) - int(math.floor(x0)) + 1, + int(math.floor(y1)) - int(math.floor(y0)) + 1) + + +def choose_zoom(south: float, west: float, north: float, east: float, + width: int = 0, max_tiles: int = MAX_TILES, + most: int = MAX_ZOOM) -> int: + """The zoom to fetch at: enough for the picture, and no more. + + ``width`` is how many pixels across the picture will be. Without it + this fetched whatever the tile budget allowed, which is the wrong + question in both directions: on a small picture it fetched more than + could be shown, and on a large one it fetched less and the map was + blown up to fit -- which is what a low-resolution map looks like. + + So it climbs until the tiles hold at least as many pixels as the + picture wants, and stops there. The budget is the backstop, for a view + so wide that no zoom can cover it without fetching half a country. + """ + best = MIN_ZOOM + for zoom in range(MIN_ZOOM, min(most, MAX_ZOOM) + 1): + wide, tall = tile_span(south, west, north, east, zoom) + if wide * tall > max_tiles: + break + best = zoom + if width and wide * TILE_PIXELS >= width * OVERSAMPLE: + break # detail enough; more would only cost time + return best + + +# --------------------------------------------------------------------------- +# Fetching them, once +# --------------------------------------------------------------------------- + +def cache_dir() -> Path: + root = os.environ.get("XDG_CACHE_HOME") or "~/.cache" + return Path(root).expanduser() / "bandsaunter" / "tiles" + + +def fetch_tile(zoom: int, x: int, y: int, url: str = TILE_URL, + timeout: float = 10.0, cache: Path | None = None) -> bytes | None: + """One tile, from the disk if it has ever been fetched before. + + Returns the PNG bytes, or None if it could not be had. A missing tile is + not an error: the map is drawn with a hole in it, which is better than no + map and much better than an exception in the middle of an animation. + """ + where = (cache if cache is not None else cache_dir()) / str(zoom) / str(x) + path = where / f"{y}.png" + try: + return path.read_bytes() + except OSError: + pass + request = urllib.request.Request(url.format(z=zoom, x=x, y=y), + headers={"User-Agent": USER_AGENT}) + try: + with urllib.request.urlopen(request, timeout=timeout) as answer: + body = answer.read(2_000_000) + except Exception: + return None + try: + where.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(".tmp") + tmp.write_bytes(body) + tmp.replace(path) + except OSError: + pass # an unwritable cache is not a reason to stop + return body + + +def mosaic(south: float, west: float, north: float, east: float, zoom: int, + fetch=fetch_tile, pause: float = FETCH_PAUSE, **kw): + """Every tile the box touches, stitched into one image. + + Returns the pixels and where their top-left corner sits in the world, in + tile-grid pixels at this zoom, so the resampling below can place them. + """ + x0, y0 = tile_of(north, west, zoom) + x1, y1 = tile_of(south, east, zoom) + left, top = int(math.floor(x0)), int(math.floor(y0)) + right, bottom = int(math.floor(x1)), int(math.floor(y1)) + span = 2 ** zoom + wide, tall = right - left + 1, bottom - top + 1 + if wide <= 0 or tall <= 0 or wide * tall > MAX_TILES * 4: + return None, 0, 0 + canvas = np.zeros((tall * TILE_PIXELS, wide * TILE_PIXELS, 3), + dtype=np.uint8) + got = 0 + for row in range(tall): + for column in range(wide): + tx, ty = (left + column) % span, top + row + if not 0 <= ty < span: + continue + body = fetch(zoom, tx, ty, **kw) + if body is None: + continue + try: + tile = decode_png(body) + except (PNGError, zlib.error, ValueError): + continue + if tile.shape[0] != TILE_PIXELS or tile.shape[1] != TILE_PIXELS: + continue + canvas[row * TILE_PIXELS:(row + 1) * TILE_PIXELS, + column * TILE_PIXELS:(column + 1) * TILE_PIXELS] = tile + got += 1 + if pause: + time.sleep(pause) + if not got: + return None, 0, 0 + return canvas, left * TILE_PIXELS, top * TILE_PIXELS + + +# --------------------------------------------------------------------------- +# What is on the ground +# --------------------------------------------------------------------------- + +def _resample(values: np.ndarray, edges: np.ndarray, axis: int) -> np.ndarray: + """Average each output cell over the source pixels that fall in it. + + A box filter, done as the difference of a running total so the whole + axis is two passes rather than a loop. Where the source is coarser than + the output -- a map zoomed in further than the tiles go -- a cell covers + less than one source pixel, and it takes that one. + """ + length = values.shape[axis] + starts = np.clip(np.floor(edges[:-1]).astype(np.int64), 0, length - 1) + ends = np.clip(np.ceil(edges[1:]).astype(np.int64), 1, length) + ends = np.maximum(ends, starts + 1) + running = np.cumsum(values, axis=axis, dtype=np.float64) + pad = np.zeros_like(np.take(running, [0], axis=axis)) + running = np.concatenate([pad, running], axis=axis) + total = (np.take(running, ends, axis=axis) + - np.take(running, starts, axis=axis)) + counts = (ends - starts).astype(np.float64) + shape = [1] * values.ndim + shape[axis] = counts.size + return (total / counts.reshape(shape)).astype(np.float32) + + +def _airport_cache(box) -> Path: + root = os.environ.get("XDG_CACHE_HOME") or "~/.cache" + name = "_".join(f"{round(v, 1):+06.1f}" for v in box) + return Path(root).expanduser() / "bandsaunter" / "airports" / f"{name}.json" + + +def airports_in(south: float, west: float, north: float, east: float, + url: str = AIRPORTS_URL, timeout: float = 45.0, + cache: Path | None = None, ask=None) -> list[dict]: + """Every aerodrome in a piece of the world, with a code and a position. + + Asked of the same OpenStreetMap data the tiles are drawn from, as a + question rather than a picture, and kept on disk afterwards: a runway + does not move, so one question covers a view for a month. + + An aerodrome with no code is left out. A map wants to say *which* + airport an aircraft is over, and there are a great many landing strips + with a name and nothing else; the ones worth marking have a code. + """ + box = (round(south, 1), round(west, 1), round(north, 1), round(east, 1)) + path = cache if cache is not None else _airport_cache(box) + try: + body = json.loads(path.read_text()) + if int(body.get("version") or 0) >= AIRPORT_CACHE_VERSION and \ + time.time() - float(body.get("fetched_at") or 0) < \ + AIRPORT_CACHE_DAYS * 86_400: + return body.get("airports") or [] + except (OSError, ValueError): + pass + + found: list[dict] = [] + try: + raw = (ask or _ask_overpass)(box, url, timeout) + for element in (raw or {}).get("elements", []): + tags = element.get("tags") or {} + code = _airport_code(tags) + lat = element.get("lat") or (element.get("center") or {}).get("lat") + lon = element.get("lon") or (element.get("center") or {}).get("lon") + if not code or lat is None or lon is None: + continue + found.append({"code": code, + "name": (tags.get("name") or "").strip(), + "icao": bool(tags.get("icao")), + "latitude": float(lat), "longitude": float(lon)}) + except Exception: + return [] # a map with no airports on it, not a crash + + # The ones with a real ICAO code first, since those are the ones an + # aircraft is likely to be flying to or from. + found.sort(key=lambda a: (0 if a.get("icao") else 1, a["code"])) + # One airport is often tagged twice -- a point for the terminal and an + # outline for the field -- and marking it twice writes its name over + # itself. + once: dict[str, dict] = {} + for one in found: + once.setdefault(one["code"], one) + found = list(once.values())[:MOST_AIRPORTS] + try: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps({"fetched_at": time.time(), + "version": AIRPORT_CACHE_VERSION, + "airports": found})) + except OSError: + pass + return found + + +def _airport_code(tags: dict) -> str: + """What to call an aerodrome, or nothing if it has no code at all. + + The ICAO code where it has one. Otherwise a reference, and only when it + is four letters: a great many landing strips carry a local identifier + like "14AZ" or "MX-0492", which names nothing anybody would recognise + and turns a map into a list of airstrips. + """ + icao = (tags.get("icao") or "").strip().upper() + if len(icao) == 4 and icao.isalpha(): + return icao + ref = (tags.get("ref") or "").strip().upper() + if len(ref) == 4 and ref.isalpha(): + return ref + return "" + + +def _ask_overpass(box, url: str, timeout: float) -> dict: + """The one question this program asks Overpass.""" + south, west, north, east = box + where = f"({south},{west},{north},{east})" + # Relations as well as nodes and ways: a big airport is a relation more + # often than not -- Tucson International and Davis-Monthan both are -- + # so asking only for the other two finds every airstrip in the county + # and misses the two the county is known for. + query = ("[out:json][timeout:25];" + f'(node["aeroway"="aerodrome"]{where};' + f' way["aeroway"="aerodrome"]{where};' + f' relation["aeroway"="aerodrome"]{where};);' + "out center tags;") + request = urllib.request.Request( + url, data=urllib.parse.urlencode({"data": query}).encode(), + headers={"User-Agent": USER_AGENT}) + with urllib.request.urlopen(request, timeout=timeout) as answer: + return json.loads(answer.read(4_000_000).decode("utf8", "replace")) + + +# --------------------------------------------------------------------------- +# Putting it under the picture +# --------------------------------------------------------------------------- + +def ground_under(south: float, west: float, north: float, east: float, + width: int, height: int, shades: int = 32, + fetch=fetch_tile, zoom: int | None = None, + **kw) -> np.ndarray | None: + """The map for one picture, as ``shades`` levels of brightness. + + The tiles are Web Mercator and the picture is not, so every output pixel + asks the mosaic where its own latitude and longitude landed rather than + the mosaic being pasted in. Over the couple of hundred miles a receiver + hears, the difference is a few pixels of drift at the top of the frame -- + which is a few pixels an aircraft would be drawn wrong by, and the whole + point of putting a coastline under it is that the coastline is where the + aircraft was. + + Returns None when nothing could be fetched, which the caller draws as the + plain grid it drew before. + """ + if width < 1 or height < 1 or north <= south or east <= west: + return None + if zoom is None: + zoom = choose_zoom(south, west, north, east, width=width) + tiles, origin_x, origin_y = mosaic(south, west, north, east, zoom, + fetch=fetch, **kw) + if tiles is None: + return None + + # The edges of each output pixel rather than its middle, so that what + # lands in it can be averaged. Taking the nearest source pixel instead + # throws away most of a tile -- the mosaic is commonly half again the + # size of the picture -- and what survives is the aliasing: hard, broken + # lettering and roads that come and go along their length. + lons = west + (east - west) * np.arange(width + 1) / width + lats = north - (north - south) * np.arange(height + 1) / height + span = float(2 ** zoom) * TILE_PIXELS + xs = (lons + 180.0) / 360.0 * span - origin_x + clipped = np.clip(lats, -85.05112878, 85.05112878) + ys = (1.0 - np.arcsinh(np.tan(np.radians(clipped))) / math.pi) / 2.0 \ + * span - origin_y + + # Brightness only, inverted, and dimmed. Inverted because a printed map + # is ink on white paper and this picture is the other way round: the + # things drawn on the map -- coastlines, roads, the names of towns -- + # are the dark parts of a tile, and they are what should show against a + # night background. Dimmed because the map is the ground under the + # aircraft rather than the subject: anything drawn on top has to stay the + # brightest thing on the picture. + whole = (0.299 * tiles[:, :, 0] + 0.587 * tiles[:, :, 1] + + 0.114 * tiles[:, :, 2]).astype(np.float32) + luma = _resample(_resample(whole, ys, axis=0), xs, axis=1) + low, high = float(luma.min()), float(luma.max()) + if high - low < 1.0: + levels = np.zeros_like(luma) + else: + levels = 1.0 - (luma - low) / (high - low) + return np.clip((levels * (shades - 1)).round(), 0, + shades - 1).astype(np.uint8) diff --git a/bandsaunter/cli.py b/bandsaunter/cli.py index 9f2b8df..9db58f0 100755 --- a/bandsaunter/cli.py +++ b/bandsaunter/cli.py @@ -7,7 +7,6 @@ import json import signal import sys import time -from datetime import datetime from pathlib import Path from rich.console import Console @@ -23,8 +22,7 @@ from .bandplan import CATEGORIES, PRESETS, fmt_hz, in_category, search from .config import (DEFAULT_CONFIG_DIR, DEFAULT_CONFIG_PATH, ScanConfig, is_first_run, list_profiles, load_config, load_default, save_config, save_default) -from .device import (RtlSdrDevice, RtlSdrError, list_devices, - set_driver_messages) +from .device import RtlSdrError, list_devices, set_driver_messages from .librtlsdr import load_error from . import settings as st from .ranges import (RangeError, ScanRange, build_plan, parse_range_list) @@ -57,7 +55,9 @@ examples: bandsaunter bands --category Aviation browse the US band plan bandsaunter devices list attached dongles bandsaunter adsb read the aircraft on 1090 MHz + bandsaunter adsb --window live map of the aircraft bandsaunter flights --out sky.gif animate what they did + bandsaunter flights --theme phosphor draw it as a vector display bandsaunter scan -b 2m --simulate try it without hardware """) # The GNU form: the version, then who holds the copyright and what the @@ -174,8 +174,15 @@ examples: "(default: adsb_