cockpit: use the JSON API, with per-command screens and docs; JSON API additions and fixes, #9454

[ThomasWaldmann]

Rebuilds the cockpit TUI (borg --cockpit) on top of the documented JSON API, adds the JSON objects that were still missing for it and fixes the gaps in borg's JSON output that showed up while doing so. Fixes #9454.

The cockpit

The cockpit runs the borg command as a subprocess with --log-json --progress injected and builds its display from the JSON objects borg emits for frontends (docs/internals/frontends.rst), like Vorta does. It no longer classifies text lines by their first character.

Architecture (src/borg/cockpit/):

• runner.py runs borg with the options injected in front of the subcommand, reads stdout/stderr as pipes and turns the stderr lines into typed events. stdin is a pipe, so prompts can be answered. On POSIX, borg runs in a new session: it has no controlling terminal and cannot paint over the TUI. Unterminated lines (prompts) are passed on after 1 s of idle time; of a line longer than 1 MiB only the start is passed on, so the buffer is bounded. The borg it runs gets BORG_COCKPIT=false, so a cockpit enabled via the environment or default.yaml does not start another one.
• events.py: parse_json_line() gives LogMessage, ProgressPercent, ProgressMessage, ArchiveProgress, FileStatus, ArchiveStatus, Question or UnknownJson; the runner adds RawLine and ProcessFinished. Tolerant to unknown types and missing keys.
• session.py: the state of one run, without any Textual dependency: the statistics, archive counts, phases (by operation id / msgid), the current progress text, rates, warnings, and a bounded line buffer that the UI drains at 5 Hz, so a --list flood costs little more than the JSON parsing.
• screens.py and widgets.py: the screen is selected by the subcommand. CreateScreen (create, import-tar, recreate, transfer): files, sizes with the deduplication ratio, the counts by status, warnings, the current path, files/s and MB/s with a sparkline. ExtractScreen (extract, export-tar): a progress bar with percentage and ETA, MB/s, the --list counts. GenericScreen (everything else): elapsed time, warnings, the exit code, the archives listed by prune / delete / undelete (marked as dry-run if nothing was changed), and the phases borg reports progress for, with a bar per phase.
• prompt.py: a dialog for question_prompt (YES / NO buttons and a free-text field, the answer goes to borg's stdin) and the quit confirmation.

Where the numbers come from: only from borg's statistics, the final --json output (added to the command line for create and import-tar) or else archive_progress. They do not depend on --list / --filter, which only determine what the log shows. What borg does not report is shown as - (the counts by status for transfer and for a dry-run).

Robustness:

• Text from borg (paths, archive names, messages) is never interpreted as markup (a path containing [/ used to crash the app and thus end the backup), and control characters are shown as U+FFFD, so a file name cannot send escape sequences to the terminal.
• borg --cockpit refuses, with an error message, what cannot work in it: commands reading from stdin (create NAME -, --paths-from-stdin, import-tar NAME -, key import - / --paper, serve), commands writing their data to stdout (extract --stdout, export-tar NAME -), and running without a terminal (it used to write escape sequences into the redirected output and never end).
• SIGTERM, SIGHUP and SIGINT end it in an orderly way: borg is terminated and waited for, the cockpit exits with borg's exit code (it was a traceback and rc 1).
• q (or Ctrl-C) asks for confirmation while borg still runs, because quitting terminates borg.

Behaviour: --list is only passed through when given. The cockpit exits with the exit code of the borg command (it was always 0 before). t toggles the translator.

Docs: a usage page (docs/usage/cockpit.rst), linked from the usage toctree and the installation docs.

JSON API: additions

• file_status objects are now also output by extract --list and export-tar --list with --log-json (status + / -), as create does. The text listing of export-tar gets the + prefix extract's listing already had.
• New archive_status objects, one per listed archive, for prune --list / --list-kept / --list-pruned, delete --list and undelete --list with --log-json: status (kept, pruned, deleted, undeleted), dry_run (the status is what would be done), name / archive, id, time and message (the text line). Prune additionally gives the keys of the archive objects of its --json output. The commands share Archiver.print_archive_status(), the counterpart of print_file_status().

Compatibility: with --log-json, these --list lines used to be log_message objects (logger borg.output.list) carrying the text line. A frontend that only shows log_message objects does not show these listings anymore until it handles file_status / archive_status.

JSON API: fixes

• recreate and transfer output archive_progress objects with --log-json --progress, as documented. They printed text progress lines, because their archive objects were created without log_json.
• The final archive_progress object (finished: true) has the final statistics. The progress is rate limited, so nothing reported what was processed after the previous object, or anything at all for a short operation (a transfer of 10000 files ended at nfiles: 5653). It had lost its statistics because they were all 0 for create, see borg create: Everything 0 archive_progress at end of backup #6570: create and import-tar reported the archive's statistics before adding the ones of the item processors. They add them first now.
• import-tar and recreate count the items by their status (files_stats), like create does. The counts were always 0, also "Added files" of import-tar --stats.
• create --dry-run and recreate --dry-run report their progress (--progress showed nothing, there were no archive_progress objects). The deduplicated size is unknown for a dry-run: absent from the JSON objects, 0 B in the text progress.

All of this is documented in the frontends docs, with examples.

Verified

• Unit tests: the event parser, the session and the runner (with a fake borg script, including a prompt round trip via stdin) run without Textual; the app tests drive each screen with a fake runner and cover the layout, the dialogs, markup-like and control characters, signals and quitting. They also pass with Textual 6.8.0, the declared minimum.
• Archiver tests for the JSON objects of create, extract, export-tar, import-tar, recreate, transfer, prune, delete and undelete, including the final statistics and the dry-runs.
• With real borg on macOS, driving the TUI headless or in a pty: create (with --list, --filter, --dry-run, a rerun on the same series; 120000 files), extract, recreate, transfer, import-tar, check --repair answering NO and YES, compact, prune, an encrypted repo without a passphrase, quitting with and without confirmation, SIGTERM / SIGHUP.
• Overhead, 120000 files: plain create 18.1 s, with text --list 19.5-20.0 s, in the cockpit with --list 20.1-20.8 s.

Known limitations

• Passphrases must come from the environment; the cockpit shows a hint if borg asks for one (it recognizes the text of Python's getpass fallback).
• Warnings share the bounded line buffer with the --list lines, and nothing is printed to the terminal after the cockpit has ended.
• The only test running real borg through the real app is the slow smoke test (FreeBSD / Windows only).
• On Windows, the child is not detached from the console (no setsid) and signals are not handled; untested there beyond CI.

Follow-ups, not in this PR

A warnings view and a summary after exit, a JSON type for passphrase prompts, time-based throttling of progress_percent. Found on the way, not touched: recreate counts the item metadata into the archive's original size (shown by borg info afterwards), and recreate --stats prints no statistics.

Commits

1. cockpit: process borg's --log-json output instead of parsing text lines
2. cockpit: per-command screens and the final statistics
3. docs: add a usage page for the cockpit TUI
4. cockpit: rename two constants bandit mistakes for hardcoded passwords
5. extract/export-tar: file_status JSON objects, prune: archive_status JSON objects
6. cockpit: use the file_status and archive_status objects
7. delete/undelete: archive_status JSON objects for --list, with a status key
8. cockpit: show the archives delete and undelete list
9. cockpit: show text from borg as it is, never as markup
10. tests: export-tar --list: do not depend on the line ending
11. cockpit: refuse commands that read from stdin
12. cockpit: do not start without a terminal
13. cockpit: bound the line buffer, refuse commands writing data to stdout
14. cockpit: the borg it runs must not start a cockpit again
15. cockpit: end in an orderly way on SIGTERM, SIGHUP and SIGINT
16. cockpit: do not show control characters
17. cockpit: ask for confirmation before quitting while borg runs
18. cockpit: the statistics are borg's statistics, not the counts of the --list lines
19. recreate: output archive_progress JSON objects with --log-json --progress
20. transfer: output archive_progress JSON objects with --log-json --progress
21. import-tar/recreate: count the items by their status, like create does
22. archive_progress: the final object has the final statistics
23. create --dry-run --progress: show the progress
24. recreate --dry-run --progress: show the progress
25. archive_status: add a dry_run key

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 92.78017% with 67 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.69%. Comparing base (219d145) to head (7007944).
⚠️ Report is 1 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/borg/cockpit/widgets.py	90.86%	17 Missing and 4 partials ⚠️
src/borg/cockpit/app.py	80.85%	12 Missing and 6 partials ⚠️
src/borg/cockpit/runner.py	81.05%	13 Missing and 5 partials ⚠️
src/borg/cockpit/session.py	98.48%	1 Missing and 3 partials ⚠️
src/borg/cockpit/screens.py	92.68%	3 Missing ⚠️
src/borg/archiver/create_cmd.py	86.66%	2 Missing ⚠️
src/borg/cockpit/prompt.py	96.87%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10353      +/-   ##
==========================================
+ Coverage   88.25%   88.69%   +0.44%     
==========================================
  Files         103      107       +4     
  Lines       18898    19619     +721     
  Branches     2938     3028      +90     
==========================================
+ Hits        16678    17401     +723     
- Misses       1544     1549       +5     
+ Partials      676      669       -7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.