Telegram Tracker: a Python-based open-source tool for Telegram

---

---

Overview

Telegram Tracker collects public Telegram channels through Telegram's API and turns the collected files into a messages dataset and a channel network.

• collect saves each channel's profile metadata and posts as JSON, plus aggregate files about the chats found in that data (e.g., forward sources). You can collect one channel or a list of channels from a text file. The same data can also go to an SQLite database, instead of the files or beside them (see Storage modes).
• build-dataset reads the collected data (the files, or the database) and writes msgs_dataset.csv and collected_chats.xlsx.
• build-network reads the collected chats and writes a directed graph, Graph.gexf, and a preview image, network.png.
• collect --include-media and download-media save the photos, videos, audio and documents attached to the collected messages. This is off by default (see Downloading media).

The two builders run offline: they read local data only and never connect to Telegram.

The changes in each release are listed in CHANGELOG.md.

Contents:

• Requirements
• Installation and setup
• Logging in and saved credentials
• Account profiles
• First login
• Commands
• Collecting channels
• Collecting part of a history
• Updating saved histories
• Batches, concurrency, and rate limits
• Output files
• Storage modes
• Reading the database
• Database backups and schema versions
• Collection status, locking, and recovery
• Building the dataset and the network
• Downloading media
• Known limits
• Running the tests
• License

Requirements

• Python 3.14. The project is tested with Python 3.14.7. Pinned library versions and the tested environment are in docs/compatibility.md.
• For --storage sqlite or both and for builds reading a collection database: an SQLite library of 3.37.2 or newer with JSON1, the one Python's sqlite3 module loads (python -c "import sqlite3; print(sqlite3.sqlite_version)"). The collection database's version check accepts 3.31 to 3.37.1, but those versions are unsupported and Python's module may fail to load them; older ones are refused. --storage files creates no collection database, but Telethon sessions still use SQLite. build-urls writes a separate SQLite database even when reading files.
• Telegram API credentials: a Telegram account, plus an app api_id and api_hash.
• The libraries in requirements.txt: Telethon 1.x, pandas, openpyxl, tqdm, NetworkX, Matplotlib, and python-louvain for community detection.

Installation and setup

1. Get the code. Clone the repository:

git clone https://github.com/estebanpdl/telegram-tracker.git
cd telegram-tracker

Or download the ZIP file from GitHub and unpack it.

2. Create a Python 3.14 environment and install the requirements. Using an isolated environment keeps the pinned versions apart from other projects. With conda or mamba:

conda create -n telegram-tracker python=3.14
conda activate telegram-tracker
python -m pip install -r requirements.txt
python -m pip check

Any other Python 3.14 virtual environment works the same way: activate it, then run the two python -m pip commands. pip check should report No broken requirements found. Always install with python -m pip from the activated environment, so packages go into that environment.

3. Log in once. Get an API ID and API hash at my.telegram.org, then run:

python main.py auth login

It asks for the API ID, the API hash (not shown while you type), and your phone number, logs in to Telegram, and saves the three values so that you never type them again (see Logging in and saved credentials). If you prefer, fill in config/config.ini instead, as earlier versions required:

[Telegram API credentials]
api_id = <your api_id>
api_hash = <your api_hash>
phone = <your phone number, international format, e.g. +<country code><number>>

Replace each <...> value, without the angle brackets. Keep these values private and do not commit them.

4. Where files go. The saved credentials and the Telegram session live in the application home, ~/.telegram_tracker (on Windows %USERPROFILE%\.telegram_tracker), so after logging in you can run commands from any folder. config/config.ini is read from the working directory when nothing is saved. Output goes to ./output/data in the working directory unless you pass --output.

5. Optional: install the tgtracker command. The scripts above work without installing the project. To also get a tgtracker command, install the project into the activated environment from the repository folder:

python -m pip install .

tgtracker (and python -m telegram_tracker) take the same arguments as python main.py and use the same application home, saved credentials, and session. ./config/config.ini and ./output/data are relative to the folder you run them from. The installed command is a copy of the code: after updating the repository, run python -m pip install . again. python -m pip show telegram-tracker shows the installed version.

Logging in and saved credentials

auth login

auth login logs in to Telegram without collecting anything. It uses the session described under Which session is used and the app credentials described under Where the app credentials come from, and asks for any that are missing. Before asking, it says where typed values will be saved. If the session is already logged in, Telegram asks for no code or password; the app credentials are still needed to connect.

After a successful login:

• When all three values were typed and nothing was saved before, they are saved to credentials.json in the application home. --no-save skips saving.
• Values from the environment or an INI file are saved only if you answer y (default no). Without a terminal they are not saved; --save saves them without asking.
• Different values already saved are replaced only if you agree, or with --save.
• A failed or interrupted login saves nothing. If the login succeeds but saving fails, the command exits 1 and says so; the session stays logged in.

While a complete set is saved, auth login does not ask for app credentials. A login code or two-step verification password may still be needed if the selected session is not authorized. To change the saved values, run auth clear-credentials and then auth login, or run auth login with the new values in the environment or in --config and agree to replace the saved ones. The complete new set is saved.

Where the app credentials come from

The first source that provides a value wins:

Order	Source	Notes
1	TELEGRAM_TRACKER_API_ID, TELEGRAM_TRACKER_API_HASH, TELEGRAM_TRACKER_PHONE	Each variable overrides one value.
2	--config PATH	An INI file with the [Telegram API credentials] section. A missing file is an error.
3	credentials.json of the profile (the application home for the default profile)	Saved by auth login.
4	./config/config.ini in the working directory	The earlier setup; default profile only.
5	Prompts	auth login only.

Only one file source (2, 3, or 4) is used, as a whole: the first one present. A value missing from it is not taken from another file. Template values (api_id = api_id, <your api_id>, or empty) count as missing, so the placeholder config/config.ini in the repository never hides saved credentials. An invalid value (for example an api_id that is not a positive number) stops the command with exit 2; it never falls back to another source. There are no command-line options for the API ID, hash, or phone, and none for the login code or password.

Saved credentials therefore hide a ./config/config.ini that holds real values. For the default profile, auth status notes it, without changing which source is used:

Credentials in use: api_id: credentials.json; api_hash: credentials.json; phone: credentials.json
Note: ./config/config.ini also sets api_id, api_hash, phone but is not used, because credentials.json comes first. Pass --config ./config/config.ini to use it instead.

The note names fields, never values. It also flags invalid ones ((invalid: api_id)), or says why the file cannot be read (./config/config.ini cannot be read (permission denied); not used, because credentials.json comes first). A file with only template values gets no note. Other commands don't open that file while saved credentials exist.

collect uses the same sources but never asks for app credentials: with values missing it exits 2 and names the ways to provide them.

Which session is used

For the default profile, the first match wins (a named profile always uses its own session; see Account profiles):

1. --session PATH (collect and auth).
2. TELEGRAM_TRACKER_SESSION.
3. session_file.session in the application home, if it exists.
4. ./session_file.session in the working directory, if it exists. Earlier versions created the session there. The command prints a one-line notice when it uses it.
5. Otherwise session_file.session in the application home, created by the first login.

Nothing is moved or copied automatically. Once a session exists in the application home, it is used even when an older ./session_file.session exists. A path without the exact lowercase .session suffix gets it, matching Telethon. Symlinks and directory junctions in the path are resolved so those aliases share a lock; a resolved filename without that exact suffix is refused before connecting. Hard links are not identified as aliases. The session file must not be inside the --output folder. The --output folder must neither be inside the application home nor contain it.

Checking, logging out, and removing saved credentials

• auth status shows the application home, the selected session, the saved credentials file, and where each value comes from: source names only, never values. It works offline and creates nothing, so a present session file does not prove that it is still logged in. auth status --check connects to Telegram to find out (exit 0 logged in, 1 not); it never asks for a code and never creates a session. With --check, the Session: line only says present, and the last line, Login: logged in or Login: not logged in, gives the result. Telethon may update the session file while checking.
• auth logout asks for confirmation (--yes skips it), then logs the session out on Telegram. That revokes it and deletes the session file. Saved credentials stay. A session that is not logged in, or no session at all, is left as it is.
• auth clear-credentials asks for confirmation (--yes skips it), then deletes credentials.json only. It does not log the session out or touch environment variables, INI files, or collected data, and it removes a damaged file without reading it.

Session, file, and network errors

collect, auth login, auth status --check, and auth logout tell problems on this computer apart from login and network failures. Each of the following exits 1. None of them is fixed by logging in again.

Message	Cause
The session is in use by another process (pid ..., started ...)	Another command of this tool holds the session. Wait for it to end. If none is running, an interrupted one left the .lock file named in the message; delete it.
cannot lock the session (<session>.lock: its folder does not exist)	--session or TELEGRAM_TRACKER_SESSION points into a folder that does not exist.
cannot lock the session (<session>.lock: permission denied)	Your account can't create files in the session's folder.
The session file <path> is locked by another program (SQLite: database is locked)	A program outside this tool has the session file open, such as another Telethon script or a database viewer. Close it.
The session file <path> cannot be written (SQLite: attempt to write a readonly database)	The session file or its folder is read-only.
The session file <path> cannot be opened (SQLite: unable to open database file)	The folder is missing, or your account can't read and write the file.
The session file <path> is damaged or not a session file (SQLite: ...)	The file isn't a Telethon session. It is left as it is.
A local file cannot be used: <path> (<reason>)	Another file failed, for example permission denied.
Could not reach Telegram (<error>)	The network or Telegram's servers.

credentials.json, profile.json, and INI files that can't be read name the reason: not found, permission denied, not valid JSON, not UTF-8 text, or not a valid INI file. Their exit codes are unchanged.

How saved credentials are protected

credentials.json holds exactly schema_version, api_id, api_hash, and phone. It is plain text protected by file permissions, not encryption: the application home and its files are readable only by your account (mode 0700/0600 on Linux and macOS; on Windows an access list that grants only your account, with inheritance from the parent folder disabled). Administrators of the computer can still read them. A session created in the application home is protected by it too: on Windows it inherits the folder's access list; on Linux and macOS the 0700 folder keeps other accounts out, while the session file's own mode follows SQLite's default and your umask. The login code and two-step verification password are never saved. The session's authorization key is stored in the session file, separate from credentials.json.

Saving writes a protected new file next to the old one and replaces it in one step. Failures before replacement leave the earlier file intact; a failed protection check after replacement reports an error with the new file already in place. If the application home cannot be created or protected, collect stops with exit 1 before connecting; set TELEGRAM_TRACKER_HOME to another absolute folder. auth login also stops when it needs to write there. A default-profile login with an external session and --no-save can proceed without a profile record. A credentials.json that is damaged, of another version, or readable by other accounts stops every command that needs it with exit 2; remove it with auth clear-credentials and log in again.

TELEGRAM_TRACKER_HOME replaces the application home for all of this (it must be an absolute path).

Account profiles

A profile is a named set of saved credentials and a session, so one installation can collect with more than one Telegram account. Everything above describes the default profile, which is the application home itself: its credentials.json, its session (chosen as under Which session is used), and nothing else. Existing setups are the default profile without any change or migration.

A named profile lives in <application home>/profiles/<name>/ and holds its own credentials.json, session_file.session, and profile.json. Select it with --profile NAME (collect and auth login, status, logout, or clear-credentials) or the TELEGRAM_TRACKER_PROFILE variable; the option wins, and without either the default profile is used. The earlier collection form takes the profile from the variable only. Nothing remembers a selection between commands.

python main.py auth login --profile research     # creates the profile
python main.py collect --profile research --telegram-channel example_channel -o ./output/research
python main.py auth profiles                      # lists the profiles

• Names. A lowercase letter, then up to 31 lowercase letters, digits, _, or -. default selects the default profile. Windows device names (con, prn, aux, nul, com1-com9, lpt1-lpt9) are refused, and so is a profile folder that is a link or junction or whose name differs only in case. Invalid names stop with exit 2 before any file is touched.
• Creating. Only auth login --profile NAME creates a named profile record, after login and account verification. Errors before recording can leave folders or session files; a later credential-save failure can leave the profile record in place. Other single-profile commands stop with exit 2 for a profile that does not exist. auth profiles inventories all profiles without selecting one.
• Separate sources. A named profile uses only its own session and refuses --session and TELEGRAM_TRACKER_SESSION (exit 2; unset the variable or use the default profile). Its credentials come from the TELEGRAM_TRACKER_API_* variables, --config PATH, or its own credentials.json, never from ./config/config.ini, ./session_file.session, or another profile's files. A profile name is a local label: it does not prove that two profiles use different Telegram accounts.
• profile.json. It holds the profile's name, a random profile ID (created once and never changed), and the Telegram account the profile is bound to. Credentials are not in it. The default profile gets its profile.json at its next successful auth login; until then its profile ID is unknown. credentials.json keeps its format in every profile, so earlier versions still read the default profile's credentials.
• Account binding. After logging in, auth login asks Telegram which account the session belongs to and records it in profile.json the first time. A later login or collection through the same profile with a session of another account stops with exit 1 (auth logout --profile NAME, then log in with the bound account, or create a new profile); the binding is never replaced. auth login also stops when the session belongs to another account than the phone number in use. Only the profile's own session binds an account: a default-profile login or collection with --session or TELEGRAM_TRACKER_SESSION checks the account but leaves the binding as it is, and such a collection with another account is recorded without the profile ID. The verified account ID is kept as private metadata in profile.json; it is not added to diagnostics or collection provenance. Collected payloads can independently contain the same user ID.
• Logout and clear. auth logout --profile NAME revokes only that profile's session. auth clear-credentials --profile NAME removes only that profile's credentials.json; its profile.json and session stay, so a later login keeps the same profile ID and binding. For a named profile, clearing credentials takes its session lease and stops with exit 1 if that profile is collecting or logging in. Clearing the default profile uses the credentials-file lease only.
• auth profiles lists the default profile and every named profile with whether each has a profile record, saved credentials, and a session file. It reads no values, works offline, and creates nothing.
• Rename. auth rename-profile OLD NEW moves a named profile's folder to the new name and rewrites the name in its profile.json. The profile ID, account binding, saved credentials, and session stay. Collected outputs, collection-status.json, database rows, and media ledgers keep the profile name they recorded; histories recorded with the profile ID continue under the new name (collect --update, download-media), and refusals add the current name (belongs to profile old (now profile new)). A history recorded with a name but no profile ID does not follow the new name. NEW must not exist, not even as an empty folder; a free name used before, including the old name after renaming away from it, can be taken again. The default profile is never a source or target. No question is asked: the two names are the confirmation.
• Delete. auth delete-profile NAME removes the profile's credentials.json, its session file with the SQLite files beside it (-journal, -wal, -shm), and its profile.json, then the empty folder, from this computer only. Collected, derived, and media data, other profiles, environment variables, and INI files are not touched, and nothing is backed up. The session is not logged out on Telegram: run auth logout --profile NAME first to revoke it, or end it later under Devices in a Telegram app. It asks first (default no); --yes skips the question, and without a terminal and without --yes it stops with exit 2. A profile folder holding anything else (another file, a subfolder, a link, a file with other hard links, a leftover *.tmp) is refused before anything is removed. A profile created later with the same name gets a new profile ID, so histories recorded with the deleted profile's ID refuse it; collect again (a full or window run) to start its own histories.
• Before renaming or deleting, stop every other tgtracker command. While they change a profile, both commands hold its session, credentials-file, and profile-file leases, and they stop with exit 1 if one of those is held by another process or was left by a crashed run (the message names the .lock file; delete it only when no tgtracker command is running). --dry-run checks and reports what would happen without taking a lease or changing anything; it exits 1 when a lock file would stop the real run.
• Interrupted commands. Run the same command again. A rename interrupted after moving the folder leaves profile.json naming the old profile; every command refuses that profile and says to finish the rename. If the rename was killed, its lock files moved with the folder: the rerun first stops at them, and after you delete them (with no tgtracker command running) it finishes. A folder at NEW that is a copy of OLD (the same profile ID under both names) is refused with exit 2 and never renamed. A deletion that fails part way names what was removed and what remains; profile.json goes last, and running the command again finishes, also after profile.json is gone.
• Reused names. Only a profile ID tells two profiles with the same name apart. A history recorded with a profile name and no profile ID is accepted by --update and download-media from any profile that later has that name, even one logged in to another Telegram account. Named profiles always record their profile ID, so such records come only from edited or copied provenance; the default profile, whose collections with --session can lack the ID, is never renamed or deleted. This risk is accepted: deleted and renamed-away names are not reserved.
• Copying profiles, aliases, switching or renaming the default profile, and export/import are not available.

One collection per account. Before its first channel request, every collection asks Telegram which account it is logged in to (one extra get_me request, counted under account_checks in collection-status.json and outside the request controls) and holds a lease for that account in <application home>/account-leases/ until it ends. A second collection with the same account, through another profile or another session file, stops before its first channel request with exit 1 and The Telegram account is in use by another process (pid ..., started ..., profile ...). Collections with different accounts can run at the same time, each into its own --output folder; each collection keeps its own request controls. Every collection therefore needs a writable application home, even with credentials and session from elsewhere; otherwise it stops with exit 1 before connecting. The coordination covers one application home: the same account used from a second home or another program at the same time is not detected. A lease left by a crashed run is not removed automatically: make sure no collection is running (check the process ID), then delete the .lock file named in the message.

First login

The first login creates session_file.session in the application home. Later runs reuse that session and skip every prompt. auth login, the canonical collect, and the earlier form (python main.py --telegram-channel ...) log in the same way.

1. Run python main.py auth login, or any collection, for example python main.py collect --telegram-channel example_channel.
2. The tool prints > Not Authorized! Sending code request... and Telegram sends a login code to your Telegram app.
3. At Enter the code: , type the code.
4. If the account has two-step verification, the tool prints > Two-step verification is enabled. and asks Enter your two-step verification password: . The password is not shown while you type.
5. > Authorized! means the session is ready, and the collection continues.

Rules and failure cases:

• Attempts. You get at most 3 attempts for the code and 3 for the password. An empty entry counts as an attempt. After the third rejected attempt, the run exits 1.
• Expired code. The run stops at once (exit 1). Run the command again to request a new code.
• Rate limit. If Telegram rate-limits the code request or the sign-in, the run stops at once with > Login failed: Telegram rate-limited the login; wait N seconds before trying again. It does not retry automatically. Wait at least that long before you run it again.
• Other refusals. Exit 1 with the Telegram error name. Messages never include the phone number, code, or password.
• Interrupting. Ctrl+C at a prompt exits 130. In every case the connection to Telegram is closed once before the program exits.
• Password handling. The password is never a command-line option. The tool does not write it to a file or log, and keeps it only for the sign-in request. Never put it in config.ini, a script, or your shell history.
• A real terminal is required. The prompts read from the terminal. If input is unavailable, the run exits 1 with No login code was entered because input is unavailable; run the command from an interactive terminal. The password prompt refuses to fall back to visible input. Activate the environment (conda activate telegram-tracker) and run python main.py ... directly. conda run does not attach the terminal's input by default, so do not use it for the first login.

Protect the session file. session_file.session gives full access to your Telegram account. Do not share, copy, or commit it (it is listed in .gitignore). One command at a time uses a session: while collect or an auth command holds it, another one stops at once (exit 1) with The session is in use by another process, even from another output folder. If no other command is running, an interrupted one left session_file.session.lock behind; delete that file and run the command again.

Commands

main.py is the entry point, with one subcommand per step:

python main.py collect --telegram-channel NAME [options]
python main.py collect --batch-file FILE [options]
python main.py build-dataset [--data-path DIR] [--source auto|files|sqlite]
python main.py build-network [--data-path DIR] [--source auto|files|sqlite]
python main.py build-urls [--data-path DIR] [--source auto|files|sqlite]
python main.py download-media [--output DIR] [--dry-run | --check] [options]
python main.py auth login | status | logout | clear-credentials | profiles
python main.py auth rename-profile OLD NEW [--dry-run]
python main.py auth delete-profile NAME [--dry-run] [--yes]

Run python main.py --help or python main.py <command> --help for the options. Help never needs credentials, a session, or the service libraries. python main.py without arguments prints the overview and exits 2.

With the project installed (step 5 above), tgtracker and python -m telegram_tracker can replace python main.py in every example in this README, including the earlier collection form: arguments, outputs, and exit codes are the same. Only the program name shown in help and error messages differs.

Earlier commands

The earlier commands still work and reach the same code. They are supported in 1.0.0, with no removal date.

Earlier command	Canonical command	Notes
python main.py --telegram-channel NAME	python main.py collect --telegram-channel NAME
python main.py --batch-file FILE	python main.py collect --batch-file FILE	Collects up to 3 channels at once, like collect
... --limit-download-to-channel-metadata	python main.py collect ... --limit-download-to-channel-metadata
... --output DIR / -o DIR	python main.py collect ... --output DIR	Default ./output/data, relative to the working directory
... --min-id N	python main.py collect ... --min-id N	Same meaning: only messages with an id above N. collect asks before replacing saved histories; the earlier form only warns
(none)	python main.py collect ... --since/--until/--max-id/--max-messages	History window: collect only
(none)	python main.py collect ... --update	Merge new posts into saved histories: collect only
(none)	python main.py collect ... --concurrency/--request-interval/--max-flood-wait/--max-retries	Request controls: collect only. The earlier form uses the defaults
(none)	python main.py collect ... --storage files|sqlite|both	Storage mode: collect only. The earlier form writes files and refuses a sqlite or both folder
python build-datasets.py [--data-path DIR]	python main.py build-dataset [--data-path DIR]	-d also works
python channels-to-network.py [--data-path DIR]	python main.py build-network [--data-path DIR]	-d also works

The earlier collection form accepts only --telegram-channel, --batch-file, --limit-download-to-channel-metadata, --output, and --min-id. Its help (python main.py --telegram-channel x --help) lists --min-id under "history window".

Exit codes

The same for every form:

Code	Meaning
0	Every requested target was collected, the build succeeded, or the auth action completed (including "nothing to do"). Empty channels and empty inputs count as success.
1	A target failed, a media file failed or the media downloads stopped, the login failed, the run stopped early (flood wait or rejected session), the output folder, its database, the session, or the Telegram account is in use, the session file or another local file cannot be used, Telegram cannot be reached, a sqlite or both folder is blocked or its database cannot be used, the session belongs to another account than the profile or the phone number, writing an output or the credentials failed, the application home cannot be created, an auth confirmation or the replacement of saved histories by a filtered run was declined, auth status --check found the session logged out, a profile rename or deletion was refused (a held lease, an existing NEW, unexpected files in the profile folder) or failed part way, a dry run of one found a lock file, the input data of a build is missing or malformed, or a library is missing.
2	Invalid arguments, including an unreadable batch file or an unsupported line in it, a --storage mode the folder cannot take (or the earlier collection form on a sqlite or both folder), an invalid or unknown profile (and the default profile, or the same name twice, given to auth rename-profile or auth delete-profile), a folder at NEW that is a copy of OLD, auth delete-profile without a terminal and without --yes, --session or TELEGRAM_TRACKER_SESSION with a named profile, a filtered collect run that would replace saved histories without a terminal to confirm it and without --yes, and missing or invalid app credentials (before any connection).
130	Interrupted with Ctrl+C.

Collecting channels

# one channel
python main.py collect --telegram-channel example_channel

# every channel listed in a text file, into a chosen folder
python main.py collect --batch-file ./channels.txt --output ./output/project-a

# profile metadata only, without posts
python main.py collect --batch-file ./channels.txt --limit-download-to-channel-metadata

--output (or -o) chooses the folder; it is created if needed. A channel is given by its username, @username, a public link (https://t.me/username, t.me/s/username), or a numeric id. Links and @ are reduced to the username, which also names the channel's folder: @Example_Channel and https://t.me/Example_Channel are both saved under Example_Channel/.

Each requested channel is collected separately. A channel that does not exist, is private, or cannot be accessed fails only that channel: it is listed in _exceptions-channels-not-found.txt, the other channels are still collected, and the exit code is 1. At the end, the run prints a summary of the targets and requests, and names the failed channels on stderr.

Metadata only saves <channel>.json and the aggregate files, but no new <channel>_messages.json. A messages file from an earlier run stays in place; the summary and collection-status.json (stale_history) name those channels. In a sqlite or both folder, the messages stored in the database stay in place the same way.

Collecting part of a history

collect accepts five history window options. They combine as AND:

Option	Selects
--min-id N	messages with an id above N (exclusive)
--max-id N	messages with an id below N (exclusive)
--since DATE	messages dated at or after DATE (inclusive)
--until DATE	messages dated before DATE (exclusive)
--max-messages N	at most N messages per channel, newest first

Dates are ISO 8601: 2026-10-01, 2026-10-01T12:00, 2026-10-01T12:00:00Z, or 2026-10-01T14:00+02:00. A value without an offset is UTC, and a date alone means midnight UTC. The earlier collection form accepts only --min-id.

# September 2026 only, into a fresh folder
python main.py collect --telegram-channel example_channel --since 2026-09-01 --until 2026-10-01 -o ./output/sep-2026

# the newest 500 messages of each channel in a batch
python main.py collect --batch-file ./channels.txt --max-messages 500 -o ./output/sample

# messages after id 12345
python main.py collect --telegram-channel example_channel --min-id 12345 -o ./output/new-posts

A filtered run replaces each channel's saved history with the selected messages only. It does not merge them into the saved history. To keep earlier histories, write filtered runs to a fresh --output, or use --update to add new posts. The run header prints the active window. Window options cannot be combined with --limit-download-to-channel-metadata in collect.

collect asks before it replaces a saved history. Before reading credentials or connecting, it looks for the requested channels that already have a saved history in the output folder:

Mode	Saved history looked for
files	<channel>/<channel>_messages.json
sqlite	A history in the database for a channel that an earlier run collected under the same name, or whose stored username it is (case-insensitive). The database is opened read-only.
both	Either of the above

If it finds any, it prints a warning on stderr with their number and the first 10 names, then asks Continue? [y/N]:

> Warning: this filtered run replaces the saved messages of 2 channel(s) in ./output/data: example_channel, other_channel. Use a fresh --output to keep them.
Continue? [y/N]

• y or yes goes on. Any other answer, or end of input, stops with exit 1 before connecting; nothing is written.
• --yes replaces them without asking, for scripts.
• Without a terminal to answer (input piped, or redirected from a file or from NUL), the run stops with exit 2 before connecting and asks for --yes or a fresh --output. It never replaces them silently.

Runs without a history window (full, --update, metadata only) never ask.

If another collection creates a messages file while this run waits for login, a second check under the output lock catches it. Without --yes, that target fails before target requests and its file is kept; rerun to be asked first. Other targets continue.

The earlier collection form (python main.py --telegram-channel NAME --min-id N) does not accept --yes, so it only prints the warning and goes on, as it did before; it writes files only. Use collect to be asked.

Before connecting, a sqlite or both folder can only be searched by name: the database keys histories by the channel's numeric id, which is known only after connecting. A channel given under a name that no earlier run used and that is not its stored username (for example a username that changed since the last run) is found after it resolves. Without --yes, that channel fails before any of its messages are requested and its stored history is kept; the reason says the history was found under another name. The run goes on with the other channels and exits 1. The failed target's record keeps the channel's id, so running the same command again asks first. In files folders a new name writes a new folder and replaces nothing.

Known limits:

• Pages are kept whole, so the chats and users of the page that crosses a bound can include entries referenced only by excluded messages.
• --since stops paging at the first page older than the date. This assumes message dates follow id order, which imported histories can break.
• count in the saved JSON (collected_actions in the dataset) is Telegram's total for the channel, not the number of messages saved.

Updating saved histories (--update)

python main.py collect --batch-file ./channels.txt --update -o ./output/data

Use the same --output as the earlier collection. For each channel, --update reads the saved history, collects only posts with a higher id, and merges them in. The saved history depends on the folder's storage mode:

Mode	Saved history read and updated
files	<channel>/<channel>_messages.json
sqlite	The channel's history in the database, found by the channel's numeric id after it is resolved
both	Both copies. They must hold the same messages first (see --update in both)

In every mode:

• Messages are merged by id. A fetched copy replaces a saved one and saved posts are kept. The messages file stays ordered newest first.
• In files, chats and users are merged by id, fetched copies first. In the database, membership uses (peer_kind, peer_id); shared peer rows retain a full object over a fetched min object, and can reflect a later collection of another channel. count and the other envelope fields come from the latest history response when new messages are merged.
• With no new posts, the saved history is left as it was: the file byte for byte, the database history with its earlier run and time.
• A failed request leaves the saved history as it was.
• A saved history that cannot be read or parsed fails that channel and is left untouched.
• A channel with no saved history is collected in full; the summary lists those channels.

--update cannot be combined with the history window options or with --limit-download-to-channel-metadata. The earlier collection form does not accept it.

	History window (--min-id, --since, ...)	--update
Saved history (file or database)	Replaced by the selection, once confirmed (or with --yes)	Kept; new posts merged in
Where to write	A fresh --output	The folder of the earlier collection
Lower bound	Given by you	The highest saved message id of each channel

Limits:

• Only new posts are fetched, so posts deleted or edited on Telegram since the earlier run are not reflected.
• Duplicates are removed inside the merged file only. build-dataset appends to msgs_dataset.csv, so building again after an update adds the earlier posts a second time. Build into a fresh dataset if you need each post once (see Building the dataset and the network).

Batches, concurrency, and rate limits

A batch file holds one target per line, in any of the forms above. Blank lines are skipped. Repeated targets are dropped case-insensitively, keeping the first. Any other line (a path, an invite or post link, text with spaces) stops the command with exit 2 and the line number, before any credentials are read. Two targets that turn out to be the same channel (for example an old and a new username) are collected once; the later one is reported as a duplicate.

--batch-file collects up to 3 channels at once by default, in both command forms. The pages of one channel are still fetched in order. A single --telegram-channel always runs at 1. After login, channel resolution, profile metadata, history pages, and discovered-chat metadata lookups go through one shared request policy:

Control	Default	Range	Effect
--concurrency N	3 for --batch-file	1–10	Channels collected at once (with --include-media, also files downloaded at once); also the most requests in flight
--request-interval SECONDS	0.25	0.1–60	Minimum gap between request attempts admitted by the shared policy
--max-flood-wait SECONDS	60	0–3600	Longest Telegram flood wait to sit out
--max-retries N	3	0–10	Additional attempts per request after a flood wait or a transient error

• Flood waits. When Telegram asks the client to wait during collection, the policy pauses new request attempts across all channels until the wait ends, and then retries the request. Requests already in flight can finish. A wait longer than --max-flood-wait, or another flood wait after --max-retries retries, stops the whole run: completed channels are kept, the rest are reported as not collected, collection-status.json says interrupted, and the exit code is 1. Wait before collecting again.
• Transient errors. Server errors, timeouts (60 seconds per attempt), and dropped connections are retried after 1, 2, 4 ... seconds (at most 30) within --max-retries. If they keep failing, only that channel fails.
• Access errors. A missing or private channel fails only that channel. A session that Telegram rejects stops the run; log in again.
• Serial fallback. --concurrency 1 collects one channel at a time with the same policy. Use it if you prefer the earlier one-by-one behavior, or when an account hits flood waits often.
• Media downloads. With --include-media, files are downloaded after every channel's messages are saved, through the same policy, one request per 512 KiB. The run downloads as many files at once as it collects channels at once (--concurrency, capped at the number of channels), so a single --telegram-channel downloads one file at a time; download-media --concurrency N downloads N at once (see docs/media.md).

Concurrency is not a Telegram quota. More workers do not raise Telegram's limits for an account: the request gap is shared across all channels, and a flood wait pauses new policy attempts for all of them. The defaults are project limits, not a rate that Telegram guarantees to be safe. On simulated 1-second requests, 3 workers took about 40% of the serial time, and 10 workers were limited by the 0.25-second gap (see docs/compatibility.md). Real timings depend on Telegram.

Login and the subsequent account check run before this policy is created; neither is paced or retried by it. Login follows the failure rules in First login. Telethon can also retry a server error once inside a policy attempt, so the interval and retry settings do not count every underlying Telegram request.

The summary prints the counts (given, after removing duplicates, succeeded, failed, duplicates, cancelled) and, per kind of request (resolve, channel, history, enrichment), the policy attempts sent, retries, flood waits, and flood-wait seconds. collection-status.json records the same numbers.

The collection summary lists the messages saved per channel. It shows the size of each saved history after a full or window run, and also the messages added after --update:

> Messages saved: 50 in 1 channel(s)
>   example_channel: 50

> Messages saved: 3 new; the saved histories of 1 channel(s) now hold 53
>   example_channel: 3 new, 53 in total

Failed channels and metadata-only runs are not listed.

Output files

output/data/                          (or the folder given with --output)
├── <channel>/
│   ├── <channel>.json                channel profile metadata
│   ├── <channel>_messages.json       posts (not written by metadata-only runs)
│   └── <channel>_provenance.json     which profile collected those posts
├── chats.txt                         ids of chats found in the collected data
├── collected_chats.csv               metadata of the chats found in the data (e.g., forwards)
├── counter.csv                       how often each chat was found in this run
├── collection-status.json            state and per-target outcome of the last collection
├── collection-storage.json           storage mode, for sqlite and both folders only
├── telegram-tracker.sqlite3          the database (--storage sqlite or both)
├── downloaded-media/                 media files and their ledger (--include-media, download-media)
├── collection-identity.json          files folders with downloaded media: ties the media ledger to this data
├── _exceptions-channels-not-found.txt   requested targets that were missing or inaccessible (when any)
├── _exceptions-users-not-found.txt      discovered chats whose metadata could not be retrieved (when any)
├── msgs_dataset.csv                  written by build-dataset
├── collected_chats.xlsx              written by build-dataset
├── dataset-build.json                written by build-dataset: what it read
├── Graph.gexf                        written by build-network
├── network.png                       written by build-network
└── network-build.json                written by build-network: what it read

What happens when you collect into a folder that already holds data:

File	On a rerun into the same folder
<channel>.json	Replaced after the channel is collected
<channel>_messages.json	Replaced (full or window), merged (--update), or kept (metadata only)
<channel>_provenance.json	Written with the messages file; see Profiles and shared folders
chats.txt	Keeps the earlier ids; new ids are added in the order found
collected_chats.csv	Keeps the earlier rows as they were; this run's rows are appended with this run's counts. No _x/_y columns are created. A run that adds no rows leaves it untouched
counter.csv	Replaced; describes this run only
collection-status.json	Replaced; describes this run only
_exceptions-*.txt	Appended to

Channel payloads written by this version hold collected Telegram data unfiltered, including the names, usernames, and phone numbers (when Telegram returns them) of users who appear in it, for example as commenters. The one exception is the collecting account itself: when it appears, its User object is kept with phone set to null.

A channel's metadata, message history, and history provenance files are written only after the whole channel was collected, so a request failure keeps its earlier files. JSON and CSV files and chats.txt are written to a temporary file and then moved into place, so a reader sees either the old or the new version of each file. There is no transaction across files: a crash can leave files from different runs (see below).

Profiles and shared folders

Several profiles can collect into the same --output folder, one run at a time (the folder lock serializes them). Each saved history records which profile collected it in <channel>/<channel>_provenance.json: the profile name and ID (profile_name, profile_id, null when unknown), how the history was last written (replaced, created, or merged), the run's start time, its message count and newest message ID, and whether it took over another profile's history. It never holds an account ID, phone number, or credential.

• --update merges only into the same profile's history. If the record has a profile ID, it must match; if it has only a name, the name must match. A history without a record (collected by an earlier version) can be updated only by the default profile, and it stays without a record. Otherwise that channel fails before any request for it, with guidance, and the other channels continue. A merge keeps the recorded profile. A record whose message count or newest ID no longer matches the messages file (a run stopped between the two files) refuses every --update of that channel.
• A full or window run replaces any history and records its own profile. When the previous profile could not have updated it, the record and the channel's entry in collection-status.json get takeover: true with the previous profile's name and ID, and the run summary lists the channel.
• Two profile names for the same Telegram account are still different profiles here, and a profile name alone cannot tell apart accounts in different application homes or on different computers. A full run resolves a refused update.
• Access hashes and file references in the collected data are valid only for the account that collected them; the provenance record and collection-status.json name the profile, not the account. Channel metadata files (<channel>.json) reflect the latest run that wrote each file. Aggregate files follow the append/replace rules above; older rows can reflect other profiles and are not individually profile-tagged.

Storage modes (--storage)

collect --storage MODE chooses where the collected data goes:

Mode	Writes	Builders
files	The files above. The default for a new folder	Run
sqlite	Only the database <output>/telegram-tracker.sqlite3, plus collection-status.json and collection-storage.json	Run on the database
both	The same data in the files (exactly as files writes them) and in the database	Run on the files; --source sqlite reads the database

The database holds the same fields as the files, plus a record of every run; see docs/sqlite-schema.md for its tables and example queries. Its path is always telegram-tracker.sqlite3 inside the --output folder.

# a new folder that keeps both the files and the database
python main.py collect --batch-file ./channels.txt --output ./output/project-b --storage both

# later runs into that folder use the recorded mode without the option
python main.py collect --batch-file ./channels.txt --output ./output/project-b --update

A folder keeps its mode. A sqlite or both folder records its mode in collection-storage.json (with the database's dataset ID) when it is created. A run without --storage uses the recorded mode; a folder without that file is a files folder. Choosing another mode for a folder exits 2 before connecting, and so does sqlite or both for a folder that already holds other files: existing outputs are not converted, so start a new or empty --output. The earlier collection form (python main.py --telegram-channel ...) writes files only and refuses a sqlite or both folder with exit 2; use collect there.

Checks before connecting. For a sqlite or both folder, the database must exist, be a Telegram Tracker database of the current schema version, belong to the folder (the dataset ID in collection-storage.json and the folder it was created for), and have no unpublished files (see below). Otherwise the run exits 1 before connecting and leaves the folder untouched. A folder that was moved or copied is refused for this reason; collect into a fresh --output. The selected session file is never opened as the database, and any other SQLite file at that path is refused untouched.

Locks. Besides .collection.lock, a database run holds telegram-tracker.sqlite3.lock, taken after the folder lock (after login) and removed when the run ends. An existing one is reported before connecting with its own message (The database ... is in use by another collection or upgrade, or a build reading it (pid ..., started ...)); a builder reading the database holds it too. A lock left by a killed run is not removed automatically: make sure no collection is running, then delete it. The checks above run again once both locks are held, before anything is written. A run that finds an earlier run still recorded as running in the database (its process ended without finishing) records that run as interrupted, with a stop reason starting with stale:, and says so on stderr.

How both keeps the two copies detectable. For each channel, the files are first written to temporary files; then the database stores the channel and marks its files pending; then the temporary files replace the channel's files; then the mark becomes written. The aggregate files follow the same order after the last channel. If channel staging or its database transaction fails, that channel's previous payloads stay in both copies; run status can still record the failure. A failure while replacing the files stops the run (exit 1), records that channel as failed in the database, and leaves the pending mark. On Windows, a replace that another program briefly refuses (an antivirus scanner, a file indexer) is retried up to 5 attempts, with 0.75 s of total configured retry delays per file, in both mode only; filesystem calls take additional time.

Recovering a blocked folder. A pending mark means the files and the database may disagree, so later collection into that folder is refused. Using the recorded mode exits 1 with <folder> is blocked: ...; choosing a different mode exits 2 under the existing mode rule. Deleting the lock files does not clear it. Keep the folder for inspection and collect into a fresh --output; there is no repair command. A process that ends after replacing the files but before recording it leaves the mark too, although the files then agree with the database: such a folder is blocked as well. A crash can also leave temporary files (<name>.<pid>.tmp.<ext>, .telegram-tracker.sqlite3.<id>.creating) next to the files; they are never read and can be deleted. A Ctrl+C or a flood-wait stop does not block the folder: the channels saved before it are in both copies. A crash during a new folder's setup, after the database was created and before collection-storage.json was written, leaves a database without a manifest: the folder then counts as a files folder, a run without --storage writes files beside that database (whose first run stays running), and --storage sqlite or both is refused because the folder is not empty. Delete the folder, or collect into a fresh --output.

--update in both. Before a channel's first history request, its saved history must be the same in both copies: the same newest message ID and the same message IDs. A channel whose copies differ, or that has a saved history in only one of them, fails with guidance, and the other channels continue; the larger copy is never taken as a repair. A channel without a history in either copy is collected in full.

After a crash or a killed run. The database stores each channel in its own transaction, so committed channels stay in it. Depending on when the process ended, aggregate rows or the end state may be missing, collection-status.json may still say running, and acquired lock files may remain. To continue:

1. Make sure no collection is running (check the process ID in the lock messages).
2. Remove the stale .collection.lock and telegram-tracker.sqlite3.lock if present. Also remove any stale session lease (<selected session>.lock) and account lease in <application home>/account-leases/ identified by the refusal messages; those leases also block collection into a fresh folder. Delete only locks whose owning processes have ended, and keep the session and database files.
3. Collect into the same folder again. If the crashed run is still recorded as running in the database, the new run records it as interrupted (stop reason stale: ...). In a both folder, a run that stops with is blocked found a pending mark: keep that folder and collect into a fresh --output (see above).

Reading the database

telegram-tracker.sqlite3 is an SQLite database: one binary file that SQLite libraries and tools open and query directly. It is not an SQL dump, the text file of CREATE TABLE and INSERT statements that some tools export; collect neither reads nor writes dumps. docs/sqlite-schema.md describes every table, column, and view, and has more example queries.

Open the database read-only, and preferably when no collection is writing to it. A connection inside an open read transaction makes a collection wait, and a write that waits more than 5 seconds stops the run with exit 1 (the database is in use by another process). Finish the query and close the connection before collecting.

From Python, with the standard library only (the paths below are relative to the folder you run from):

import sqlite3

con = sqlite3.connect('file:output/project-b/telegram-tracker.sqlite3?mode=ro', uri=True)
query = '''
SELECT p.username, count(*) AS posts, min(m.date) AS first_post, max(m.date) AS last_post
FROM messages m
JOIN peers p ON p.peer_kind = 'channel' AND p.peer_id = m.channel_id
GROUP BY m.channel_id
ORDER BY posts DESC
'''
for row in con.execute(query):
    print(row)
con.close()

With pandas (installed with the requirements), pandas.read_sql_query(query, con) returns the same rows as a DataFrame. Columns not listed in the schema are still in each row's raw JSON; read them with json_extract(raw, '$.field').

To export an SQLite SQL dump as a text archive (other database systems may require adapting its SQLite-specific schema and syntax):

import sqlite3

con = sqlite3.connect('file:output/project-b/telegram-tracker.sqlite3?mode=ro', uri=True)
with open('project-b.sql', 'w', encoding='utf-8') as dump:
    for statement in con.iterdump():
        dump.write(statement + '\n')
con.close()

A dump does not carry the database's identity (its application ID and schema version), so a database rebuilt from a dump is refused by collect as not being a Telegram Tracker database. Use a backup (below) to keep a copy that collect accepts.

Database backups and schema versions

Schema version. A database records its schema version, 1 in this release, and an application ID that marks it as a Telegram Tracker database. collect uses only a database of the current version: an older or newer one, another SQLite database, an empty file, or a damaged one is refused before connecting (exit 1) and left untouched. collect never upgrades a database.

No upgrade command yet. Version 1 is the first schema version, so there is nothing to upgrade, and this release has no migrate command. The package contains the upgrade procedure for later versions (a checked backup first, then one transaction per version step, under the database lock), but no upgrade step is defined.

Application version. The database and the media ledger record the Telegram Tracker version that created them and that ran each collection or media run (app_version), read from the installed package. python main.py records the version of an installed copy, or unknown when the project is not installed. The version is only a record: a database or ledger made by an earlier version is used as it is, without an upgrade.

Backups. Back up while no collection is running, through SQLite's backup API, which copies a consistent state:

import sqlite3

source = sqlite3.connect('file:output/project-b/telegram-tracker.sqlite3?mode=ro', uri=True)
backup = sqlite3.connect('project-b-backup.sqlite3')
source.backup(backup)
backup.close()
source.close()

The sqlite3 command-line shell, if you have it, does the same with sqlite3 telegram-tracker.sqlite3 ".backup copy.sqlite3". Copying the file itself is safe only while no collection is running and no telegram-tracker.sqlite3-journal file is beside it.

A backup belongs to the folder its database was created for. Copied back over telegram-tracker.sqlite3 in that folder (while no collection is running), it is accepted again. In any other folder, collect refuses it as a moved or copied folder, but it stays readable. The restored database must also match the folder manifest's dataset ID and have no pending publication marks. In a both folder, restoring only the database can make the two copies differ, and --update then fails the channels whose histories differ; collect those with a full run, or start a fresh --output. The database holds personal data from the collected content (see docs/sqlite-schema.md); store backups as carefully as the database.

Collection status, locking, and recovery

Each collection writes collection-status.json in the output folder. It records the state, the profile (profile_name, profile_id), the parameters (full, window with its bounds, update, or metadata only), the request controls and statistics, the account check (account_checks), and each target's outcome (succeeded, failed with a reason, duplicate, cancelled; a target that a window run failed because its history was found under another name also keeps its channel_id), with the saved history's size (messages), the derived min_id, history mode, and messages added (new_messages) for --update, and any takeover. It holds no credentials, account IDs, or message text. Status files written by earlier versions have no profile fields: their profile is unknown.

State	Meaning	Builders
running	Collecting, or stopped without finishing (a crash)	Refuse
complete	Every target was collected	Run
partial	Finished; some targets failed	Run, with a note naming the failed targets
interrupted	Stopped early (Ctrl+C, a flood-wait or session stop, a storage error, or another fatal error); completed channels are saved	Refuse

build-dataset also notes when the last collection used a history window or --update. Folders without a status file (from earlier versions) are accepted when the builders read files; reading a database needs one.

One collection per folder. A run acquires .collection.lock in the output folder after login and removes it when it ends. An existing lock is checked before connecting and causes exit 1 with <folder> is in use by another collection (pid ..., started ...). If two runs pass that check before either acquires the lock, both can reach login; only one can hold the lock and write collection output at a time. The lock protects the folder; the session lease and the account lease (see Account profiles) keep a second collection off the same session or Telegram account, even into another folder.

Stale lock. A lock left by a crashed or killed run is not removed automatically. To recover:

1. Make sure no collection is running (check the process id named in the message).
2. Either keep the folder for inspection and collect into a fresh --output, or delete <folder>/.collection.lock to reuse the folder.

Interrupted or crashed run. If collection-status.json says interrupted, or still says running after the process ended, the aggregate files may be incomplete or from different runs, and the builders refuse the folder. To recover:

1. Keep that folder as it is, for inspection.
2. Run the collection again into a fresh --output.
3. Build the dataset and the network from the new folder.

The completed <channel>.json and <channel>_messages.json files in the old folder stay readable. If a run is cancelled while a finished channel is still waiting to be saved, that channel may be reported as not collected; collect it again.

Memory: each channel's history is held in memory until it is saved, and the run keeps the chat observations of every completed channel until the end. Very large batches or histories need correspondingly more memory.

Building the dataset and the network

Run the builders after a collection has finished, on the same folder:

python main.py build-dataset --data-path ./output/data
python main.py build-network --data-path ./output/data

Without --data-path (or -d), both use ./output/data. They read local data only, need no credentials or session, and can run without a network connection.

What they read (--source). Each builder reads one source, never a mix of the two:

Folder	--source auto (default)	--source files	--source sqlite
files (or an earlier version's folder)	the files	the files	refused (exit 1)
sqlite	the database	refused (exit 1)	the database
both	the files	the files	the database

A source that cannot be read is refused; the builders never fall back to the other copy. A sqlite folder builds the same way as a files folder:

python main.py collect --batch-file ./channels.txt --output ./output/project-s --storage sqlite
python main.py build-dataset --data-path ./output/project-s
python main.py build-network --data-path ./output/project-s

build-dataset reads every saved message history and the collected chats, and writes:

• msgs_dataset.csv: one row per post. Rows are appended to an existing file, so building twice, or again after --update, duplicates rows; the build says so on stderr when the file already exists. To get each post once, delete or move msgs_dataset.csv before building, or build from a fresh collection folder.
• collected_chats.xlsx: replaced on every build.
• dataset-build.json: replaced on every build; see below.

From the files, the histories are every <channel>_messages.json in sorted path order, and the collected chats are collected_chats.csv. From the database, the histories are the current stored ones, each under the collection target name it was last saved with, in the order the files would sort, and newest message first. The collected chats are rebuilt from the stored runs, oldest first, exactly as collected_chats.csv appends them: earlier rows as written, each run's rows with that run's counts, and rows repeated across runs kept. The record of each run (run tables) never adds dataset rows.

build-network reads the collected chats (username, counter, source) and writes Graph.gexf, a directed, unweighted graph from each source channel to the chats found in its data (e.g., forwards), and network.png, a preview with Louvain communities, plus network-build.json. All are replaced on every build. Open the GEXF file in tools such as Gephi.

Build records. dataset-build.json and network-build.json record what a build read: the source (files or sqlite), the folder's storage mode, the database's dataset ID and schema version (for the database), the last run (state, mode, times, profile; the run ID only from the database), and the output counts. dataset-build.json also lists each history read: its channel name and ID, message count, and provenance: the profile name and ID that last wrote it, how (replaced, created, or merged), and whether that was a takeover. From the files, provenance comes from <channel>_provenance.json (unknown, null, without one); from the database, from the stored history, with the ID of the run that last changed it. Collected-chat rows carry no profile in either source. Profile names and IDs are local labels, not verified accounts. The records hold no message text or credentials.

Same data, same outputs. A both folder gives the same msgs_dataset.csv, workbook and graph from either source, on a first build and on repeated builds. Two differences come from what each copy keeps:

• Renamed channels. A channel collected under an old and then a new username has a messages file under each name, and the files builder reads both (the old one is no longer updated). The database keeps one current history per channel, under the newest name. The collected chats and the network are the same; the dataset from the database has no rows, and the workbook no statistics, for the old name.
• Nested dates in the workbook. A date inside an object column of the collected chats (for example until_date in default_banned_rights) reads datetime.datetime(2038, 1, 19, ...) from the files and '2038-01-19 03:14:07+00:00' from the database, which stores JSON text. Top-level columns such as date read the same.

Checks before a build. Both builders refuse a folder whose collection-status.json says running or interrupted (exit 1, with the recovery steps above). Reading a database also needs:

• collection-status.json to exist: a database folder without one is refused, not taken as finished;
• the database to exist and be a Telegram Tracker database of the current schema version, the one named in collection-storage.json (dataset ID); the builders never create, repair or upgrade it;
• its last run to be the one collection-status.json describes, and finished (complete or partial);
• no pending mark (see Recovering a blocked folder); a both folder with one still builds from its files.

A folder moved or copied after collection is read, with a note. A database builder holds telegram-tracker.sqlite3.lock, rereads the manifest and checks the current sidecar under that lock, then reads through a read-only connection in one read transaction, so it sees one consistent state; a collection started meanwhile exits with The database ... is in use by another collection or upgrade, or a build reading it. The lock is removed before the outputs are written, and when a build fails. A lock left by a killed build is reported like a collection's: make sure no build or collection is running, then delete it.

Missing or malformed inputs stop the build with exit 1 before anything is written. With no messages and no existing msgs_dataset.csv, the dataset has a header only; an existing dataset keeps its earlier rows. With no edges, the graph is empty and the image blank.

Memory. The files builder holds one messages file at a time while reading and all dataset rows until it writes them; reading the database is the same, with messages fetched in batches of 500. The collected chats are held in memory whole, and rebuilding them from the database reads the rows kept so far again for each later run.

Listing shared URLs

build-urls extracts URL occurrences from collected message text, the hidden targets of hyperlinks, and stored link previews. It reads the same sources as the other builders, with the same --source choice and checks:

python main.py build-urls --data-path ./output/data

It writes three files to the folder and replaces them on every build:

• urls_dataset.csv: one row per occurrence, with the message, the URL as found and normalized, its parts, and its registered domain (bbc.co.uk, foo.blogspot.com; IP addresses and unknown suffixes go into buckets such as [ipv4]);
• urls.sqlite3: the same rows, plus counts per domain, per channel, and per channel and domain;
• urls-build.json: the build record.

Nothing is fetched: links are not opened, resolved or expanded, and the public suffix list is the copy bundled with tldextract. Two build-urls runs on one folder refuse each other (urls.sqlite3.lock). docs/urls-dataset.md gives the evidence and counting rules, every column, and example queries.

Downloading media

Media downloads are off by default: without --include-media or download-media, no file is requested.

# download the files of the collected messages after collecting them
python main.py collect --batch-file ./channels.txt --include-media

# download the files of an existing folder; first see what a run would request
python main.py download-media --output ./output/data --dry-run
python main.py download-media --output ./output/data --media-types image,video --max-media-total 5G

# check the stored files against the ledger and the saved messages
python main.py download-media --output ./output/data --check

• Where. Files go to <output>/downloaded-media/<channel_id>/{images,videos,audios,files}/, under generated names, one copy per channel and content, recorded in downloaded-media/media.sqlite3. The collected data is not changed. Files are saved, never opened.
• Which. collect --include-media downloads the files of the messages the run covered (with --update, the new messages). download-media downloads every file of the selected kinds in the folder's saved histories that is not stored yet, including files that failed or were skipped before, fetching the messages again for fresh file references. It needs the profile that collected each channel.
• Limits. --media-types image,video,audio,file selects kinds; --max-media-size SIZE skips larger files; --max-media-total SIZE caps the bytes received in the run. There is no size limit by default.
• Interruptions. Completed files are never deleted. Ctrl+C and failures remove the file in progress; a killed run's partial file is removed by the next run that writes the ledger. Failed and skipped files are recorded with a reason, and download-media tries them again.
• Sharing. Copying or sharing the output folder shares the downloaded files.

docs/media.md describes the options, the folder layout, speed and concurrency, --dry-run and --check, exit codes, and failure handling; docs/media-ledger.md is the ledger's contract.

Known limits

The behavior in this README is validated by the offline test suite, with synthetic data and a fake Telegram client, plus an install check of the built package. What that validation does not cover:

• Platforms. Tested on Windows 11 on local NTFS drives, and on GitHub-hosted Windows Server 2025, Ubuntu 24.04 and macOS 15 runners (permissions, locks, path spellings, interruption, and the installed commands). Other systems, network file systems, and runs as root are untested. docs/compatibility.md lists the supported, unknown, and refused combinations of system, Python, installation, and SQLite version, with the evidence for each.
• Ctrl+C. On Windows, interruption is tested with a real SIGINT that the collecting process raises itself; Ctrl+C sent to a Windows console is untested. Ctrl+C typed on a terminal (a pseudo-terminal in the tests) and SIGINT sent by another process are tested on Linux and macOS.
• Database modes and profiles against Telegram. --storage sqlite and both, and named profiles, are validated offline; this validation includes no collection from Telegram in those modes or profiles.
• Request rates. The concurrency and request-interval defaults are project limits. The rates that Telegram accepts for an account were not measured.
• Builders on a database. Read from the database, a renamed channel's old-name history is not built, and a date nested in an object column of the collected chats reads as stored text (see Building the dataset and the network). Rebuilding the collected chats from a database rereads the rows kept so far for each later run, so builds of folders with many runs take longer.
• Dataset appends. build-dataset appends to msgs_dataset.csv, so building twice, or again after --update, duplicates rows (see Building the dataset and the network).
• URLs. build-urls finds only URLs that are in the collected messages; it does not read URL buttons, and it checks no link against the web. Its public suffix list is as current as the pinned tldextract release.
• Media against Telegram. Media downloads are validated offline, with a fake client serving real Telethon objects; no file was downloaded from Telegram by the tests. Default application request pacing corresponds to about 2 MiB/s of requested file chunks within a run; this is not a network bandwidth cap, and real download rates were not measured (see docs/media.md).

Running the tests

The test suite runs offline with synthetic fixtures. It never connects to Telegram, reads config/config.ini, opens a session file, or touches your application home: every test process uses a temporary TELEGRAM_TRACKER_HOME. From the repository folder, in the project environment:

python -m unittest discover -s tests
python -m pip check

The tests import the telegram_tracker package from src/ in this checkout, so the project does not need to be installed to run them, and an installed copy is not used. Tests that need Linux or macOS (file modes, symlinked lock files, Ctrl+C on a terminal) are reported as skipped on Windows, with the reason. python -m pytest runs the same tests when pytest is installed. docs/compatibility.md describes the tested environment, the CLI contract, and the offline test harness.

Faster local runs

Two options shorten a local run. Neither changes what the command above checks, and CI runs only that command.

Parallel workers. With pytest and pytest-xdist installed in the environment (neither is in requirements.txt):

python -m pytest tests -p no:cacheprovider -n 4 --dist=loadfile

--dist=loadfile keeps each test module on one worker, and every worker process gets its own temporary application home from the offline guard. Serial and parallel runs give the same result for every test (docs/compatibility.md). On a hard disk the workers queue for the same disk, so a parallel run can be slower than a serial one unless disk syncs are also skipped.

Skipping disk syncs (local runs only). On a slow disk, most of the suite's time goes into waiting for SQLite commits and fsync to reach the disk. With TGTRACKER_TEST_NO_SYNC=1, each test process opens SQLite connections with PRAGMA synchronous = OFF, and os.fsync checks its descriptor without flushing:

TGTRACKER_TEST_NO_SYNC=1 python -m pytest tests -p no:cacheprovider -n 4 --dist=loadfile

In PowerShell, set $env:TGTRACKER_TEST_NO_SYNC = '1' first. Written data still reaches the operating system, and SQLite documents this setting as safe when a process crashes or is killed, so only durability across an operating-system crash or a power loss goes untested. Tests that patch fsync or sqlite3.connect themselves still get their own patches, and the processes that tests start keep real syncs. The default command and CI never set the variable. Any value other than 1, 0, or empty is refused: the offline guard fails to load, so every test module errors.

License

Licensed under the Apache License 2.0.