A personalized notification router that decides whether each incoming message should
interrupt the user now (notify), wait for later (digest), or be suppressed (mute) —
reading not just text, but the images and voice notes attached to it.
Built solo for the HackerRank Orchestrate (August 2026) 24-hour challenge, then hardened.
Most routers of this kind stop at keywords. This one adds four real perception layers, each behind a strict validation boundary, and then measures whether they actually helped.
| Layer | Model | Configuration |
|---|---|---|
| Speech (voice notes) | faster-whisper / CTranslate2, small |
CPU, int8, beam=5, temperature 0.0, no VAD |
| In-image text (OCR) | RapidOCR (ch_PP-OCRv3) on ONNX Runtime |
CPU, deterministic |
| Visual semantics | BLIP Salesforce/blip-image-captioning-base (PyTorch + HF Transformers) |
CPU float32, greedy, no sampling, pinned revision |
| Video | PyAV decode → scene-change keyframes → the same BLIP captioner | timestamp-based sampling, histogram scene detection |
All of them run offline, locally, at temperature 0 — no API calls, no network during inference.
Evaluated on the 30 solved examples, with leakage controls: labels are stripped before feature construction, and historical self-retrieval is excluded by id and by timestamp.
action accuracy 0.967 (29/30)
action macro-F1 0.966
message_type accuracy 0.967 (29/30)
message_type macro-F1 0.975
confidence MAE vs gold 0.058
confidence on correct 0.853 vs on wrong 0.776
The evaluator also emits per-class precision/recall/F1, confusion matrices for both
targets, and 8 behaviour slices (safety_related, promotional_content, has_media,
muted_group, missing_context, …). With n=30 these slices are directional only, and
the CLI says so rather than dressing them up.
Perception is gated behind a 4-mode ablation switch (off / auto / cache-required /
required) so each layer's contribution can be isolated and measured.
Turning vision + ASR on did not improve action accuracy (0.967 either way), and message-type macro-F1 fell from 0.975 to 0.860. That result is reported here rather than buried, because the point of building the ablation harness was to find out — and a 30-example evaluation set is too small to justify shipping a regression.
A clip is not thirty still images. Captioning every frame is mostly paying to describe the same scene over and over, and averaging the results throws away the one thing video actually carries: when things changed.
The video layer works in four steps:
- Sample by timestamp, not frame index. One frame per stride of presentation time, so a variable frame rate clip and a constant one cut from the same footage land on the same sample points.
- Select keyframes by scene change. Each sample is reduced to a 32-bin luminance histogram; a sample becomes a keyframe when its L1/2 distance from the last kept keyframe exceeds the threshold. Anchoring on the last kept frame rather than the previous sample is what lets a slow pan register once it has genuinely drifted, instead of never.
- Caption only the keyframes, through the exact same BLIP provider and quality gate the still-image path uses — so a video frame has to clear the same blank, repetition and generation-probability bars an attached photo does.
- Merge back into time. Consecutive keyframes described the same way collapse into one segment with a start and an end, and the clip summary is a join of those segments in playback order.
Measured on synthetic clips you can regenerate yourself — --plan reports this without
loading the model, so the table below costs nothing to check:
python code/media/make_sample_clips.py --output clips/
python code/media/video_extract.py --input clips/ --plan| Clip | Sampled | Keyframes | Captioner calls avoided | Result |
|---|---|---|---|---|
| Three 4s scenes | 12 | 3 | 9 (75%) | cuts found at 0.0s / 4.0s / 8.0s |
| Static 8s clip | 8 | 1 | 7 (87.5%) | one segment spanning the clip |
| Uniform grey 8s | 8 | 1 | 7 | blank — refuses to describe it |
Re-sampling the three-scene clip at a 2.0s stride instead of 1.0s finds the same cut points, which is the property timestamp-based sampling exists to give. Repeat runs produce identical segments, summary and quality.
The blank case matters most: BLIP captions a uniform grey frame as "a woman in a white dress"
with high confidence. The video is reported as blank with no segments rather than passing
that sentence downstream.
The summary is deliberately a mechanical join, not a second generation pass — a model asked to narrate a list of captions will happily invent the connective tissue between them.
# tune stride and threshold against a clip before spending any model time
python code/media/video_extract.py --input clip.mp4 --plan --stride 2.0 --scene-distance 0.15
# full understanding, against a local copy of the pinned BLIP revision
python code/media/video_extract.py --input clip.mp4 --model-dir /path/to/blip --jsonFrame selection and temporal merging live in code/media/video_keyframes.py, which is
standard-library only and holds no decoder or model state — so the parts of video
understanding that are actual decisions are unit-tested without PyAV, Torch, or a 990 MB
checkpoint on disk.
Scope: this layer is built, tested and runnable on its own. It is not yet wired into the
router's decision path — messages.csv has no video column to wire it to.
Understanding is earned, not assumed. A transcript or caption is only trusted when strict validation passes: content hash, schema, source path/size/SHA-256, and the complete provider/model/extraction configuration must match a code-owned expected config. A cache record is never allowed to authorize its own configuration. Non-empty output is not sufficient on its own.
Bad model output is detected, not consumed. Blank/uniform images (luminance stddev < 2.0),
images under 64px on the short side, repetition hallucinations, mostly-silence and failed
decodes are tiered low/none and never reach the decision layer — the row falls back to
text-only behaviour instead of acting on a confident-sounding guess. A capability probe
confirmed BLIP captions a blank image as "a woman in a white dress"; that is exactly the
failure this gate exists to catch.
Prompt injection is never obeyed. Text rendered inside an image instructing the router how to classify itself is a tier-0 mute, and the instruction is ignored. An adversarial probe recorded 0 instructions executed and 0 URLs opened.
Safety outranks personalization. Rules are tiered — SAFETY (0) > CONSENT (1) >
URGENCY (2) > VALUE (3) > DAMPENER (4). A scam is muted regardless of how much the user
engaged with that sender before, and the emitted reason cites the factor that actually drove
the decision.
Confidence comes from evidence quality, not from the chosen class, so a clear mute can
outrank an uncertain notify. There is a test for exactly that.
Validate, then publish. Output is written to a temp file in the destination directory,
fsync'd, validated, and only then atomically os.replaced into place. An invalid run can
never clobber a previously valid output.csv. (Writing first and validating after was a real
defect found and fixed in the Phase-1 audit.)
The router itself is Python standard library only — no pip install, no API keys.
Verified on CPython 3.13 (Windows 11).
# tests (synthetic fixtures, no dataset needed)
python -m unittest discover -s code/tests -t code -v # 277 tests
# predictions
python code/main.py --input dataset/messages.csv --output output.csv
# evaluation
python code/evaluation/main.py --all
# explain one decision end to end
python code/main.py --trace msg_048 --trace-onlyPerception layers need their own isolated environments and are extraction-only — routing consumes the frozen cache and never imports them:
python -m venv .venv-media
.venv-media/Scripts/python -m pip install -r code/media/requirements-media.txt
.venv-media/Scripts/python code/media/extract.py --model smallThe challenge dataset (dataset/ — messages, user/group/business tables, images, audio)
is the organizer's material and is not mine to redistribute, so it is not included.
The extracted transcript and caption caches are excluded for the same reason: they contain
verbatim content derived from that audio and those images.
Model weights are excluded too (the BLIP checkpoint alone is ~990 MB) and are fetched from Hugging Face at the pinned revision.
The test suite does not depend on any of it. code/tests/fixtures.py builds a small
synthetic dataset on disk, so all 277 tests run on a clean clone with nothing but Python.
To run the router on real data, supply files matching the schema in
code/REQUIREMENTS_MATRIX.md.
Message ─> ContextBuilder ──> MessageContext (all joins eager; missing = explicit)
──> content_profile (text signals)
──> media lookup (validated ASR / OCR / caption cache)
──> evidence.retrieve (this user's strictly-older history)
──> build_signals (content + context + behaviour)
──> RULES (tiered, scored)
──> action, message_type, confidence, reason, evidence
Every threshold lives in one file (code/router/policy.py). There is no message-id-specific
logic — a test greps router/*.py for dataset id literals and fails if any appear.
Deeper implementation notes: code/README.md.
MIT