Skip to content

[OpenSpec] [openregister] manual-entity-anonymisation #1593

Open

Description

@rjzondervan

OpenSpec change: manual-entity-anonymisation
Tracking PR: #1592 (scaffolding — proposal, design, capability spec delta, tasks)
Target capability: entity-relation-grondslagen (extending)

Summary

DocuDesk operators have a recurring need: anonymise a piece of text that Presidio detection missed. The legacy "pass extra entities to the anonymise endpoint" flow no longer works post-entity-relation-grondslagen — the HTTP path is relations-driven.

This change adds the missing write path with an operator-friendly endpoint shape: one POST per "operator decides to anonymise this text in this file", server-side string matching against the file's already-extracted chunks, atomic creation of the catalogue entry (or reuse of an existing one) plus one relation row per occurrence found, with full audit-trail capture.

New endpoint: POST /api/files/{fileId}/manual-entities — body { value, type, category?, wholeWord?, caseSensitive? }.

Defaults: whole-word and case-sensitive matching. Lookup-or-create on (value, type) keeps cross-document placeholder IDs stable. Idempotent on re-call (per-position dedup). Zero-match returns HTTP 200 with the catalogue entry created and a message. RBAC: caller MUST have write access to the file (same gate as markAsAnonymized / PATCH /api/entity-relations/{id}).

Overlap-aware matcher: TextExtractionService produces overlapping chunks (default 200-char overlap). Naive concatenation would double-count. The spec mandates per-chunk preg_match_all + absolute-position dedup with deterministic canonical-chunk selection.

Specs

This change adds 6 ADDED Requirements to the entity-relation-grondslagen capability spec:

  1. POST /api/files/{fileId}/manual-entities atomic create / reuse contract
  2. Chunk-aware exact-string matching with overlap-region dedup
  3. Catalogue lookup-or-create on (value, type) dedup key
  4. Audit-trail rows: entity_create (conditional) + entity_relations_batch_create (always)
  5. HTTP logs and error responses MUST NOT contain operator-supplied value (ADR-005)
  6. Substitution pass remains value-keyed; positions on manual relations are forensic markers

Tasks

1. Value objects and constants

  • 1.1 Create lib/Service/File/ChunkTextMatcher.php utility class
  • 1.2 Create lib/Exception/ManualEntityException.php with typed reason codes
  • 1.3 Add DetectionMethod constants (extend with 'manual')

2. Mapper extensions

  • 2.1 GdprEntityMapper::findOneByValueAndType lookup helper
  • 2.2 EntityRelationMapper::existsForFileAtPosition idempotency probe
  • 2.3 EntityRelationMapper::insertBatch atomic multi-row insert

3. ManualEntityService orchestrator

  • 3.1 Create lib/Service/File/ManualEntityService.php with DI wiring
  • 3.2 Implement addManualEntity() (transaction + lookup-or-create + match + batch insert + audit)
  • 3.3 Implement writeAuditTrails() (entity_create when new + entity_relations_batch_create always)
  • 3.4 Implement assertFileWriteAccess() (reuse existing helper)
  • 3.5 Create ManualEntityResult DTO

4. Controller endpoint

  • 4.1 Add FileTextController::addManualEntity(int $fileId): JSONResponse
  • 4.2 Register POST /api/files/{fileId}/manual-entities route
  • 4.3 PII-redacted request logging (no value in logs)

5. DocumentProcessingHandler — no changes required

  • 5.1 Verify findEntitiesForAnonymization includes manual-method rows (add test)
  • 5.2 Verify findEntityIdsByValueForFile includes manual-method rows (add test)

6. Spec maintenance

  • 6.1 Update openspec/specs/entity-relation-grondslagen/spec.md change list to in-progress

7. Tests

  • 7.1 ChunkTextMatcherTest (overlap dedup, whole-word, case-sensitivity, Unicode, needle-too-long, determinism)
  • 7.2 ManualEntityServiceTest (happy path, reuse, idempotency, zero-match, RBAC denial, audit-failure rollback)
  • 7.3 FileTextControllerTest extension (415 / 400 / 401 / 403 / 404 / 422 / 500 / 201 / 200 cases + log redaction)
  • 7.4 GdprEntityMapperTest extension (findOneByValueAndType lookup, dedup-invariant violation warning)
  • 7.5 EntityRelationMapperTest extension (existsForFileAtPosition, insertBatch, manual-method in findEntitiesForAnonymization)
  • 7.6 Integration test against stacked OR: upload → extract → POST manual-entity → anonymise → verify

8. Documentation

  • 8.1 Extend docs/api/objects.md with the new endpoint contract
  • 8.2 CHANGELOG entry under ### Added

9. Quality and verification

  • 9.1 composer check:strict clean
  • 9.2 openspec validate manual-entity-anonymisation clean
  • 9.3 Manual smoke against dev stack (Postman flow end-to-end)
  • 9.4 PHPCS + Conduction named-parameters rules clean

10. Cross-app coordination

  • 10.1 No DocuDesk-side code change in this PR (UI consumes the endpoint in a separate change)
  • 10.2 Open DocuDesk tracking issue for the operator UI work
  • 10.3 No softwarecatalog / opencatalogi changes required

Full task detail with implementation notes and acceptance criteria is in openspec/changes/manual-entity-anonymisation/tasks.md on PR #1592.

Flagged future work (out of scope here):

  • position-aware-substitution — substitution stays value-keyed; positions are forensic markers only.
  • presidio-overlap-dedup — Presidio detection has the same overlap-double-count behaviour as the naive matcher; this change makes manual entities smarter but doesn't fix the detection side.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions