nameparser 2.0 implementation#288
Draft
derek73 wants to merge 118 commits into
Draft
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v2 core design uses enum.StrEnum (3.11+). Full #257 (dropping typing_extensions, CI matrix) remains tracked on the issue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Validate that all replace() field values are str (user error if None) - Append missing-role synthetics in canonical Role order, not kwargs order - Unquote return type annotation (postponed annotations in effect) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Casefolded seven-component tuple in canonical Role order for dedup, dict keys, and sorting. Semantic layer comparison; __eq__ remains strict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds tests/v2/test_layering.py to mechanically enforce the conventions
doc's import layering and the public v2 export surface. Appends the v2
core types (Span, Role, Token, Ambiguity, AmbiguityKind, ParsedName,
Lexicon, Policy, PolicyPatch, PatronymicRule, UNSET, GIVEN_FIRST,
FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, Locale) to nameparser/__init__.py
alongside the existing v1 HumanName export.
Turns on component-flag mypy strictness for the four v2 modules and
check_untyped_defs for tests/v2, then fixes the resulting test-only
type mismatches by using the precise constructed types (Span(...),
frozenset({...}), tuple-of-pairs) where the literal type didn't matter
to the test, and adding narrow # type: ignore[arg-type] comments only
where a test deliberately exercises runtime coercion/validation of an
intentionally mismatched static type (e.g. Token span/Ambiguity kind
coercion tests, PatronymicRule string coercion, dict aliasing test).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pickle.dumps(Lexicon.default()) raised TypeError because the default slots-dataclass pickle path serializes the _cap_map MappingProxyType. Ship every other slot and rebuild the proxy from the canonical capitalization_exceptions tuple on load. Parser (Plan 3) is picklable by construction per the core spec, and a Parser holds a Lexicon. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
requires-python is >=3.11 on this branch, so the 3.10 job can no longer install the package (uv sync either fails or silently substitutes a managed 3.11). The rest of #257 (typing_extensions, classifiers) stays tracked on the issue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With requires-python at >=3.11, Self imports directly from typing and the conditional dependency can never activate; ruff (UP035/UP036) flags the dead version block once the floor is raised. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v1 suite is fully annotated (since #250 put tests/ under mypy), so pyproject carries no per-file ANN ignores; the new tests/v2 modules must be annotated too or 'ruff check' fails in CI. Also drops an unused import ruff flagged (F401). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/simplify (all four reviewers converged on the first): the trickiest membership expression in the vocabulary layer -- unambiguous acronym OR suffix word, ambiguous acronym only with periods -- was spelled in full in both classify and is_suffix_strict, aligned only by comments. Drift would make the comma-structure decision disagree with piece classification for the same token. Now _vocab.suffix_as_written is the single source; classify and the strict/lenient predicates compose it (and normalize each token once instead of twice on the lenient path). The three-comma set was likewise defined in tokenize and extract under different names with no sync note; it moves to _state.COMMA_CHARS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/simplify: the piece predicates were typed for group's internal mutable lists, so every cross-stage call paid a triple list/set/tuple conversion (including O(n) token-tuple copies inside assign's peel loops and group's maiden scan). They now take Sequence/Set and the call-site churn is gone. Also: the leading-title peel was spelled twice (assign's main path and the FAMILY_COMMA branch, the latter with a dead 'not given_done' guard) -- extracted as _peel_leading_titles; the trailing all-suffix segment loops collapse into one loop over the structure's tail; the five hand-rolled pieces/ptags parallel-slice merges in _group_segment become one merge() closure that keeps the arrays in lockstep; post_rules' 'others' list was only ever tested for truthiness (any()); the patronymic comma gate is a named rotations_apply boolean instead of falsifying the policy value; and the module docstring records the GIVEN_FIRST positional assumption the rules inherit from v1 (#270). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/simplify efficiency findings: with the default strip flags every non-space character paid two failing regex calls in _ignorable; both strip classes are entirely non-ASCII, so an isascii() fast path skips them for virtually all real input. Span is a NamedTuple and already sorts as a tuple, so the tuple(...) sort keys in tokenize/extract were pure allocation. The case runner's seven-field list now derives from Role, whose declaration order the type declares canonical -- a new Role member fails the runner instead of being silently skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…proxy Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…a shared parser cache Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…fix_delimiter parity) Migration plan M5. v1's suffix_delimiter fired at three parse sites (expand_suffix_delimiter, parser.py:1318/1332/1409): post-comma suffix-segment DETECTION and tail-segment CONSUMPTION. The v2 wiring: - segment: a delimiter-core token (configured delimiter, whitespace stripped) is transparent in the all-suffix tests, and a token that a no-whitespace core splits into all-suffix parts counts as a suffix (splits_into_suffixes) -- covers the SUFFIX_COMMA determination and suppresses the spurious COMMA_STRUCTURE flag v1 never emitted - group: tail segments (FAMILY_COMMA >=2, SUFFIX_COMMA >=1) drop delimiter-core tokens via the same structural mechanism as maiden markers; assemble already omits dropped tokens - the 'not yet consumed' Policy warning is gone: the field is consumed Five case rows pinned live against v1.4 (2026-07-16); the one intentional divergence -- a no-whitespace core keeps the token whole (anti-#100), rendering 'RN/CRNA' where v1 rendered 'RN, CRNA' -- is classified fix(suffix-delimiter-rendering) for the release log. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…y-tracked parsing Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ity equality Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…iliation follows) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bucket A) empty_attribute_default was removed in 2.0 (#255; nameparser/_config_shim.py's Constants.__setattr__ now raises AttributeError on assignment, and there is no stored default to read either). conftest.py's autouse fixture used to parametrize every test over ['', None] to reproduce tests.py's old dual-path regression check, plus snapshot/restore CONSTANTS around each test. The dual path no longer exists, so only the isolation half remains, un-parametrized. Also drops "regexes" from the restored collection attrs: it's a read-only _RegexesProxy in 2.0, so it can neither be mutated by a test nor reassigned by this fixture's restore loop. tests/base.py's `m()` helper had the same dependency (falling back to hn.C.empty_attribute_default, which no longer exists); empty attributes are always '' now, so it falls back to '' directly. pyproject.toml/conftest.py's TEMPORARY exclude lists are re-synced to the current not-yet-reconciled file set for this batch (test_python_api.py and test_constants.py go back on the list -- see the batch report for why). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file used ast to recover Sphinx "attribute docstring" doctests (bare string literals following an attribute assignment) from the Constants class in nameparser.config, since that convention is invisible to Python's own doctest.DocTestFinder. After the M11 swap, nameparser/config/__init__.py is a pure re-export shim -- it no longer defines a Constants class at all (the real one lives in nameparser/_config_shim.py), so the module-level AST walk raised ValueError on import: `(class_node,) = (... 0 matches ...)`. Even pointed at the right module, there is nothing left to discover: the shim's Constants fields are bare type annotations (`patronymic_name_order: bool`), not `attr = default` followed by a prose+doctest string literal, so the mechanism this file existed to protect no longer has any doctest examples to run. Deleting rather than porting, per bucket A. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
M12 reconciliation surfaced this: dropping empty buckets renumbered
the segments, so in 'Doe,, Jr.' the tail suffix landed in the given
segment and the lone-post-comma-title heuristic claimed it
(title='Jr.'). v1's mechanics, pinned live: collapse_whitespace strips
exactly ONE trailing comma as cosmetic; every other empty part keeps
its position -- an empty given segment ('Doe,,'), a failed
suffix-comma detection ('John Smith,, MD'), silent consumption of
empty tails ('Doe, John,, Jr.'). Five case rows pin the matrix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r, matches message Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
158 dual-run cases -> 67 tests (the empty_attribute_default dual-run parametrization died in the previous batch commit; per-run count 79 -> 67). Bucket A (deleted machinery): - test_unpickle_legacy_state_without_derived_sets: pinned __setstate__ backfill of the v1 per-parse _derived_* sets and the is_title predicate, both gone with the v1 parser. - the six is_prefix/is_conjunction/is_suffix list-fallthrough tests: the v1 predicate methods are not part of the facade (#280 hook list). - test_get_full_name_attribute_references_internal_lists: pinned the v1 full_name getter re-rendering from the live *_list state; 2.0 deliberately returns the stored input string (spec section 2 mutation semantics, pinned in tests/v2/test_facade.py::test_field_assignment_str_list_none) and the *_list attributes are snapshots without setters (spec section 2 exc. 1). Bucket B (scheduled removals now raising / silently changed): - #245 bytes: both HumanName(bytes) and the full_name setter now pytest.raises(TypeError, match="decode"). - #223 ==/hash: rewritten to assert the identity semantics (distinct instances unequal, strings never match, hash == object default, no warnings), with matches()/comparison_key() shown as the replacements. - #258 slices/__setitem__: slice access raises TypeError naming the issue; item assignment raises TypeError for every key; str-key reads stay silent. - #255: given_names empty expectation is '' outright. - Constants(regexes=...) raises TypeError pointing at Policy (deliberate 2.0 divergence, migration spec section 3 uniform rule). 3 of 67 tests still fail on one confirmed pipeline bug (SUFFIX_COMMA detection tests all post-comma segments where v1 tested only parts[1]; repro 'Dr. John P. Doe-Ray, CLU, CFP, LUTC'), so the file stays in collect_ignore_glob with a pointer comment; it is mypy-clean and leaves the mypy exclude. test_constants.py stays ignored, blocked on a wider shim-regression inventory (see conftest comment). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ulary
M12 reconciliation surfaced two more Plan-3 misports, both pinned
live against v1.4 (2026-07-16):
- the suffix-comma structure is decided by parts[1] ALONE
(parser.py:1318); v2 wrongly required every post-first segment to
be suffix-shaped, so one unrecognized credential ('LUTC') collapsed
the whole parse into FAMILY_COMMA. Non-suffix tails now ride along
as suffixes with the COMMA_STRUCTURE flag (v1 consumed them
silently; the flag is the v2 value-add).
- v1's period-joined derivation (parse_pieces): a token with an
interior period whose chunks include ANY title is a title
('Lt.Gov.', 'Mr.Smith'), else ANY suffix chunk makes it a suffix
('JD.CPA'). Title wins. Ported into classify with the ANY rule
kept deliberately.
Six case rows pin the matrix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Also lifts test_python_api.py's collect_ignore_glob entry: its 3 held tests pass after the suffix-comma and period-joined-vocabulary pipeline fixes (7bf53f3), making the file fully green (67/67). test_constants.py: 240 dual-run cases -> 94 tests (the dual-run parametrization died with #255; per-run 120 -> 94). Bucket A (deleted machinery): - the suffixes_prefixes_titles cached-union property block (~13 tests of cache priming/invalidation incl. the timeit performance guard and the is_rootname consistency pair); change tracking is the shim generation counter, covered in tests/v2/test_config_shim.py. Two contracts survived and are re-pinned on parsing surface: post-unpickle mutations are honored, and wholesale manager replacement re-wires tracking. - .elements raw-set accessor uses rewritten to set(...) (operators also return plain set now); _repr_collection_attrs/_repr_scalar_attrs class internals replaced by test-owned name lists. - __setstate__ missing-field validation test (the shim deliberately accepts partial state to load v1.4 blobs). - the legacy-unpickle warn-once pair (#279's warning machinery is gone). Bucket B (scheduled removals now raising): - #255 empty_attribute_default: assignment raises AttributeError; the six None-mode behavior tests collapse to one removal test (scalar pickle/copy round-trip intent re-pinned on string_format). - #243: SetManager() call -> TypeError not callable; missing-member remove() -> KeyError (mixed-args call still applies the present removal). - #245/#263: add_with_encoding -> AttributeError; bytes add() -> TypeError with decode hint. - #256: unknown tuple-manager keys -> AttributeError naming known keys; regexes proxy raises naming the missing key; sunder probes now report absent (AttributeError) instead of returning manager defaults. - #261: constants=None raises TypeError at constructor, positional, and C-setter entry points (class renamed ConstantsNoneRemovalTests). - deliberate 2.0 divergences pinned: custom nickname delimiters raise pointing at Policy (spec section 3); plain-iterable set-field assignment auto-wraps in a validated SetManager where v1 demanded a pre-built one (bare strings still raise, #241 stays impossible); class-not-instance constants message no longer carries the "did you mean Constants()" hint. Bucket C (shared-CONSTANTS mutation): - test_can_change_global_constants wraps its shared mutation in pytest.deprecated_call (it exercises exactly that deprecation). 2 of 94 tests still fail on one remaining v1.4-parity gap -- the shim Constants.__init__ lacks the patronymic_name_order/middle_name_as_last bool kwargs -- so the file stays in collect_ignore_glob and the mypy exclude with a pointer comment; remove both once the kwargs land. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
patronymic_name_order and middle_name_as_last were v1.4 constructor kwargs (docs/customize.rst doctests use them); the M12 reconciliation of test_constants.py caught the gap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Constants(patronymic_name_order=True) / Constants(middle_name_as_last= True) work again (c1b5730), so the two held ConstantsReprTests pass: 94/94 green. Removes the file's collect_ignore_glob and mypy-exclude entries and settles the three mypy items its un-excluding surfaced (|= re-wrap annotations, a dict[str, object] state annotation, and a note that the 2.0 TupleManager subscripts via dict's __class_getitem__ rather than v1's Generic). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
23 tests -> 22, all green. Bucket B (scheduled removals / deliberate divergences): - the two #254 None-mode scrubbing regressions merge into one residual test: the None display mode is gone (#255), but literal 'None'/'Nonez' name text must still survive formatting unscathed. - regexes.emoji/.bidi = False opt-outs raise TypeError in 2.0 (migration spec section 3 uniform rule); the tests now pin the error naming the Policy(strip_emoji/strip_bidi=False) replacements. Bucket C (shared-CONSTANTS mutation): - the four *_constants_attribute tests (string_format, capitalize_name, force_mixed_case_capitalization, both) move onto a private Constants passed as constants= -- the migration-guide idiom -- instead of mutating the shared singleton. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
33 tests -> 31, all green. Bucket B: the two #255 None-mode regressions (literal-"None" interpolation, all-empty fallback) collapse to one test pinning the 2.0 invariant -- a fully-empty initials() renders ''. Bucket C: the three *_constants tests (initials_format, initials_delimiter, delimiter+separator+format combo) move onto a private Constants passed as constants= instead of mutating the shared singleton; the fixture-restore sentinel test keeps asserting the shared default is ' '. Bucket E: the #232 doubled-space test's direct middle_list assignment becomes field assignment; the v1 hole it pinned (unnormalized *_list elements crashing initials) is unreachable in 2.0 -- list snapshots plus setter whitespace normalization -- so it now pins the normalized result. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… bug
The file needs no reconciliation edits (20 pass, 1 pre-existing v1
xfail), but one test fails on a render-layer parity bug: _render.py's
_cap_word lowercases every conjunction-set word, missing v1
is_conjunction's is_an_initial exclusion, so an initial that doubles as
a conjunction stays lowercase:
HumanName('scott e. werner').capitalize() -> 'Scott e. Werner'
(v1: 'Scott E. Werner'; 'e' is in CONJUNCTIONS)
The file stays in collect_ignore_glob with a pointer comment; it is
mypy-clean and leaves the mypy exclude.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The file needs no reconciliation edits (8 pass, 2 pre-existing v1
xfails), but one test fails on a confirmed pipeline parity bug: in the
family-comma format, a trailing suffix inside the family segment is not
split out of it.
Parser().parse('Smith Jr., John') -> family='Smith Jr.', suffix=''
(v1: last='Smith', first='John', suffix='Jr.')
Stays in collect_ignore_glob with a pointer comment; mypy-clean, so it
leaves the mypy exclude.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
57 tests: the stale 'U.S. District Judge' xfail is removed -- the 2.0
period-joined title derivation makes the long-desired expectation hold
(logged in docs/superpowers/plans/notes-m12-diffs.md for the M14
expected-changes list). 54 pass, 1 pre-existing v1 xfail remains.
2 tests fail on confirmed pipeline parity bugs, so the file stays in
collect_ignore_glob (mypy-clean, leaves the mypy exclude):
1. _lexicon._normalize strips INTERIOR periods, so 'J.R.' normalizes to
'jr' -- a TITLES vocabulary entry; v1's lc() kept 'j.r' unmatched.
HumanName('Smith, J.R.') -> title='J.R.'
(v1: first='J.R.', last='Smith')
2. _assign's nickname rule (plan deviation #2) counts name pieces AFTER
title peeling; v1's p_len==1 counted the whole segment before it.
HumanName('Xyz. (Bud) Smith') -> last='Smith'
(v1: title='Xyz.', first='Smith', nickname='Bud')
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All pinned live against v1.4 (2026-07-17):
- _normalize now matches v1's lc(): casefold + EDGE periods only.
'J.R.' no longer collapses to 'jr' and hits the periodless titles
vocabulary. Suffix-ACRONYM membership alone uses the period-free
form ('M.D.' -> 'md'), mirroring v1's is_suffix, which removed
periods only for the acronym test. The capitalization-exception
lookup tries both forms (v1 cap_word parity), keeping
'phd' -> 'Ph.D.' idempotent.
- _cap_word's conjunction lowering excludes initial-shaped words
(v1 is_conjunction): 'scott e. werner' capitalizes to
'Scott E. Werner'; an uppercase lone 'Y' stays 'Y'.
- FAMILY_COMMA's family segment peels per-piece suffixes after the
first piece ('Smith Jr., John' -> family=Smith, suffix=Jr.).
- the lone-piece nickname rule counts the segment BEFORE title
peeling ('Xyz. (Bud) Smith' -> title, given, nickname).
Six case rows pin the matrix; the three stale v2 unit tests that
pinned the pre-fix semantics are updated with v1 citations.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation branch for the 2.0 design — see the RFC in #285 and the umbrella issue #284 for the design discussion and feedback questions. Implementation learnings that change the design will land as amendment commits on #285, per the plan described there.
Progress
Span,Role,Token,Ambiguity/AmbiguityKind,ParsedName;Lexicon,Policy/PolicyPatch/apply_patch,Locale. Frozen, slotted, hashable, picklable dataclasses with eager fail-loud validation; ~110 dedicated tests undertests/v2/.render()/initials()/capitalized()/__str__)Parser/parse()/parser_for()/matches()+ shared case tableHumanNamefacade +CONSTANTSshim (existing test corpus as regression harness)ru,tr_az) + differential-parse harnessNotes on what's here so far
TypeError, well-typed bad value →ValueError, enum lookups stayValueErrorper stdlib precedent.tests/v2/test_layering.py), and the seven-field canonical order derives fromRoledeclaration order everywhere — no ordering is ever stated twice.nameparser/configdata modules through 2.x, including the newmaiden_markers.py(Recognize maiden-name markers like "née" and "geb." (Jane Smith née Jones) #274 — marker words like "née"/"geb."; decisions recorded on the issue).🤖 Generated with Claude Code