Skip to content

Northern Sámi TN - #472

Open
jimregan wants to merge 35 commits into
NVIDIA:mainfrom
jimregan:northern-sami-tn2
Open

Northern Sámi TN#472
jimregan wants to merge 35 commits into
NVIDIA:mainfrom
jimregan:northern-sami-tn2

Conversation

@jimregan

@jimregan jimregan commented Aug 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? Use git commit -s to sign.
  • Do all unittests finish successfully before sending PR?
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
  • Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
  • Remove import guards (try import: ... except: ...) if not already done.
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

If you haven't finished some of the above items you can still open "Draft" PR.

@jimregan
jimregan force-pushed the northern-sami-tn2 branch 3 times, most recently from f0c2ce0 to 1bcf746 Compare August 16, 2026 22:43
mgrafu and others added 27 commits August 17, 2026 00:50
@mgrafu @shrpawar-alt
@pre-commit-ci @jimregan
Mege staging to main - Hi TN v3 (NVIDIA#464)
9de7248 
* Hi TN: fraction class improvements with lexical mappings for special cases (NVIDIA#420)

* Fix: fraction class improvements with lexical mappings

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* move common fraction mappings to TSV and add ASCII digit support

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Jenkins date conflict resolved

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Hi TN date class accuracy improvement (NVIDIA#418)

* date class accuracy improvement

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Jenkins file date update for Hi TN

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address Reviewer feedback regarding date tagger, tsv files and inclusion of more test cases

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* hi-tn-date: minor formatting fix

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* date tagger modification according to feedback removed support for d-mm-yyyy, dd-m-yyyy and mm-yyyy

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* replace open()+union() with string_map for suffix and prefix unions

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Hi TN money class bug fix (NVIDIA#424)

* Hi TN money class bug fix

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added test cases with decimal portion having more than two digits for Indian Currency

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Hi TN Electronic: Fix partial TSV matching, chemical formula logic, and relative file paths (NVIDIA#425)

* Fix partial TSV matching, added support for chemical formulas, and relative file paths

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor electronic class with dynamic symbol matrix and chemical rules. Replaced hardcoded TSVs and Python symbol rules with a data-driven symbol_classes.tsv and dynamic elements.tsv. Fixed greedy over-tagging of standard English words and added Unicode support for complex chemical ions without breaking URL hyphen logic.

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Remove single capital letter units from Measure class (NVIDIA#429)

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Hi TN: Implement Serial tagger (NVIDIA#440)

* Hi TN Serial: Implement SerialFst tagger for Devanagari-numeric mixtures, number chains, and mathematical powers

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address feedback for Hindi Serial Tagger PR

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Hi TN: Implement Roman semiotic class (NVIDIA#442)

* Hi TN Roman class implementation

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Refactor Roman FST to utilize shared serial Devanagari chars

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Add __init__.py for serial and roman data folders

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update roman_graph weight to 1.07 and other minor changes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Feedback changes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* reducing uncommon entries in electronic class tsv data files (NVIDIA#458)

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Hi TN: Address, Electronic, Serial, and Cardinal FST Optimization and Bug Fix (NVIDIA#461)

* Hi TN Address, Electronic, Serial, and Cardinal FST Optimizations and Bug Fixes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* review feedback changes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feedback review changes for address and electronic

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Hi tn v3: staging to main error fixed (NVIDIA#466)

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Signed-off-by: Mariana <47233618+mgrafu@users.noreply.github.com>
Co-authored-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Jim O'Regan <joregan@kth.se>
@tbartley94 @bbae0312
@pre-commit-ci @mgrafu @jimregan
Ko TN: Add serial grammar and review fixes (NVIDIA#457) (NVIDIA#465)
5bb4b58 
* Ko TN: Add serial grammar and review fixes (NVIDIA#457)

* Add Korean TN post-processing rules for particle agreement and month handling

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Korean TN fraction test cases for particle agreement

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* Fix Korean fraction verbalization with particle-aware handling and remove post_processing dependency

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix date and fraction normalization issues based on review feedback

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Korean TN serial and review fixes

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address Korean TN review feedback

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Resolve Korean telephone conflict and remove serial references

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address serial and telephone review comments

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* KO TN: Update Jenkins cache (NVIDIA#467)

* Add Korean TN post-processing rules for particle agreement and month handling

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Korean TN fraction test cases for particle agreement

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* Fix Korean fraction verbalization with particle-aware handling and remove post_processing dependency

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix date and fraction normalization issues based on review feedback

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Korean TN serial and review fixes

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address Korean TN review feedback

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Resolve Korean telephone conflict and remove serial references

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address serial and telephone review comments

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update Korean TN Jenkins cache

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

---------

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

---------

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
Signed-off-by: Mariana <47233618+mgrafu@users.noreply.github.com>
Co-authored-by: Jinwoo Bae <bbae7050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Mariana <47233618+mgrafu@users.noreply.github.com>
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
recreate commits from NVIDIA#11 (Northern Sámi TN)
766481b 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@kathiasi @jimregan
Northern Sami symbol names
0e8abb5 
Signed-off-by: Katri Hiovain-Asikainen <katri.hiovain-asikainen@uit.no>
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Extend Northern Sami text normalization
8421434 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Add Northern Sami Sparrowhawk tests
9471060 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Add Northern Sami minor currency normalization
af82e7e 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Add Northern Sami arithmetic normalization
059d1a2 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@pre-commit-ci @jimregan
[pre-commit.ci] auto fixes from pre-commit.com hooks
f00219a 
for more information, see https://pre-commit.ci

Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Move minor currency symbols to data files
b0ae778 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Implement Northern Sami telephone normalization
38f3ba0 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Export Northern Sami leading-zero cardinals
d4d3926 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Add Northern Sami decimal normalization
085cc01 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Expand Northern Sami currency normalization
e03bb3c 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Add Northern Sami year readings
d8f93a0 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Expand Northern Sami date and time normalization
8599fba 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Add documented Northern Sami abbreviations
3448903 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Add Northern Sami mixed alphanumeric normalization
7ef5ed2 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Simplify Northern Sami telephone verbalization
677bea1 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Compose Northern Sami fractional currencies
a79dbfc 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Add Northern Sami date and year ranges
6eba73c 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Preserve spaces in mixed alphanumeric output
29578d6 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Replace Swedish fraction normalization
d1ba1d4 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Add Northern Sami cross-month date ranges
513259d 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Integrate Northern Sami alphanumeric words
1169917 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@pre-commit-ci @jimregan
[pre-commit.ci] auto fixes from pre-commit.com hooks
f2757c6 
for more information, see https://pre-commit.ci

Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Complete months_gen.tsv
9ec94a2 
Signed-off-by: Jim O’Regan <joregan@kth.se>
Signed-off-by: Jim O'Regan <joregan@kth.se>
jimregan and others added 5 commits August 17, 2026 00:50
@jimregan
Unify Northern Sami month mappings
4a04fe7 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Revert "Unify Northern Sami month mappings"
a4928e3 
This reverts commit 1bcf746.

Signed-off-by: Jim O'Regan <joregan@kth.se>
@pre-commit-ci @jimregan
[pre-commit.ci] auto fixes from pre-commit.com hooks
af3698d 
for more information, see https://pre-commit.ci

Signed-off-by: Jim O'Regan <joregan@kth.se>
@pre-commit-ci @jimregan
[pre-commit.ci] auto fixes from pre-commit.com hooks
af11cbd 
for more information, see https://pre-commit.ci

Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Implement Northern Sami vulgar fractions
dc80464 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Merge branch 'main' into northern-sami-tn2
ad147a8 
Signed-off-by: Jim O’Regan <joregan@kth.se>
@jimregan
Fix Northern Sami classifier interactions
4c77a2f 
Signed-off-by: Jim O'Regan <joregan@kth.se>
@jimregan
Merge branch 'northern-sami-tn2' of github.com:jimregan/NeMo-text-pro…
b1ed6ed 
…cessing into northern-sami-tn2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jimregan @mgrafu @tbartley94 @kathiasi