Tamil TN Cardinal Semiotic Class - #449
Conversation
b06d8e8 to
6b431c2
Compare
Signed-off-by: surendran-246 <surendrans@nvidia.com>
Signed-off-by: surendran-246 <surendrans@nvidia.com>
Signed-off-by: surendran-246 <surendrans@nvidia.com>
Signed-off-by: surendran-246 <surendrans@nvidia.com>
Signed-off-by: surendran-246 <surendrans@nvidia.com>
Signed-off-by: surendran-246 <surendrans@nvidia.com>
Signed-off-by: surendran-246 <surendrans@nvidia.com>
for more information, see https://pre-commit.ci Signed-off-by: surendran-246 <surendrans@nvidia.com>
Signed-off-by: surendran-246 <surendrans@nvidia.com>
231dd35 to
eaf2223
Compare
Signed-off-by: surendran-246 <surendrans@nvidia.com>
surendran-246
changed the title
Signed-off-by: surendran-246 <surendrans@nvidia.com>
Signed-off-by: surendran-246 <surendrans@nvidia.com>
Signed-off-by: surendran-246 <surendrans@nvidia.com>
Signed-off-by: surendran-246 <surendrans@nvidia.com>
for more information, see https://pre-commit.ci
Signed-off-by: surendran-246 <surendrans@nvidia.com>
|
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
Signed-off-by: surendran-246 <surendrans@nvidia.com>
0e62120 to
2907744
Compare
Signed-off-by: surendran-246 <surendrans@nvidia.com>
for more information, see https://pre-commit.ci
Signed-off-by: surendran-246 <surendrans@nvidia.com>
|
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
Signed-off-by: surendran-246 <surendrans@nvidia.com>
for more information, see https://pre-commit.ci
folivoramanh
left a comment
folivoramanh
left a comment
There was a problem hiding this comment.
make sure both pytest and sparrrowhawk test pass
There was a problem hiding this comment.
is there any solution that not listing 10 - 99? (eg 20 - 29 share same rule ಮೂ + digit?)
There was a problem hiding this comment.
In Tamil, combining tens + digit isn't simple concatenation — the words fuse at the joint (sandhi). E.g. "20"+"1" isn't "இருபதுஒன்று", it's இருபத்தொன்று; "20"+"2" fuses to இருபத்திரண்டு. This fusion is specific to digits 1 and 2 and can't be generalized safely, so we keep 10–99 fully listed rather than generating them at runtime.
There was a problem hiding this comment.
same question: is there any solution that not listing 10 - 99? (eg 20 - 29 share same rule ಮೂ + digit?)
|
|
||
| class CardinalFst(GraphFst): | ||
| """ | ||
| Finite state transducer for classifying cardinals, e.g. |
There was a problem hiding this comment.
please check other language's format and refractor the docstring
There was a problem hiding this comment.
I've checked the implementations in other languages and updated it accordingly.
| digit = sf("digit") | ||
| zero = sf("zero") | ||
| teens_ties = pynini.union(sf("teens_and_ties"), sf("teens_and_ties_en")) | ||
| teens_and_ties = pynutil.add_weight(teens_ties, -0.1) |
There was a problem hiding this comment.
try not to add custom weight
There was a problem hiding this comment.
Updated the cardinal.py file and removed the custom weight
| def __init__(self, deterministic: bool = True, lm: bool = False): | ||
| super().__init__(name="cardinal", kind="classify", deterministic=deterministic) | ||
|
|
||
| def sf(name): |
| super().__init__(name="word", kind="classify", deterministic=deterministic) | ||
|
|
||
| # Define Tamil characters and symbols using pynini.union | ||
| TAMIL_CHAR = pynini.union( |
There was a problem hiding this comment.
Referred to the Japanese word.py implementation and removed the unnecessary code accordingly.
There was a problem hiding this comment.
if dont use post processing, is there any redundant blank space? if yes, try to figure out the root cause from your implementation (punct, word, etc...), do not abuse post processing
There was a problem hiding this comment.
Investigated the blank space issue in the punctuation and word processing.I removed the post_processing.py file and its dependency.
| output = SPACE_DUP.sub(' ', output[1:]) | ||
|
|
||
| if self.lang in ["en", "hi", "vi"] and hasattr(self, 'post_processor') and self.post_processor is not None: | ||
| if ( |
There was a problem hiding this comment.
if self.lang in ["en", "hi", "ta", "vi"] and hasattr(self, 'post_processor') and self.post_processor is not None:
There was a problem hiding this comment.
done, also removed ta language from there as you suggested not to use post_processing.
| to_lower = args.input_case == "lower_cased" | ||
| training_data = load_files([file_path], to_lower=to_lower) | ||
|
|
||
| # print("Loading training data: " + file_path) |
| @pytest.mark.unit | ||
| def test_norm(self, test_input, expected): | ||
| pred = self.normalizer.normalize(test_input, verbose=False) | ||
| assert pred.strip() == expected.strip() |
Signed-off-by: surendran-246 <surendrans@nvidia.com>
Signed-off-by: surendran-246 <surendrans@nvidia.com>
for more information, see https://pre-commit.ci
2 participants
What does this PR do ?
Add a one line overview of what this PR aims to accomplish.
Before your PR is "Ready for review"
Pre checks:
git commit -sto sign.pytestor (if your machine does not have GPU)pytest --cpufrom the root folder (given you marked your test cases accordingly@pytest.mark.run_only_on('CPU')).bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...pytestand Sparrowhawk here.__init__.pyfor every folder and subfolder, includingdatafolder which has .TSV files?Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.to all newly added Python files?Copyright 2015 and onwards Google, Inc.. See an example here.try import: ... except: ...) if not already done.PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.