Repository navigation

TokenizeAnything

Public

About

A re-implementation of redpony/cdec's tokenize-anything.pl script in python

8 stars

1 watching

Report repository

Name		Name	Last commit message	Last commit date
samples		samples
tests		tests
tok		tok
README.md		README.md
scrape.py		scrape.py
tokenize-anything.py		tokenize-anything.py

Repository files navigation

TokenizeAnything

A re-implementation of redpony/cdec's tokenize-anything.pl script in python

samples/ is a bunch of data pulled from Wikipedia in a bunch of languages.

tok/ holds the same data, as tokenized by the original tokenize-anything.pl.

About

A re-implementation of redpony/cdec's tokenize-anything.pl script in python

8 stars

1 watching

Report repository

Releases

Packages

Contributors

Languages