The textproc/py-tokenizers port
py3-tokenizers-0.23.1p0 – tokenizers with a focus on performance (cvsweb github mirror)
Description
An implementation of today's most used tokenizers, with a focus on performance.WWW: https://pypi.org/project/tokenizers
Maintainer
The OpenBSD ports mailing-list
Only for arches
aarch64 alpha amd64 arm hppa i386 mips64 mips64el powerpc powerpc64 riscv64 sparc64
Categories
lang/python lang/rust textproc
Library dependencies
Build dependencies
Run dependencies
Test dependencies
Files
- /usr/local/lib/python3.14/site-packages/tokenizers-0.23.1.dist-info/
- /usr/local/lib/python3.14/site-packages/tokenizers-0.23.1.dist-info/METADATA
- /usr/local/lib/python3.14/site-packages/tokenizers-0.23.1.dist-info/RECORD
- /usr/local/lib/python3.14/site-packages/tokenizers-0.23.1.dist-info/WHEEL
- /usr/local/lib/python3.14/site-packages/tokenizers/
- /usr/local/lib/python3.14/site-packages/tokenizers/__init__.py
- /usr/local/lib/python3.14/site-packages/tokenizers/__init__.pyi
- /usr/local/lib/python3.14/site-packages/tokenizers/__pycache__/
- /usr/local/lib/python3.14/site-packages/tokenizers/__pycache__/__init__.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/__pycache__/__init__.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/decoders.pyi
- /usr/local/lib/python3.14/site-packages/tokenizers/decoders/
- /usr/local/lib/python3.14/site-packages/tokenizers/decoders/__init__.py
- /usr/local/lib/python3.14/site-packages/tokenizers/decoders/__pycache__/
- /usr/local/lib/python3.14/site-packages/tokenizers/decoders/__pycache__/__init__.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/decoders/__pycache__/__init__.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__init__.py
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__pycache__/
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__pycache__/__init__.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__pycache__/__init__.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__pycache__/base_tokenizer.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__pycache__/base_tokenizer.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__pycache__/bert_wordpiece.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__pycache__/bert_wordpiece.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__pycache__/byte_level_bpe.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__pycache__/byte_level_bpe.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__pycache__/char_level_bpe.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__pycache__/char_level_bpe.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__pycache__/sentencepiece_bpe.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__pycache__/sentencepiece_bpe.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__pycache__/sentencepiece_unigram.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/__pycache__/sentencepiece_unigram.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/base_tokenizer.py
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/bert_wordpiece.py
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/byte_level_bpe.py
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/char_level_bpe.py
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/sentencepiece_bpe.py
- /usr/local/lib/python3.14/site-packages/tokenizers/implementations/sentencepiece_unigram.py
- /usr/local/lib/python3.14/site-packages/tokenizers/models.pyi
- /usr/local/lib/python3.14/site-packages/tokenizers/models/
- /usr/local/lib/python3.14/site-packages/tokenizers/models/__init__.py
- /usr/local/lib/python3.14/site-packages/tokenizers/models/__pycache__/
- /usr/local/lib/python3.14/site-packages/tokenizers/models/__pycache__/__init__.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/models/__pycache__/__init__.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/normalizers.pyi
- /usr/local/lib/python3.14/site-packages/tokenizers/normalizers/
- /usr/local/lib/python3.14/site-packages/tokenizers/normalizers/__init__.py
- /usr/local/lib/python3.14/site-packages/tokenizers/normalizers/__pycache__/
- /usr/local/lib/python3.14/site-packages/tokenizers/normalizers/__pycache__/__init__.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/normalizers/__pycache__/__init__.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/pre_tokenizers.pyi
- /usr/local/lib/python3.14/site-packages/tokenizers/pre_tokenizers/
- /usr/local/lib/python3.14/site-packages/tokenizers/pre_tokenizers/__init__.py
- /usr/local/lib/python3.14/site-packages/tokenizers/pre_tokenizers/__pycache__/
- /usr/local/lib/python3.14/site-packages/tokenizers/pre_tokenizers/__pycache__/__init__.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/pre_tokenizers/__pycache__/__init__.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/processors.pyi
- /usr/local/lib/python3.14/site-packages/tokenizers/processors/
- /usr/local/lib/python3.14/site-packages/tokenizers/processors/__init__.py
- /usr/local/lib/python3.14/site-packages/tokenizers/processors/__pycache__/
- /usr/local/lib/python3.14/site-packages/tokenizers/processors/__pycache__/__init__.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/processors/__pycache__/__init__.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/py.typed
- /usr/local/lib/python3.14/site-packages/tokenizers/tokenizers.abi3.so
- /usr/local/lib/python3.14/site-packages/tokenizers/tokenizers.pyi
- /usr/local/lib/python3.14/site-packages/tokenizers/tools/
- /usr/local/lib/python3.14/site-packages/tokenizers/tools/__init__.py
- /usr/local/lib/python3.14/site-packages/tokenizers/tools/__pycache__/
- /usr/local/lib/python3.14/site-packages/tokenizers/tools/__pycache__/__init__.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/tools/__pycache__/__init__.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/tools/__pycache__/visualizer.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/tools/__pycache__/visualizer.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/tools/visualizer-styles.css
- /usr/local/lib/python3.14/site-packages/tokenizers/tools/visualizer.py
- /usr/local/lib/python3.14/site-packages/tokenizers/trainers.pyi
- /usr/local/lib/python3.14/site-packages/tokenizers/trainers/
- /usr/local/lib/python3.14/site-packages/tokenizers/trainers/__init__.py
- /usr/local/lib/python3.14/site-packages/tokenizers/trainers/__init__.pyi
- /usr/local/lib/python3.14/site-packages/tokenizers/trainers/__pycache__/
- /usr/local/lib/python3.14/site-packages/tokenizers/trainers/__pycache__/__init__.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/trainers/__pycache__/__init__.cpython-314.pyc