The textproc/py-tokenizers port
py3-tokenizers-0.23.1 – tokenizers with a focus on performance (cvsweb github mirror)
Description
An implementation of today's most used tokenizers, with a focus on performance.WWW: https://pypi.org/project/tokenizers
Maintainer
The OpenBSD ports mailing-list
Only for arches
aarch64 amd64 riscv64 sparc64
Categories
lang/python lang/rust textproc
Build dependencies
Run dependencies
Test dependencies
Reverse dependencies
Files
- /usr/local/lib/python3.14/site-packages/tokenizers/
- /usr/local/lib/python3.14/site-packages/tokenizers/__init__.py
- /usr/local/lib/python3.14/site-packages/tokenizers/__pycache__/
- /usr/local/lib/python3.14/site-packages/tokenizers/__pycache__/__init__.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/__pycache__/__init__.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tokenizers/tokenizers.abi3.so
- /usr/local/lib/python3.14/site-packages/tokenizers_python-0.23.1.dist-info/
- /usr/local/lib/python3.14/site-packages/tokenizers_python-0.23.1.dist-info/METADATA
- /usr/local/lib/python3.14/site-packages/tokenizers_python-0.23.1.dist-info/RECORD
- /usr/local/lib/python3.14/site-packages/tokenizers_python-0.23.1.dist-info/WHEEL