The textproc/py-tiktoken port
py3-tiktoken-0.13.0 – fast BPE tokenizer for OpenAI models (cvsweb github mirror)
Description
tiktoken is a fast BPE (byte pair encoding) tokenizer for use with OpenAI's models.WWW: https://pypi.org/project/tiktoken
Maintainer
The OpenBSD ports mailing-list
Only for arches
aarch64 amd64 riscv64 sparc64
Categories
lang/python lang/rust textproc
Build dependencies
Run dependencies
Test dependencies
Reverse dependencies
Files
- /usr/local/lib/python3.14/site-packages/tiktoken-0.13.0.dist-info/
- /usr/local/lib/python3.14/site-packages/tiktoken-0.13.0.dist-info/METADATA
- /usr/local/lib/python3.14/site-packages/tiktoken-0.13.0.dist-info/RECORD
- /usr/local/lib/python3.14/site-packages/tiktoken-0.13.0.dist-info/WHEEL
- /usr/local/lib/python3.14/site-packages/tiktoken-0.13.0.dist-info/licenses/
- /usr/local/lib/python3.14/site-packages/tiktoken-0.13.0.dist-info/licenses/LICENSE
- /usr/local/lib/python3.14/site-packages/tiktoken-0.13.0.dist-info/top_level.txt
- /usr/local/lib/python3.14/site-packages/tiktoken/
- /usr/local/lib/python3.14/site-packages/tiktoken/__init__.py
- /usr/local/lib/python3.14/site-packages/tiktoken/__pycache__/
- /usr/local/lib/python3.14/site-packages/tiktoken/__pycache__/__init__.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tiktoken/__pycache__/__init__.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tiktoken/__pycache__/_educational.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tiktoken/__pycache__/_educational.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tiktoken/__pycache__/core.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tiktoken/__pycache__/core.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tiktoken/__pycache__/load.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tiktoken/__pycache__/load.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tiktoken/__pycache__/model.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tiktoken/__pycache__/model.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tiktoken/__pycache__/registry.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tiktoken/__pycache__/registry.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tiktoken/_educational.py
- /usr/local/lib/python3.14/site-packages/tiktoken/_tiktoken.cpython-314.so
- /usr/local/lib/python3.14/site-packages/tiktoken/core.py
- /usr/local/lib/python3.14/site-packages/tiktoken/load.py
- /usr/local/lib/python3.14/site-packages/tiktoken/model.py
- /usr/local/lib/python3.14/site-packages/tiktoken/py.typed
- /usr/local/lib/python3.14/site-packages/tiktoken/registry.py
- /usr/local/lib/python3.14/site-packages/tiktoken_ext/
- /usr/local/lib/python3.14/site-packages/tiktoken_ext/__pycache__/
- /usr/local/lib/python3.14/site-packages/tiktoken_ext/__pycache__/openai_public.cpython-314.opt-1.pyc
- /usr/local/lib/python3.14/site-packages/tiktoken_ext/__pycache__/openai_public.cpython-314.pyc
- /usr/local/lib/python3.14/site-packages/tiktoken_ext/openai_public.py