License notices
Licenses and attribution notices for core projects and the available model tokenizers. See the source package manifest for the complete dependency list.
Loading tool…
Licenses and attribution notices for core projects and the available model tokenizers. See the source package manifest for the complete dependency list.
Loading tool…
Licenses and attribution notices for core projects and the available model tokenizers. See the source package manifest for the complete dependency list.
Tokenizing for solar-pro3 uses the tokenizer Upstage publishes. It is covered by the Upstage Solar License, and a copy of that license ships next to the tokenizer files.
The tokenizer JSON files are unmodified copies from a pinned upstream revision. Vocabulary, merge order, added-token order and token IDs are unchanged.
Solar is a trademark of Upstage Co., Ltd. This site is not an official Upstage product and is not sponsored, endorsed, or certified by Upstage.
This site is a fork of Tiktokenizer by dqbd. The repository itself carries the MIT license inherited from the original. The licenses below cover what this site ships or depends on directly.
Remixed by Bora Lee · Based on the original Tiktokenizer · Licenses