Commit 95b1fa9

committed

Add support: o200k_base tokenizer.

1 parent f7f95ca commit 95b1fa9Copy full SHA for 95b1fa9

4 files changed

+121

-128

lines changed

Cargo.lock
ext/tiktoken_ruby
- Cargo.toml
- src
  - lib.rs
lib
- tiktoken_ruby.rb

4 files changed

+121

-128

lines changed

Comments

(0)