Skip to content

Commit 1377c27

Browse files
committed
fix missing token issue
1 parent fa284c7 commit 1377c27

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

convert-pt-to-ggml.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -271,7 +271,7 @@ def bytes_to_unicode():
271271
fout.write(struct.pack("i", len(tokens)))
272272

273273
for key in tokens:
274-
text = bytearray([byte_decoder[c] for c in key]).decode('utf-8', errors='replace').encode('utf-8')
274+
text = bytearray([byte_decoder[c] for c in key])
275275
fout.write(struct.pack("i", len(text)))
276276
fout.write(text)
277277

0 commit comments

Comments
 (0)