Nick Hill 49782fcb76 [Misc] Some minor simplifications to detokenization logic (#3670) ...

Some simplifications made for clarity.

Also moves detokenization-related functions from tokenizer.py to detokenizer.py.

2024-04-01 13:22:06 -07:00

7.8 KiB

Raw Blame History

View Raw