xtuner.v1.train.toy\_tokenizer.UTF8ByteTokenizer
================================================

.. currentmodule:: xtuner.v1.train.toy_tokenizer

.. autoclass:: UTF8ByteTokenizer
   

   
   .. rubric:: Methods
   

   .. autosummary::
   
   
   
      ~UTF8ByteTokenizer.convert_tokens_to_ids
   
   
      ~UTF8ByteTokenizer.decode
   
   
      ~UTF8ByteTokenizer.encode
   
   
      ~UTF8ByteTokenizer.save_pretrained
   

   
   
   
   
   .. automethod:: convert_tokens_to_ids
   
   
   
   .. automethod:: decode
   
   
   
   .. automethod:: encode
   
   
   
   .. automethod:: save_pretrained