■ GPT2TokenizerFast 클래스의 from_pretrained 정적 메소드를 사용해 GPT2TokenizerFast 객체를 만드는 방법을 보여준다.
▶ main.py
|
1 2 3 4 5 |
from transformers import GPT2TokenizerFast gpt2TokenizerFast = GPT2TokenizerFast.from_pretrained("gpt2") |
▶ requirements.txt
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
certifi==2024.6.2 charset-normalizer==3.3.2 filelock==3.15.4 fsspec==2024.6.1 huggingface-hub==0.23.4 idna==3.7 numpy==1.26.4 packaging==24.1 PyYAML==6.0.1 regex==2024.5.15 requests==2.32.3 safetensors==0.4.3 tokenizers==0.19.1 tqdm==4.66.4 transformers==4.42.3 typing_extensions==4.12.2 urllib3==2.2.2 |
※ pip install transformers 명령을 실행했다.