[PYTHON/LANGCHAIN] LlamaCpp 클래스 : 생성자에서 callback_manager 인자를 사용해 스트리밍 표준 출력 콜백 핸들러 설정하기

■ LlamaCpp 클래스의 생성자에서 callback_manager 인자를 사용해 스트리밍 표준 출력 콜백 핸들러를 설정하는 방법을 보여준다.

▶ main.py


from langchain_community.llms import LlamaCpp
from langchain_core.callbacks import CallbackManager
from langchain_core.callbacks import StreamingStdOutCallbackHandler

llamaCpp = LlamaCpp(
    model_path       = "./llama-2-13b-chat.Q4_0.gguf", # llama-2-13b-chat.Q4_0.gguf 파일 경로를 설정한다.
    n_gpu_layers     = 1,                              # GPU 메모리에 로드할 레이어 수, 모델의 한 레이어만 GPU 메모리에 로드된다(1개이면 충분하다).
    n_batch          = 2048,                           # 모델이 병렬로 처리해야 하는 토큰 수, 1과 n_ctx 사이의 값을 선택하는 것이 좋다
    n_ctx            = 2048,                           # 토큰 컨텍스트 창, 모델은 한 번에 2048개의 토큰 창을 고려한다.
    f16_kv           = True,                           # 모델이 키/값 캐시에 반정밀도를 사용해야 하는지 여부를 설정한다.
                                                       # 모델은 메모리 효율성이 더 높은 반정밀도를 사용한다.
                                                       # 반드시 True로 설정해야 하며, 그렇지 않으면 몇 번의 호출 후에 문제가 발생하게 된다.
    callback_manager = CallbackManager([StreamingStdOutCallbackHandler()]),
    verbose          = False
)

llamaCpp.invoke("The first man on the moon was ... Let's think step by step")

"""
The Apollo 11 mission was launched in 1969. The spacecraft had three astronauts: Neil Armstrong, Edwin "Buzz" Aldrin, and Michael Collins.

The astronauts were trained to perform a variety of tasks on the moon's surface, including collecting samples, conducting experiments, and setting up equipment for future missions.

On July 20, 1969, Armstrong and Aldrin suited up in their spacesuits and prepared to descend to the moon's surface in the lunar module Eagle. Collins remained in orbit around the moon in the command module Columbia.

As they descended, Armstrong radioed back to Mission Control on Earth, "Houston, Tranquility Base here. The Eagle has landed."

Finally, it was time for Armstrong to step out of the lunar module and become the first person to set foot on the moon. He famously declared, "That's one small step for man, one giant leap for mankind."

So, the first man on the moon was ... Neil Armstrong!
"""

from langchain_community.llms import LlamaCpp

from langchain_core.callbacks import CallbackManager

from langchain_core.callbacks import StreamingStdOutCallbackHandler

llamaCpp = LlamaCpp(

model_path = "./llama-2-13b-chat.Q4_0.gguf", # llama-2-13b-chat.Q4_0.gguf 파일 경로를 설정한다.

n_gpu_layers = 1, # GPU 메모리에 로드할 레이어 수, 모델의 한 레이어만 GPU 메모리에 로드된다(1개이면 충분하다).

n_batch = 2048, # 모델이 병렬로 처리해야 하는 토큰 수, 1과 n_ctx 사이의 값을 선택하는 것이 좋다

n_ctx = 2048, # 토큰 컨텍스트 창, 모델은 한 번에 2048개의 토큰 창을 고려한다.

f16_kv = True, # 모델이 키/값 캐시에 반정밀도를 사용해야 하는지 여부를 설정한다.

# 모델은 메모리 효율성이 더 높은 반정밀도를 사용한다.

# 반드시 True로 설정해야 하며, 그렇지 않으면 몇 번의 호출 후에 문제가 발생하게 된다.

callback_manager = CallbackManager([StreamingStdOutCallbackHandler()]),

verbose = False

)

llamaCpp.invoke("The first man on the moon was ... Let's think step by step")

"""

The Apollo 11 mission was launched in 1969. The spacecraft had three astronauts: Neil Armstrong, Edwin "Buzz" Aldrin, and Michael Collins.

The astronauts were trained to perform a variety of tasks on the moon's surface, including collecting samples, conducting experiments, and setting up equipment for future missions.

On July 20, 1969, Armstrong and Aldrin suited up in their spacesuits and prepared to descend to the moon's surface in the lunar module Eagle. Collins remained in orbit around the moon in the command module Columbia.

As they descended, Armstrong radioed back to Mission Control on Earth, "Houston, Tranquility Base here. The Eagle has landed."

Finally, it was time for Armstrong to step out of the lunar module and become the first person to set foot on the moon. He famously declared, "That's one small step for man, one giant leap for mankind."

So, the first man on the moon was ... Neil Armstrong!

"""

▶ requirements.txt


aiohttp==3.9.5
aiosignal==1.3.1
annotated-types==0.7.0
async-timeout==4.0.3
attrs==23.2.0
certifi==2024.6.2
charset-normalizer==3.3.2
dataclasses-json==0.6.7
diskcache==5.6.3
frozenlist==1.4.1
greenlet==3.0.3
idna==3.7
Jinja2==3.1.4
jsonpatch==1.33
jsonpointer==3.0.0
langchain==0.2.6
langchain-community==0.2.6
langchain-core==0.2.10
langchain-text-splitters==0.2.2
langsmith==0.1.82
llama_cpp_python==0.2.79
MarkupSafe==2.1.5
marshmallow==3.21.3
multidict==6.0.5
mypy-extensions==1.0.0
numpy==1.26.4
orjson==3.10.5
packaging==24.1
pydantic==2.7.4
pydantic_core==2.18.4
PyYAML==6.0.1
requests==2.32.3
SQLAlchemy==2.0.31
tenacity==8.4.2
typing-inspect==0.9.0
typing_extensions==4.12.2
urllib3==2.2.2
yarl==1.9.4

aiohttp==3.9.5

aiosignal==1.3.1

annotated-types==0.7.0

async-timeout==4.0.3

attrs==23.2.0

certifi==2024.6.2

charset-normalizer==3.3.2

dataclasses-json==0.6.7

diskcache==5.6.3

frozenlist==1.4.1

greenlet==3.0.3

idna==3.7

Jinja2==3.1.4

jsonpatch==1.33

jsonpointer==3.0.0

langchain==0.2.6

langchain-community==0.2.6

langchain-core==0.2.10

langchain-text-splitters==0.2.2

langsmith==0.1.82

llama_cpp_python==0.2.79

MarkupSafe==2.1.5

marshmallow==3.21.3

multidict==6.0.5

mypy-extensions==1.0.0

numpy==1.26.4

orjson==3.10.5

packaging==24.1

pydantic==2.7.4

pydantic_core==2.18.4

PyYAML==6.0.1

requests==2.32.3

SQLAlchemy==2.0.31

tenacity==8.4.2

typing-inspect==0.9.0

typing_extensions==4.12.2

urllib3==2.2.2

yarl==1.9.4

※ pip install langchain-community llama-cpp-python 명령을 실행했다.