■ ChatOpenAI 클래스의 with_structured_output 메소드를 사용해 쿼리 분석 시스템을 만드는 방법을 보여준다.
▶ main.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 |
import datetime import os from typing import Optional from langchain_core.pydantic_v1 import BaseModel, Field from langchain_community.document_loaders import YoutubeLoader from langchain_text_splitters import RecursiveCharacterTextSplitter from langchain_openai import OpenAIEmbeddings from langchain_chroma import Chroma from langchain_core.prompts import ChatPromptTemplate from langchain_openai import ChatOpenAI from langchain_core.runnables import RunnablePassthrough os.environ["OPENAI_API_KEY"] = "<OPENAI_API_KEY>" # 검색 모델 클래스를 정의한다. class Search(BaseModel): """Search over a database of tutorial videos about a software library.""" query : str = Field(... , description = "Similarity search query applied to video transcripts.") publish_year : Optional[int] = Field(None, description = "Year video was published") urlList = [ "https://www.youtube.com/watch?v=HAn9vnJy6S4", "https://www.youtube.com/watch?v=dA1cHGACXCo", "https://www.youtube.com/watch?v=ZcEMLz27sL4", "https://www.youtube.com/watch?v=hvAPnpSfSGo", "https://www.youtube.com/watch?v=EhlPDL4QrWY", "https://www.youtube.com/watch?v=mmBo8nlu2j0", "https://www.youtube.com/watch?v=rQdibOsL1ps", "https://www.youtube.com/watch?v=28lC4fqukoc", "https://www.youtube.com/watch?v=es-9MgxB-uc", "https://www.youtube.com/watch?v=wLRHwKuKvOE", "https://www.youtube.com/watch?v=ObIltMaRJvY", "https://www.youtube.com/watch?v=DjuXACWYkkU", "https://www.youtube.com/watch?v=o7C9ld6Ln-M" ] documentList = [] # 유튜브 동영상 정보를 로드한다. for url in urlList: documentList.extend(YoutubeLoader.from_youtube_url(url, add_video_info = True).load()) print(f"유튜브 동영상 정보 로드... {len(documentList)}") print() # 유튜브 동영상 메타 정보에 발행연도(publish-year) 항목을 추가한다. for document in documentList: document.metadata["publish_year"] = int(datetime.datetime.strptime(document.metadata["publish_date"], "%Y-%m-%d %H:%M:%S").strftime("%Y")) # 문서를 분할한다. recursiveCharacterTextSplitterr = RecursiveCharacterTextSplitter(chunk_size=2000) splitDocumentList = recursiveCharacterTextSplitterr.split_documents(documentList) # 벡터 저장소를 만든다. openAIEmbeddings = OpenAIEmbeddings(model="text-embedding-3-small") chroma = Chroma.from_documents( splitDocumentList, openAIEmbeddings, ) # 채팅 프롬프트 템플리트를 설정한다. system = """You are an expert at converting user questions into database queries. \ You have access to a database of tutorial videos about a software library for building LLM-powered applications. \ Given a question, return a list of database queries optimized to retrieve the most relevant results. If there are acronyms or words you are not familiar with, do not try to rephrase them.""" chatPromptTemplate = ChatPromptTemplate.from_messages([("system", system), ("human", "{question}")]) # OpenAI 모델을 초기화한다. chatOpenAI = ChatOpenAI(model = "gpt-3.5-turbo-0125", temperature = 0) # 구조화된 출력 시퀀스 및 쿼리 분석 시퀀스를 설정한다. structuredOutputRunnableSequence = chatOpenAI.with_structured_output(Search) queryAnalyzerRunnableSequence = {"question" : RunnablePassthrough()} | chatPromptTemplate | structuredOutputRunnableSequence # 질의 응답을 실행한다. search1 = queryAnalyzerRunnableSequence.invoke("how do I build a RAG agent") print( "질문 : how do I build a RAG agent") print(f"답변 : {search1}") print() search2 = queryAnalyzerRunnableSequence.invoke("videos on RAG published in 2023") print( "질문 : videos on RAG published in 2023") print(f"답변 : {search2}") """ 유튜브 동영상 정보 로드... 1 유튜브 동영상 정보 로드... 2 유튜브 동영상 정보 로드... 3 유튜브 동영상 정보 로드... 4 유튜브 동영상 정보 로드... 5 유튜브 동영상 정보 로드... 6 유튜브 동영상 정보 로드... 7 유튜브 동영상 정보 로드... 8 유튜브 동영상 정보 로드... 9 유튜브 동영상 정보 로드... 10 유튜브 동영상 정보 로드... 11 유튜브 동영상 정보 로드... 12 유튜브 동영상 정보 로드... 13 질문 : how do I build a RAG agent 답변 : query='build RAG agent' publish_year=None 질문 : videos on RAG published in 2023 답변 : query='RAG' publish_year=2023 """ |
▶ requirements.txt
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 |
aiohttp==3.9.5 aiosignal==1.3.1 annotated-types==0.7.0 anyio==4.4.0 asgiref==3.8.1 async-timeout==4.0.3 attrs==23.2.0 backoff==2.2.1 bcrypt==4.1.3 build==1.2.1 cachetools==5.3.3 certifi==2024.6.2 charset-normalizer==3.3.2 chroma-hnswlib==0.7.3 chromadb==0.5.0 click==8.1.7 coloredlogs==15.0.1 dataclasses-json==0.6.7 Deprecated==1.2.14 distro==1.9.0 dnspython==2.6.1 email_validator==2.1.1 exceptiongroup==1.2.1 fastapi==0.111.0 fastapi-cli==0.0.4 filelock==3.15.1 flatbuffers==24.3.25 frozenlist==1.4.1 fsspec==2024.6.0 google-auth==2.30.0 googleapis-common-protos==1.63.1 greenlet==3.0.3 grpcio==1.64.1 h11==0.14.0 httpcore==1.0.5 httptools==0.6.1 httpx==0.27.0 huggingface-hub==0.23.3 humanfriendly==10.0 idna==3.7 importlib_metadata==7.1.0 importlib_resources==6.4.0 Jinja2==3.1.4 jsonpatch==1.33 jsonpointer==3.0.0 kubernetes==30.1.0 langchain==0.2.3 langchain-chroma==0.1.1 langchain-community==0.2.4 langchain-core==0.2.5 langchain-openai==0.1.8 langchain-text-splitters==0.2.1 langsmith==0.1.77 markdown-it-py==3.0.0 MarkupSafe==2.1.5 marshmallow==3.21.3 mdurl==0.1.2 mmh3==4.1.0 monotonic==1.6 mpmath==1.3.0 multidict==6.0.5 mypy-extensions==1.0.0 numpy==1.26.4 oauthlib==3.2.2 onnxruntime==1.18.0 openai==1.34.0 opentelemetry-api==1.25.0 opentelemetry-exporter-otlp-proto-common==1.25.0 opentelemetry-exporter-otlp-proto-grpc==1.25.0 opentelemetry-instrumentation==0.46b0 opentelemetry-instrumentation-asgi==0.46b0 opentelemetry-instrumentation-fastapi==0.46b0 opentelemetry-proto==1.25.0 opentelemetry-sdk==1.25.0 opentelemetry-semantic-conventions==0.46b0 opentelemetry-util-http==0.46b0 orjson==3.10.4 overrides==7.7.0 packaging==23.2 posthog==3.5.0 protobuf==4.25.3 pyasn1==0.6.0 pyasn1_modules==0.4.0 pydantic==2.7.4 pydantic_core==2.18.4 Pygments==2.18.0 PyPika==0.48.9 pyproject_hooks==1.1.0 python-dateutil==2.9.0.post0 python-dotenv==1.0.1 python-multipart==0.0.9 pytube==15.0.0 PyYAML==6.0.1 regex==2024.5.15 requests==2.32.3 requests-oauthlib==2.0.0 rich==13.7.1 rsa==4.9 shellingham==1.5.4 six==1.16.0 sniffio==1.3.1 SQLAlchemy==2.0.30 starlette==0.37.2 sympy==1.12.1 tenacity==8.3.0 tiktoken==0.7.0 tokenizers==0.19.1 tomli==2.0.1 tqdm==4.66.4 typer==0.12.3 typing-inspect==0.9.0 typing_extensions==4.12.2 ujson==5.10.0 urllib3==2.2.1 uvicorn==0.30.1 uvloop==0.19.0 watchfiles==0.22.0 websocket-client==1.8.0 websockets==12.0 wrapt==1.16.0 yarl==1.9.4 youtube-transcript-api==0.6.2 zipp==3.19.2 |
※ pip install langchain langchain-chroma langchain-community langchain-openai pytube youtube-transcript-api 명령을 실행했다.