FAISS Archives - icodebroker

[PYTHON/LANGCHAIN] RunnableWithMessageHistory 클래스 : invoke 메소드를 사용해 채팅하기 1

■ RunnableWithMessageHistory 클래스의 invoke 메소드를 사용해 채팅하는 방법을 보여준다. ※ OPENAI_API_KEY 환경 변수 값은 .env 파일에 정의한다. ▶ main.py


from dotenv                                     import load_dotenv
from langchain_community.tools.tavily_search    import TavilySearchResults
from langchain_community.document_loaders       import WebBaseLoader
from langchain_text_splitters                   import RecursiveCharacterTextSplitter
from langchain_openai                           import OpenAIEmbeddings
from langchain_community.vectorstores           import FAISS
from langchain.tools.retriever                  import create_retriever_tool
from langchain_openai                           import ChatOpenAI
from langchain                                  import hub
from langchain.agents                           import create_tool_calling_agent
from langchain.agents                           import AgentExecutor
from langchain_core.chat_history                import BaseChatMessageHistory
from langchain_community.chat_message_histories import ChatMessageHistory
from langchain_core.runnables.history           import RunnableWithMessageHistory

load_dotenv()

chatOpenAI = ChatOpenAI(model = "gpt-4o")

tavilySearchResults = TavilySearchResults(max_results = 2)

webBaseLoader = WebBaseLoader("https://docs.smith.langchain.com/overview")

documentList = webBaseLoader.load()

recursiveCharacterTextSplitter = RecursiveCharacterTextSplitter(chunk_size = 1000, chunk_overlap = 200)

splitDocumentList = recursiveCharacterTextSplitter.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

vectorStoreRetrieverTool = create_retriever_tool(
    vectorStoreRetriever,
    "langsmith_search",
    "Search for information about LangSmith. For any questions about LangSmith, you must use this tool!",
)

toolList = [tavilySearchResults, vectorStoreRetrieverTool]

chatPromptTemplate = hub.pull("hwchase17/openai-functions-agent")

runnableSequence = create_tool_calling_agent(chatOpenAI, toolList, chatPromptTemplate)

agentExecutor = AgentExecutor(agent = runnableSequence, tools = toolList)

sessionIDDictionary = {}

def getChatMessageHistory(sessionID : str) -> BaseChatMessageHistory:
    if sessionID not in sessionIDDictionary:
        sessionIDDictionary[sessionID] = ChatMessageHistory()
    return sessionIDDictionary[sessionID]

runnableWithMessageHistory = RunnableWithMessageHistory(
    agentExecutor,
    getChatMessageHistory,
    input_messages_key   = "input",
    history_messages_key = "chat_history",
)

responseDictionary1 = runnableWithMessageHistory.invoke(
    {"input" : "hi! I'm bob"},
    config = {"configurable" : {"session_id" : "<foo>"}}
)

print(responseDictionary1)
print("-" * 100)

responseDictionary2 = runnableWithMessageHistory.invoke(
    {"input" : "what's my name?"},
    config = {"configurable" : {"session_id" : "<foo>"}}
)

print(responseDictionary2)
print("-" * 100)

"""
{'input': "hi! I'm bob", 'chat_history': [], 'output': 'Hello Bob! How can I assist you today?'}

from dotenv import load_dotenv

from langchain_community.tools.tavily_search import TavilySearchResults

from langchain_community.document_loaders import WebBaseLoader

from langchain_text_splitters import RecursiveCharacterTextSplitter

from langchain_openai import OpenAIEmbeddings

from langchain_community.vectorstores import FAISS

from langchain.tools.retriever import create_retriever_tool

from langchain_openai import ChatOpenAI

from langchain import hub

from langchain.agents import create_tool_calling_agent

from langchain.agents import AgentExecutor

from langchain_core.chat_history import BaseChatMessageHistory

from langchain_community.chat_message_histories import ChatMessageHistory

from langchain_core.runnables.history import RunnableWithMessageHistory

load_dotenv()

chatOpenAI = ChatOpenAI(model = "gpt-4o")

tavilySearchResults = TavilySearchResults(max_results = 2)

webBaseLoader = WebBaseLoader("https://docs.smith.langchain.com/overview")

documentList = webBaseLoader.load()

recursiveCharacterTextSplitter = RecursiveCharacterTextSplitter(chunk_size = 1000, chunk_overlap = 200)

splitDocumentList = recursiveCharacterTextSplitter.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

vectorStoreRetrieverTool = create_retriever_tool(

vectorStoreRetriever,

"langsmith_search",

"Search for information about LangSmith. For any questions about LangSmith, you must use this tool!",

)

toolList = [tavilySearchResults, vectorStoreRetrieverTool]

chatPromptTemplate = hub.pull("hwchase17/openai-functions-agent")

runnableSequence = create_tool_calling_agent(chatOpenAI, toolList, chatPromptTemplate)

agentExecutor = AgentExecutor(agent = runnableSequence, tools = toolList)

sessionIDDictionary = {}

def getChatMessageHistory(sessionID : str) -> BaseChatMessageHistory:

if sessionID not in sessionIDDictionary:

sessionIDDictionary[sessionID] = ChatMessageHistory()

return sessionIDDictionary[sessionID]

runnableWithMessageHistory = RunnableWithMessageHistory(

agentExecutor,

getChatMessageHistory,

input_messages_key = "input",

history_messages_key = "chat_history",

)

responseDictionary1 = runnableWithMessageHistory.invoke(

{"input" : "hi! I'm bob"},

config = {"configurable" : {"session_id" : "<foo>"}}

)

print(responseDictionary1)

print("-" * 100)

responseDictionary2 = runnableWithMessageHistory.invoke(

{"input" : "what's my name?"},

config = {"configurable" : {"session_id" : "<foo>"}}

)

print(responseDictionary2)

print("-" * 100)

"""

{'input': "hi! I'm bob", 'chat_history': [], 'output': 'Hello Bob! How can I assist you today?'}

{'input': "what's

[PYTHON/LANGCHAIN] create_retriever_tool 함수 : FAISS 벡터 스토어 검색 도구 만들기 2

■ create_retriever_tool 함수를 사용해 FAISS 벡터 스토어 검색 도구를 만드는 방법을 보여준다. ※ OPENAI_API_KEY 환경 변수 값은 .env 파일에 정의한다. ▶ main.py

[PYTHON/LANGCHAIN] FAISS 클래스 : from_documents 정적 메소드를 사용해 FAISS 객체 만들기

■ FAISS 클래스의 from_documents 정적 메소드를 사용해 FAISS 객체를 만드는 방법을 보여준다. ※ OPENAI_API_KEY 환경 변수 값은 .env 파일에 정의한다. ▶ main.py

[PYTHON/LANGCHAIN] TimeWeightedVectorStoreRetriever 클래스 : mock_now 함수를 사용해 가상 시간 설정하기

■ TimeWeightedVectorStoreRetriever 클래스에서 mock_now 함수를 사용해 가상 시간을 설정하는 방법을 보여준다. ▶ main.py


import faiss

from langchain_openai                 import OpenAIEmbeddings
from langchain_community.docstore     import InMemoryDocstore
from langchain_community.vectorstores import FAISS
from langchain.retrievers             import TimeWeightedVectorStoreRetriever
from datetime                         import datetime
from datetime                         import timedelta
from langchain_core.documents         import Document
from langchain_core.utils             import mock_now

openAIEmbeddings = OpenAIEmbeddings()

embeddingSize = 1536

indexFlatL2 = faiss.IndexFlatL2(embeddingSize)

inMemoryDocstore = InMemoryDocstore({})

faiss = FAISS(openAIEmbeddings, indexFlatL2, inMemoryDocstore, {})

timeWeightedVectorStoreRetriever = TimeWeightedVectorStoreRetriever(vectorstore = faiss, decay_rate = 0.999, k = 1)

yesterday = datetime.now() - timedelta(days = 1)

timeWeightedVectorStoreRetriever.add_documents([Document(page_content = "hello world", metadata = {"last_accessed_at" : yesterday})])
timeWeightedVectorStoreRetriever.add_documents([Document(page_content = "hello foo")])

with mock_now(datetime(2024, 9, 16, 10, 11)):
    resultDocumentList = timeWeightedVectorStoreRetriever.get_relevant_documents("hello world")
    for resultDocument in resultDocumentList:
        print(resultDocument)

import faiss

from langchain_openai import OpenAIEmbeddings

from langchain_community.docstore import InMemoryDocstore

from langchain_community.vectorstores import FAISS

from langchain.retrievers import TimeWeightedVectorStoreRetriever

from datetime import datetime

from datetime import timedelta

from langchain_core.documents import Document

from langchain_core.utils import mock_now

openAIEmbeddings = OpenAIEmbeddings()

embeddingSize = 1536

indexFlatL2 = faiss.IndexFlatL2(embeddingSize)

inMemoryDocstore = InMemoryDocstore({})

faiss = FAISS(openAIEmbeddings, indexFlatL2, inMemoryDocstore, {})

timeWeightedVectorStoreRetriever = TimeWeightedVectorStoreRetriever(vectorstore = faiss, decay_rate = 0.999, k = 1)

yesterday = datetime.now() - timedelta(days = 1)

timeWeightedVectorStoreRetriever.add_documents([Document(page_content = "hello world", metadata = {"last_accessed_at" : yesterday})])

timeWeightedVectorStoreRetriever.add_documents([Document(page_content = "hello foo")])

with mock_now(datetime(2024, 9, 16, 10, 11)):

resultDocumentList = timeWeightedVectorStoreRetriever.get_relevant_documents("hello world")

for resultDocument in resultDocumentList:

print(resultDocument)

▶ requirements.txt


aiohappyeyeballs==2.4.0
aiohttp==3.10.5
aiosignal==1.3.1
annotated-types==0.7.0
anyio==4.4.0
attrs==24.2.0
certifi==2024.8.30
charset-normalizer==3.3.2
colorama==0.4.6
dataclasses-json==0.6.7
distro==1.9.0
faiss-cpu==1.8.0.post1
frozenlist==1.4.1
greenlet==3.1.0
h11==0.14.0
httpcore==1.0.5
httpx==0.27.2
idna==3.10
jiter==0.5.0
jsonpatch==1.33
jsonpointer==3.0.0
langchain==0.3.0
langchain-community==0.3.0
langchain-core==0.3.0
langchain-openai==0.2.0
langchain-text-splitters==0.3.0
langsmith==0.1.121
marshmallow==3.22.0
multidict==6.1.0
mypy-extensions==1.0.0
numpy==1.26.4
openai==1.45.1
orjson==3.10.7
packaging==24.1
pydantic==2.9.1
pydantic-settings==2.5.2
pydantic_core==2.23.3
python-dotenv==1.0.1
PyYAML==6.0.2
regex==2024.9.11
requests==2.32.3
sniffio==1.3.1
SQLAlchemy==2.0.35
tenacity==8.5.0
tiktoken==0.7.0
tqdm==4.66.5
typing-inspect==0.9.0
typing_extensions==4.12.2
urllib3==2.2.3
yarl==1.11.1

aiohappyeyeballs==2.4.0

aiohttp==3.10.5

aiosignal==1.3.1

annotated-types==0.7.0

anyio==4.4.0

attrs==24.2.0

certifi==2024.8.30

charset-normalizer==3.3.2

colorama==0.4.6

dataclasses-json==0.6.7

distro==1.9.0

faiss-cpu==1.8.0.post1

frozenlist==1.4.1

greenlet==3.1.0

h11==0.14.0

httpcore==1.0.5

httpx==0.27.2

idna==3.10

jiter==0.5.0

jsonpatch==1.33

jsonpointer==3.0.0

langchain==0.3.0

langchain-community==0.3.0

langchain-core==0.3.0

langchain-openai==0.2.0

langchain-text-splitters==0.3.0

langsmith==0.1.121

marshmallow==3.22.0

multidict==6.1.0

mypy-extensions==1.0.0

numpy==1.26.4

openai==1.45.1

orjson==3.10.7

packaging==24.1

pydantic==2.9.1

pydantic-settings==2.5.2

pydantic_core==2.23.3

python-dotenv==1.0.1

PyYAML==6.0.2

regex==2024.9.11

requests==2.32.3

sniffio==1.3.1

SQLAlchemy==2.0.35

tenacity==8.5.0

tiktoken==0.7.0

tqdm==4.66.5

typing-inspect==0.9.0

typing_extensions==4.12.2

urllib3==2.2.3

yarl==1.11.1

※ pip install langchain-community langchain-openai

[PYTHON/LANGCHAIN] TimeWeightedVectorStoreRetriever 클래스 : 높은 감쇠율로 문서 검색하기

■ TimeWeightedVectorStoreRetriever 클래스를 사용해 높은 감쇠율로 문서를 검색하는 방법을 보여준다. ※ 높은 감소율(예 : 9가 여러 개)에서는 최근성 점수가 빠르게 0으로 떨어진다!

[PYTHON/LANGCHAIN] TimeWeightedVectorStoreRetriever 클래스 : 낮은 감쇠율로 문서 검색하기

■ TimeWeightedVectorStoreRetriever 클래스를 사용해 낮은 감쇠율로 문서를 검색하는 방법을 보여준다. ※ 낮은 감쇠율(여기서는 극단적으로 0에 가깝게 설정했다)은 기억이 더 오랫동안 "기억"된다는 것을

[PYTHON/LANGCHAIN] TimeWeightedVectorStoreRetriever 클래스 : 생성자에서 vectorstore/decay_rate 인자를 사용해 TimeWeightedVectorStoreRetriever 객체 만들기

■ TimeWeightedVectorStoreRetriever 클래스의 생성자에서 vectorstore/decay_rate 인자를 사용해 TimeWeightedVectorStoreRetriever 객체를 만드는 방법을 보여준다. ▶ main.py


import faiss

from langchain_openai                 import OpenAIEmbeddings
from langchain_community.docstore     import InMemoryDocstore
from langchain_community.vectorstores import FAISS
from langchain.retrievers             import TimeWeightedVectorStoreRetriever

openAIEmbeddings = OpenAIEmbeddings()

embeddingSize = 1536

indexFlatL2 = faiss.IndexFlatL2(embeddingSize)

inMemoryDocstore = InMemoryDocstore({})

faiss = FAISS(openAIEmbeddings, indexFlatL2, inMemoryDocstore, {})

timeWeightedVectorStoreRetriever = TimeWeightedVectorStoreRetriever(vectorstore = faiss, decay_rate = 0.0000000000000000000000001, k = 1)

import faiss

from langchain_openai import OpenAIEmbeddings

from langchain_community.docstore import InMemoryDocstore

from langchain_community.vectorstores import FAISS

from langchain.retrievers import TimeWeightedVectorStoreRetriever

openAIEmbeddings = OpenAIEmbeddings()

embeddingSize = 1536

indexFlatL2 = faiss.IndexFlatL2(embeddingSize)

inMemoryDocstore = InMemoryDocstore({})

faiss = FAISS(openAIEmbeddings, indexFlatL2, inMemoryDocstore, {})

timeWeightedVectorStoreRetriever = TimeWeightedVectorStoreRetriever(vectorstore = faiss, decay_rate = 0.0000000000000000000000001, k = 1)

▶ requirements.txt


aiohappyeyeballs==2.4.0
aiohttp==3.10.5
aiosignal==1.3.1
annotated-types==0.7.0
anyio==4.4.0
attrs==24.2.0
certifi==2024.8.30
charset-normalizer==3.3.2
colorama==0.4.6
dataclasses-json==0.6.7
distro==1.9.0
faiss-cpu==1.8.0.post1
frozenlist==1.4.1
greenlet==3.1.0
h11==0.14.0
httpcore==1.0.5
httpx==0.27.2
idna==3.10
jiter==0.5.0
jsonpatch==1.33
jsonpointer==3.0.0
langchain==0.3.0
langchain-community==0.3.0
langchain-core==0.3.0
langchain-openai==0.2.0
langchain-text-splitters==0.3.0
langsmith==0.1.121
marshmallow==3.22.0
multidict==6.1.0
mypy-extensions==1.0.0
numpy==1.26.4
openai==1.45.1
orjson==3.10.7
packaging==24.1
pydantic==2.9.1
pydantic-settings==2.5.2
pydantic_core==2.23.3
python-dotenv==1.0.1
PyYAML==6.0.2
regex==2024.9.11
requests==2.32.3
sniffio==1.3.1
SQLAlchemy==2.0.35
tenacity==8.5.0
tiktoken==0.7.0
tqdm==4.66.5
typing-inspect==0.9.0
typing_extensions==4.12.2
urllib3==2.2.3
yarl==1.11.1

aiohappyeyeballs==2.4.0

aiohttp==3.10.5

aiosignal==1.3.1

annotated-types==0.7.0

anyio==4.4.0

attrs==24.2.0

certifi==2024.8.30

charset-normalizer==3.3.2

colorama==0.4.6

dataclasses-json==0.6.7

distro==1.9.0

faiss-cpu==1.8.0.post1

frozenlist==1.4.1

greenlet==3.1.0

h11==0.14.0

httpcore==1.0.5

httpx==0.27.2

idna==3.10

jiter==0.5.0

jsonpatch==1.33

jsonpointer==3.0.0

langchain==0.3.0

langchain-community==0.3.0

langchain-core==0.3.0

langchain-openai==0.2.0

langchain-text-splitters==0.3.0

langsmith==0.1.121

marshmallow==3.22.0

multidict==6.1.0

mypy-extensions==1.0.0

numpy==1.26.4

openai==1.45.1

orjson==3.10.7

packaging==24.1

pydantic==2.9.1

pydantic-settings==2.5.2

pydantic_core==2.23.3

python-dotenv==1.0.1

PyYAML==6.0.2

regex==2024.9.11

requests==2.32.3

sniffio==1.3.1

SQLAlchemy==2.0.35

tenacity==8.5.0

tiktoken==0.7.0

tqdm==4.66.5

typing-inspect==0.9.0

typing_extensions==4.12.2

urllib3==2.2.3

yarl==1.11.1

※ pip install langchain-community

[PYTHON/LANGCHAIN] VectorStoreRetriever 클래스 : configurable_fields 메소드에서 search_kwargs 인자에 ConfigurableField 객체 설정하기

■ VectorStoreRetriever 클래스의 configurable_fields 메소드에서 search_kwargs 인자에 ConfigurableField 객체를 설정하는 방법을 보여준다. ▶ main.py


from langchain_community.retrievers   import BM25Retriever
from langchain_openai                 import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain_core.runnables         import ConfigurableField
from langchain.retrievers             import EnsembleRetriever

stringList1 = [
    "I like apples",
    "I like oranges",
    "Apples and oranges are fruits"
]

bm25Retriever = BM25Retriever.from_texts(stringList1, metadatas = [{"source" : 1}] * len(stringList1))

bm25Retriever.k = 2

stringList2 = [
    "You like apples",
    "You like oranges"
]

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_texts(stringList2, openAIEmbeddings, metadatas = [{"source" : 2}] * len(stringList2))

vectorStoreRetriever = faiss.as_retriever(search_kwargs = {"k" : 2})

runnableConfigurableFields = vectorStoreRetriever.configurable_fields(
        search_kwargs = ConfigurableField(
            id          = "search_kwargs_faiss",
            name        = "Search Kwargs",
            description = "The search kwargs to use",
        )
    )

ensembleRetriever = EnsembleRetriever(retrievers = [bm25Retriever, runnableConfigurableFields], weights = [0.5, 0.5])

configurationDictionary = {"configurable" : {"search_kwargs_faiss" : {"k" : 1}}}

documentList = ensembleRetriever.invoke("apples", config = configurationDictionary)

for document in documentList:
    print(document.page_content)

from langchain_community.retrievers import BM25Retriever

from langchain_openai import OpenAIEmbeddings

from langchain_community.vectorstores import FAISS

from langchain_core.runnables import ConfigurableField

from langchain.retrievers import EnsembleRetriever

stringList1 = [

"I like apples",

"I like oranges",

"Apples and oranges are fruits"

]

bm25Retriever = BM25Retriever.from_texts(stringList1, metadatas = [{"source" : 1}] * len(stringList1))

bm25Retriever.k = 2

stringList2 = [

"You like apples",

"You like oranges"

]

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_texts(stringList2, openAIEmbeddings, metadatas = [{"source" : 2}] * len(stringList2))

vectorStoreRetriever = faiss.as_retriever(search_kwargs = {"k" : 2})

runnableConfigurableFields = vectorStoreRetriever.configurable_fields(

search_kwargs = ConfigurableField(

id = "search_kwargs_faiss",

name = "Search Kwargs",

description = "The search kwargs to use",

)

ensembleRetriever = EnsembleRetriever(retrievers = [bm25Retriever, runnableConfigurableFields], weights = [0.5, 0.5])

configurationDictionary = {"configurable" : {"search_kwargs_faiss" : {"k" : 1}}}

documentList = ensembleRetriever.invoke("apples", config = configurationDictionary)

for document in documentList:

print(document.page_content)

▶ requirements.txt


aiohappyeyeballs==2.4.0
aiohttp==3.10.5
aiosignal==1.3.1
annotated-types==0.7.0
anyio==4.4.0
attrs==24.2.0
certifi==2024.8.30
charset-normalizer==3.3.2
colorama==0.4.6
dataclasses-json==0.6.7
distro==1.9.0
faiss-cpu==1.8.0.post1
frozenlist==1.4.1
greenlet==3.1.0
h11==0.14.0
httpcore==1.0.5
httpx==0.27.2
idna==3.8
jiter==0.5.0
jsonpatch==1.33
jsonpointer==3.0.0
langchain==0.2.16
langchain-community==0.2.16
langchain-core==0.2.39
langchain-openai==0.1.23
langchain-text-splitters==0.2.4
langsmith==0.1.120
marshmallow==3.22.0
multidict==6.1.0
mypy-extensions==1.0.0
numpy==1.26.4
openai==1.45.0
orjson==3.10.7
packaging==24.1
pydantic==2.9.1
pydantic_core==2.23.3
PyYAML==6.0.2
rank-bm25==0.2.2
regex==2024.9.11
requests==2.32.3
sniffio==1.3.1
SQLAlchemy==2.0.34
tenacity==8.5.0
tiktoken==0.7.0
tqdm==4.66.5
typing-inspect==0.9.0
typing_extensions==4.12.2
urllib3==2.2.3
yarl==1.11.1

aiohappyeyeballs==2.4.0

aiohttp==3.10.5

aiosignal==1.3.1

annotated-types==0.7.0

anyio==4.4.0

attrs==24.2.0

certifi==2024.8.30

charset-normalizer==3.3.2

colorama==0.4.6

dataclasses-json==0.6.7

distro==1.9.0

faiss-cpu==1.8.0.post1

frozenlist==1.4.1

greenlet==3.1.0

h11==0.14.0

httpcore==1.0.5

httpx==0.27.2

idna==3.8

jiter==0.5.0

jsonpatch==1.33

jsonpointer==3.0.0

langchain==0.2.16

langchain-community==0.2.16

langchain-core==0.2.39

langchain-openai==0.1.23

langchain-text-splitters==0.2.4

langsmith==0.1.120

marshmallow==3.22.0

multidict==6.1.0

mypy-extensions==1.0.0

numpy==1.26.4

openai==1.45.0

orjson==3.10.7

packaging==24.1

pydantic==2.9.1

pydantic_core==2.23.3

PyYAML==6.0.2

rank-bm25==0.2.2

regex==2024.9.11

requests==2.32.3

sniffio==1.3.1

SQLAlchemy==2.0.34

tenacity==8.5.0

tiktoken==0.7.0

tqdm==4.66.5

typing-inspect==0.9.0

typing_extensions==4.12.2

urllib3==2.2.3

yarl==1.11.1

※ install langchain-community langchain-openai

[PYTHON/LANGCHAIN] EnsembleRetriever 클래스 : 생성자에서 retrievers/weights 인자를 사용해 EnsembleRetriever 객체 만들기

■ EnsembleRetriever 클래스의 생성자에서 retrievers/weights 인자를 사용해 EnsembleRetriever 객체를 만드는 방법을 보여준다. ▶ main.py


from langchain_community.retrievers   import BM25Retriever
from langchain_openai                 import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain.retrievers             import EnsembleRetriever

stringList1 = [
    "I like apples",
    "I like oranges",
    "Apples and oranges are fruits"
]

bm25Retriever = BM25Retriever.from_texts(stringList1, metadatas = [{"source" : 1}] * len(stringList1))

bm25Retriever.k = 2

stringList2 = [
    "You like apples",
    "You like oranges"
]

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_texts(stringList2, openAIEmbeddings, metadatas = [{"source" : 2}] * len(stringList2))

vectorStoreRetriever = faiss.as_retriever(search_kwargs = {"k" : 2})

ensembleRetriever = EnsembleRetriever(retrievers = [bm25Retriever, vectorStoreRetriever], weights = [0.5, 0.5])

documentList = ensembleRetriever.invoke("apples")

for document in documentList:
    print(document.page_content)

from langchain_community.retrievers import BM25Retriever

from langchain_openai import OpenAIEmbeddings

from langchain_community.vectorstores import FAISS

from langchain.retrievers import EnsembleRetriever

stringList1 = [

"I like apples",

"I like oranges",

"Apples and oranges are fruits"

]

bm25Retriever = BM25Retriever.from_texts(stringList1, metadatas = [{"source" : 1}] * len(stringList1))

bm25Retriever.k = 2

stringList2 = [

"You like apples",

"You like oranges"

]

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_texts(stringList2, openAIEmbeddings, metadatas = [{"source" : 2}] * len(stringList2))

vectorStoreRetriever = faiss.as_retriever(search_kwargs = {"k" : 2})

ensembleRetriever = EnsembleRetriever(retrievers = [bm25Retriever, vectorStoreRetriever], weights = [0.5, 0.5])

documentList = ensembleRetriever.invoke("apples")

for document in documentList:

print(document.page_content)

▶ requirements.txt


aiohappyeyeballs==2.4.0
aiohttp==3.10.5
aiosignal==1.3.1
annotated-types==0.7.0
anyio==4.4.0
attrs==24.2.0
certifi==2024.8.30
charset-normalizer==3.3.2
colorama==0.4.6
dataclasses-json==0.6.7
distro==1.9.0
faiss-cpu==1.8.0.post1
frozenlist==1.4.1
greenlet==3.1.0
h11==0.14.0
httpcore==1.0.5
httpx==0.27.2
idna==3.8
jiter==0.5.0
jsonpatch==1.33
jsonpointer==3.0.0
langchain==0.2.16
langchain-community==0.2.16
langchain-core==0.2.39
langchain-openai==0.1.23
langchain-text-splitters==0.2.4
langsmith==0.1.120
marshmallow==3.22.0
multidict==6.1.0
mypy-extensions==1.0.0
numpy==1.26.4
openai==1.45.0
orjson==3.10.7
packaging==24.1
pydantic==2.9.1
pydantic_core==2.23.3
PyYAML==6.0.2
rank-bm25==0.2.2
regex==2024.9.11
requests==2.32.3
sniffio==1.3.1
SQLAlchemy==2.0.34
tenacity==8.5.0
tiktoken==0.7.0
tqdm==4.66.5
typing-inspect==0.9.0
typing_extensions==4.12.2
urllib3==2.2.3
yarl==1.11.1

aiohappyeyeballs==2.4.0

aiohttp==3.10.5

aiosignal==1.3.1

annotated-types==0.7.0

anyio==4.4.0

attrs==24.2.0

certifi==2024.8.30

charset-normalizer==3.3.2

colorama==0.4.6

dataclasses-json==0.6.7

distro==1.9.0

faiss-cpu==1.8.0.post1

frozenlist==1.4.1

greenlet==3.1.0

h11==0.14.0

httpcore==1.0.5

httpx==0.27.2

idna==3.8

jiter==0.5.0

jsonpatch==1.33

jsonpointer==3.0.0

langchain==0.2.16

langchain-community==0.2.16

langchain-core==0.2.39

langchain-openai==0.1.23

langchain-text-splitters==0.2.4

langsmith==0.1.120

marshmallow==3.22.0

multidict==6.1.0

mypy-extensions==1.0.0

numpy==1.26.4

openai==1.45.0

orjson==3.10.7

packaging==24.1

pydantic==2.9.1

pydantic_core==2.23.3

PyYAML==6.0.2

rank-bm25==0.2.2

regex==2024.9.11

requests==2.32.3

sniffio==1.3.1

SQLAlchemy==2.0.34

tenacity==8.5.0

tiktoken==0.7.0

tqdm==4.66.5

typing-inspect==0.9.0

typing_extensions==4.12.2

urllib3==2.2.3

yarl==1.11.1

※ install langchain-community langchain-openai

[PYTHON/LANGCHAIN] ContextualCompressionRetriever 클래스 : DocumentCompressorPipeline 객체를 사용해 컨텍스트 압축 검색하기

■ ContextualCompressionRetriever 클래스에서 DocumentCompressorPipeline 객체를 사용해 컨텍스트 압축 검색하는 방법을 보여준다. ※ OPENAI_API_KEY 환경 변수 값은 .env 파일에 정의한다. ▶ main.py


from dotenv                                    import load_dotenv
from langchain_community.document_loaders      import TextLoader
from langchain_text_splitters                  import CharacterTextSplitter
from langchain_openai                          import OpenAIEmbeddings
from langchain_community.vectorstores          import FAISS
from langchain_community.document_transformers import EmbeddingsRedundantFilter
from langchain.retrievers.document_compressors import EmbeddingsFilter
from langchain.retrievers.document_compressors import DocumentCompressorPipeline
from langchain.retrievers                      import ContextualCompressionRetriever

load_dotenv()

def printDocumentList(documentList):
    print(
        f"\n{'-' * 100}\n".join(
            [f"Document {i + 1} :\n\n" + document.page_content for i, document in enumerate(documentList)]
        )
    )

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter1 = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter1.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

characterTextSplitter2 = CharacterTextSplitter(chunk_size = 300, chunk_overlap = 0, separator = ". ")

embeddingsRedundantFilter = EmbeddingsRedundantFilter(embeddings = openAIEmbeddings)

embeddingsFilter = EmbeddingsFilter(embeddings = openAIEmbeddings, similarity_threshold = 0.76)

documentCompressorPipeline = DocumentCompressorPipeline(transformers = [characterTextSplitter2, embeddingsRedundantFilter, embeddingsFilter])

contextualCompressionRetriever = ContextualCompressionRetriever(base_compressor = documentCompressorPipeline, base_retriever = vectorStoreRetriever)

resultDocumentList = contextualCompressionRetriever.invoke("What did the president say about Ketanji Jackson Brown")

printDocumentList(resultDocumentList)

from dotenv import load_dotenv

from langchain_community.document_loaders import TextLoader

from langchain_text_splitters import CharacterTextSplitter

from langchain_openai import OpenAIEmbeddings

from langchain_community.vectorstores import FAISS

from langchain_community.document_transformers import EmbeddingsRedundantFilter

from langchain.retrievers.document_compressors import EmbeddingsFilter

from langchain.retrievers.document_compressors import DocumentCompressorPipeline

from langchain.retrievers import ContextualCompressionRetriever

load_dotenv()

def printDocumentList(documentList):

print(

f"\n{'-' * 100}\n".join(

[f"Document {i + 1} :\n\n" + document.page_content for i, document in enumerate(documentList)]

)

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter1 = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter1.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

characterTextSplitter2 = CharacterTextSplitter(chunk_size = 300, chunk_overlap = 0, separator = ". ")

embeddingsRedundantFilter = EmbeddingsRedundantFilter(embeddings = openAIEmbeddings)

embeddingsFilter = EmbeddingsFilter(embeddings = openAIEmbeddings, similarity_threshold = 0.76)

documentCompressorPipeline = DocumentCompressorPipeline(transformers = [characterTextSplitter2, embeddingsRedundantFilter, embeddingsFilter])

contextualCompressionRetriever = ContextualCompressionRetriever(base_compressor = documentCompressorPipeline, base_retriever = vectorStoreRetriever)

resultDocumentList = contextualCompressionRetriever.invoke("What did the president say about Ketanji Jackson Brown")

printDocumentList(resultDocumentList)

[PYTHON/LANGCHAIN] ContextualCompressionRetriever 클래스 : EmbeddingsFilter 객체를 사용해 컨텍스트 압축 검색하기

■ ContextualCompressionRetriever 클래스에서 EmbeddingsFilter 객체를 사용해 컨텍스트 압축 검색하는 방법을 보여준다. ※ OPENAI_API_KEY 환경 변수 값은 .env 파일에 정의한다. ▶ main.py


from dotenv                                    import load_dotenv
from langchain_community.document_loaders      import TextLoader
from langchain_text_splitters                  import CharacterTextSplitter
from langchain_openai                          import OpenAIEmbeddings
from langchain_community.vectorstores          import FAISS
from langchain.retrievers.document_compressors import EmbeddingsFilter
from langchain.retrievers                      import ContextualCompressionRetriever

load_dotenv()

def printDocumentList(documentList):
    print(
        f"\n{'-' * 100}\n".join(
            [f"Document {i + 1} :\n\n" + document.page_content for i, document in enumerate(documentList)]
        )
    )

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

embeddingsFilter = EmbeddingsFilter(embeddings = openAIEmbeddings, similarity_threshold = 0.76)

contextualCompressionRetriever = ContextualCompressionRetriever(base_compressor = embeddingsFilter, base_retriever = vectorStoreRetriever)

resultDocumentList = contextualCompressionRetriever.invoke("What did the president say about Ketanji Jackson Brown")

printDocumentList(resultDocumentList)

from dotenv import load_dotenv

from langchain_community.document_loaders import TextLoader

from langchain_text_splitters import CharacterTextSplitter

from langchain_openai import OpenAIEmbeddings

from langchain_community.vectorstores import FAISS

from langchain.retrievers.document_compressors import EmbeddingsFilter

from langchain.retrievers import ContextualCompressionRetriever

load_dotenv()

def printDocumentList(documentList):

print(

f"\n{'-' * 100}\n".join(

[f"Document {i + 1} :\n\n" + document.page_content for i, document in enumerate(documentList)]

)

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

embeddingsFilter = EmbeddingsFilter(embeddings = openAIEmbeddings, similarity_threshold = 0.76)

contextualCompressionRetriever = ContextualCompressionRetriever(base_compressor = embeddingsFilter, base_retriever = vectorStoreRetriever)

resultDocumentList = contextualCompressionRetriever.invoke("What did the president say about Ketanji Jackson Brown")

printDocumentList(resultDocumentList)

[PYTHON/LANGCHAIN] FAISS 클래스 : as_retriever 메소드에서 search_kwargs 인자를 사용해 벡터 스토어 검색기 구하기 2

■ FAISS 클래스의 as_retriever 메소드에서 search_kwargs 인자를 사용해 벡터 스토어 검색기를 구하는 방법올 보여준다. ※ OPENAI_API_KEY 환경 변수 값은 .env 파일에 정의한다.

[PYTHON/LANGCHAIN] ContextualCompressionRetriever 클래스 : LLMListwiseRerank 객체를 사용해 컨텍스트 압축 검색하기

■ ContextualCompressionRetriever 클래스에서 LLMListwiseRerank 객체를 사용해 컨텍스트 압축 검색하는 방법을 보여준다. ※ OPENAI_API_KEY 환경 변수 값은 .env 파일에 정의한다. ▶ main.py


from dotenv                                    import load_dotenv
from langchain_community.document_loaders      import TextLoader
from langchain_text_splitters                  import CharacterTextSplitter
from langchain_openai                          import OpenAIEmbeddings
from langchain_community.vectorstores          import FAISS
from langchain_openai                          import ChatOpenAI
from langchain.retrievers.document_compressors import LLMListwiseRerank
from langchain.retrievers                      import ContextualCompressionRetriever

load_dotenv()

def printDocumentList(documentList):
    print(f"\n{'-' * 100}\n".join([f"Document {index + 1}:\n\n" + document.page_content for index, document in enumerate(documentList)]))

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

chatOpenAI = ChatOpenAI(model = "gpt-3.5-turbo-0125", temperature = 0)

llmListwiseRerank = LLMListwiseRerank.from_llm(chatOpenAI, top_n = 1)

contextualCompressionRetriever = ContextualCompressionRetriever(base_compressor = llmListwiseRerank, base_retriever = vectorStoreRetriever)

compressedDocumentList = contextualCompressionRetriever.invoke("What did the president say about Ketanji Jackson Brown")

printDocumentList(compressedDocumentList)

"""
Document 1:

Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections.

Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.

One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.

And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.
"""

from dotenv import load_dotenv

from langchain_community.document_loaders import TextLoader

from langchain_text_splitters import CharacterTextSplitter

from langchain_openai import OpenAIEmbeddings

from langchain_community.vectorstores import FAISS

from langchain_openai import ChatOpenAI

from langchain.retrievers.document_compressors import LLMListwiseRerank

from langchain.retrievers import ContextualCompressionRetriever

load_dotenv()

def printDocumentList(documentList):

print(f"\n{'-' * 100}\n".join([f"Document {index + 1}:\n\n" + document.page_content for index, document in enumerate(documentList)]))

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

chatOpenAI = ChatOpenAI(model = "gpt-3.5-turbo-0125", temperature = 0)

llmListwiseRerank = LLMListwiseRerank.from_llm(chatOpenAI, top_n = 1)

contextualCompressionRetriever = ContextualCompressionRetriever(base_compressor = llmListwiseRerank, base_retriever = vectorStoreRetriever)

compressedDocumentList = contextualCompressionRetriever.invoke("What did the president say about Ketanji Jackson Brown")

printDocumentList(compressedDocumentList)

"""

Document 1:

Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections.

Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.

One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.

And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.

"""

[PYTHON/LANGCHAIN] ContextualCompressionRetriever 클래스 : LLMChainFilter 객체를 사용해 컨텍스트 압축 검색하기

■ ContextualCompressionRetriever 클래스에서 LLMChainFilter 객체를 사용해 컨텍스트 압축 검색하는 방법을 보여준다. ※ OPENAI_API_KEY 환경 변수 값은 .env 파일에 정의한다. ▶ main.py


from dotenv                                    import load_dotenv
from langchain_community.document_loaders      import TextLoader
from langchain_text_splitters                  import CharacterTextSplitter
from langchain_openai                          import OpenAIEmbeddings
from langchain_community.vectorstores          import FAISS
from langchain_openai                          import OpenAI
from langchain.retrievers.document_compressors import LLMChainFilter
from langchain.retrievers                      import ContextualCompressionRetriever

load_dotenv()

def printDocumentList(documentList):
    print(f"\n{'-' * 100}\n".join([f"Document {index + 1}:\n\n" + document.page_content for index, document in enumerate(documentList)]))

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

openAI = OpenAI(temperature = 0)

llmChainFilter = LLMChainFilter.from_llm(openAI)

contextualCompressionRetriever = ContextualCompressionRetriever(base_compressor = llmChainFilter, base_retriever = vectorStoreRetriever)

compressedDocumentList = contextualCompressionRetriever.invoke("What did the president say about Ketanji Jackson Brown")

printDocumentList(compressedDocumentList)

"""
Document 1:

Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections.

Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.

One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.

And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.
"""

from dotenv import load_dotenv

from langchain_community.document_loaders import TextLoader

from langchain_text_splitters import CharacterTextSplitter

from langchain_openai import OpenAIEmbeddings

from langchain_community.vectorstores import FAISS

from langchain_openai import OpenAI

from langchain.retrievers.document_compressors import LLMChainFilter

from langchain.retrievers import ContextualCompressionRetriever

load_dotenv()

def printDocumentList(documentList):

print(f"\n{'-' * 100}\n".join([f"Document {index + 1}:\n\n" + document.page_content for index, document in enumerate(documentList)]))

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

openAI = OpenAI(temperature = 0)

llmChainFilter = LLMChainFilter.from_llm(openAI)

contextualCompressionRetriever = ContextualCompressionRetriever(base_compressor = llmChainFilter, base_retriever = vectorStoreRetriever)

compressedDocumentList = contextualCompressionRetriever.invoke("What did the president say about Ketanji Jackson Brown")

printDocumentList(compressedDocumentList)

"""

Document 1:

One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.

And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.

"""

[PYTHON/LANGCHAIN] ContextualCompressionRetriever 클래스 : 생성자에서 base_compressor/base_retriever 인자를 사용해 ContextualCompressionRetriever 객체 만들기

■ ContextualCompressionRetriever 클래스의 생성자에서 base_compressor/base_retriever 인자를 사용해 ContextualCompressionRetriever 객체를 만드는 방법을 보여준다. ※ OPENAI_API_KEY 환경 변수 값은 .env 파일에 정의한다. ▶ main.py

[PYTHON/LANGCHAIN] ContextualCompressionRetriever 클래스 : LLMChainExtractor 객체를 사용해 컨텍스트 압축 검색하기

■ ContextualCompressionRetriever 클래스에서 LLMChainExtractor 객체를 사용해 컨텍스트 압축 검색하는 방법을 보여준다. ※ OPENAI_API_KEY 환경 변수 값은 .env 파일에 정의한다. ▶ main.py


from dotenv                                    import load_dotenv
from langchain_community.document_loaders      import TextLoader
from langchain_text_splitters                  import CharacterTextSplitter
from langchain_openai                          import OpenAIEmbeddings
from langchain_community.vectorstores          import FAISS
from langchain_openai                          import OpenAI
from langchain.retrievers.document_compressors import LLMChainExtractor
from langchain.retrievers                      import ContextualCompressionRetriever

load_dotenv()

def printDocumentList(documentList):
    print(f"\n{'-' * 100}\n".join([f"Document {index + 1}:\n\n" + document.page_content for index, document in enumerate(documentList)]))

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

openAI = OpenAI(temperature = 0)

llmChainExtractor = LLMChainExtractor.from_llm(openAI)

contextualCompressionRetriever = ContextualCompressionRetriever(base_compressor = llmChainExtractor, base_retriever = vectorStoreRetriever)

compressedDocumentList = contextualCompressionRetriever.invoke("What did the president say about Ketanji Jackson Brown")

printDocumentList(compressedDocumentList)

"""
Document 1:

I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson.
"""

from dotenv import load_dotenv

from langchain_community.document_loaders import TextLoader

from langchain_text_splitters import CharacterTextSplitter

from langchain_openai import OpenAIEmbeddings

from langchain_community.vectorstores import FAISS

from langchain_openai import OpenAI

from langchain.retrievers.document_compressors import LLMChainExtractor

from langchain.retrievers import ContextualCompressionRetriever

load_dotenv()

def printDocumentList(documentList):

print(f"\n{'-' * 100}\n".join([f"Document {index + 1}:\n\n" + document.page_content for index, document in enumerate(documentList)]))

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

openAI = OpenAI(temperature = 0)

llmChainExtractor = LLMChainExtractor.from_llm(openAI)

contextualCompressionRetriever = ContextualCompressionRetriever(base_compressor = llmChainExtractor, base_retriever = vectorStoreRetriever)

compressedDocumentList = contextualCompressionRetriever.invoke("What did the president say about Ketanji Jackson Brown")

printDocumentList(compressedDocumentList)

"""

Document 1:

I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson.

"""

[PYTHON/LANGCHAIN] VectorStoreRetriever 클래스 : invoke 메소드 사용하기 2

■ VectorStoreRetriever 클래스의 invoke 메소드를 사용하는 방법을 보여준다. ※ OPENAI_API_KEY 환경 변수 값은 .env 파일에 정의한다. ▶ main.py


from dotenv                               import load_dotenv
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters             import CharacterTextSplitter
from langchain_openai                     import OpenAIEmbeddings
from langchain_community.vectorstores     import FAISS

load_dotenv()

def printDocumentList(documentList):
    print(f"\n{'-' * 100}\n".join([f"Document {index + 1}:\n\n" + document.page_content for index, document in enumerate(documentList)]))

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

resultDocumentList = vectorStoreRetriever.invoke("What did the president say about Ketanji Brown Jackson")

printDocumentList(resultDocumentList)

"""
Document 1:

Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections.

Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.

One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.

And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.

from dotenv import load_dotenv

from langchain_community.document_loaders import TextLoader

from langchain_text_splitters import CharacterTextSplitter

from langchain_openai import OpenAIEmbeddings

from langchain_community.vectorstores import FAISS

load_dotenv()

def printDocumentList(documentList):

print(f"\n{'-' * 100}\n".join([f"Document {index + 1}:\n\n" + document.page_content for index, document in enumerate(documentList)]))

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

resultDocumentList = vectorStoreRetriever.invoke("What did the president say about Ketanji Brown Jackson")

printDocumentList(resultDocumentList)

"""

Document 1:

One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.

And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.

Document 2: A

[PYTHON/LANGCHAIN] FAISS 클래스 : as_retriever 메소드에서 search_type/search_kwargs 인자를 사용해 벡터 스토어 검색기 구하기

■ FAISS 클래스의 as_retriever 메소드에서 search_type/search_kwargs 인자를 사용해 벡터 스토어 검색기를 구하는 방법을 보여준다. ※ 유사도 점수 임계값을 설정하여 해당 임계값보다 높은

[PYTHON/LANGCHAIN] FAISS 클래스 : as_retriever 메소드에서 search_type 인자를 사용해 벡터 스토어 검색기 구하기

■ FAISS 클래스의 as_retriever 메소드에서 search_type 인자를 사용해 벡터 스토어 검색기를 구하는 방법을 보여준다. ※ 기본적으로 벡터 스토어 검색기는 유사성 검색을 사용한다.

[PYTHON/LANGCHAIN] VectorStoreRetriever 클래스 : invoke 메소드 사용하기 1

■ VectorStoreRetriever 클래스의 invoke 메소드를 사용하는 방법을 보여준다. ※ OPENAI_API_KEY 환경 변수 값은 .env 파일에 정의한다. ▶ main.py


from dotenv import load_dotenv

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters             import CharacterTextSplitter
from langchain_openai                     import OpenAIEmbeddings
from langchain_community.vectorstores     import FAISS

load_dotenv()

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

responseDocumentList = vectorStoreRetriever.invoke("what did the president say about ketanji brown jackson?")

print(len(responseDocumentList))

print(responseDocumentList[0].page_content)

"""
4
Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections.

Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.

One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.

And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.
"""

from dotenv import load_dotenv

from langchain_community.document_loaders import TextLoader

from langchain_text_splitters import CharacterTextSplitter

from langchain_openai import OpenAIEmbeddings

from langchain_community.vectorstores import FAISS

load_dotenv()

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter.split_documents(documentList)

openAIEmbeddings = OpenAIEmbeddings()

faiss = FAISS.from_documents(splitDocumentList, openAIEmbeddings)

vectorStoreRetriever = faiss.as_retriever()

responseDocumentList = vectorStoreRetriever.invoke("what did the president say about ketanji brown jackson?")

print(len(responseDocumentList))

print(responseDocumentList[0].page_content)

"""

One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.

And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.

"""

▶ requirements.txt


aiohappyeyeballs==2.3.5
aiohttp==3.10.3
aiosignal==1.3.1
annotated-types==0.7.0
anyio==4.4.0
async-timeout==4.0.3
attrs==24.2.0
certifi==2024.7.4
charset-normalizer==3.3.2
dataclasses-json==0.6.7
distro==1.9.0
exceptiongroup==1.2.2
faiss-gpu==1.7.2
frozenlist==1.4.1
greenlet==3.0.3
h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
idna==3.7
jiter==0.5.0
jsonpatch==1.33
jsonpointer==3.0.0
langchain==0.2.12
langchain-community==0.2.11
langchain-core==0.2.29
langchain-openai==0.1.21
langchain-text-splitters==0.2.2
langsmith==0.1.98
marshmallow==3.21.3
multidict==6.0.5
mypy-extensions==1.0.0
numpy==1.26.4
openai==1.40.3
orjson==3.10.7
packaging==24.1
pydantic==2.8.2
pydantic_core==2.20.1
python-dotenv==1.0.1
PyYAML==6.0.2
regex==2024.7.24
requests==2.32.3
sniffio==1.3.1
SQLAlchemy==2.0.32
tenacity==8.5.0
tiktoken==0.7.0
tqdm==4.66.5
typing-inspect==0.9.0
typing_extensions==4.12.2
urllib3==2.2.2
yarl==1.9.4

aiohappyeyeballs==2.3.5

aiohttp==3.10.3

aiosignal==1.3.1

annotated-types==0.7.0

anyio==4.4.0

async-timeout==4.0.3

attrs==24.2.0

certifi==2024.7.4

charset-normalizer==3.3.2

dataclasses-json==0.6.7

distro==1.9.0

exceptiongroup==1.2.2

faiss-gpu==1.7.2

frozenlist==1.4.1

greenlet==3.0.3

h11==0.14.0

httpcore==1.0.5

httpx==0.27.0

idna==3.7

jiter==0.5.0

jsonpatch==1.33

jsonpointer==3.0.0

langchain==0.2.12

langchain-community==0.2.11

langchain-core==0.2.29

langchain-openai==0.1.21

langchain-text-splitters==0.2.2

langsmith==0.1.98

marshmallow==3.21.3

multidict==6.0.5

mypy-extensions==1.0.0

numpy==1.26.4

openai==1.40.3

orjson==3.10.7

packaging==24.1

pydantic==2.8.2

pydantic_core==2.20.1

python-dotenv==1.0.1

PyYAML==6.0.2

regex==2024.7.24

requests==2.32.3

sniffio==1.3.1

SQLAlchemy==2.0.32

tenacity==8.5.0

tiktoken==0.7.0

tqdm==4.66.5

typing-inspect==0.9.0

typing_extensions==4.12.2

urllib3==2.2.2

yarl==1.9.4

[PYTHON/LANGCHAIN] FAISS 클래스 : as_retriever 메소드를 사용해 벡터 스토어 검색기 구하기

■ FAISS 클래스의 as_retriever 메소드를 사용해 벡터 스토어 검색기를 구하는 방법을 보여준다. ※ OPENAI_API_KEY 환경 변수 값은 .env 파일에 정의한다. ▶ 예제

[PYTHON/LANGCHAIN] InMemoryByteStore 클래스 : 메모리 파일 저장소 만들기

■ InMemoryByteStore 클래스를 사용해 메모리 파일 저장소를 만드는 방법을 보여준다. ※ OPENAI_API_KEY 환경 변수 값은 .env 파일에 정의한다. ▶ main.py


from dotenv                               import load_dotenv
from langchain_openai                     import OpenAIEmbeddings
from langchain.storage                    import InMemoryByteStore
from langchain.embeddings                 import CacheBackedEmbeddings
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters             import CharacterTextSplitter
from langchain_community.vectorstores     import FAISS

load_dotenv()

openAIEmbeddings = OpenAIEmbeddings()

inMemoryByteStore = InMemoryByteStore()

cacheBackedEmbeddings = CacheBackedEmbeddings.from_bytes_store(openAIEmbeddings, inMemoryByteStore, namespace = openAIEmbeddings.model)

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter.split_documents(documentList)

list1 = list(inMemoryByteStore.yield_keys())

faiss = FAISS.from_documents(splitDocumentList, cacheBackedEmbeddings)

list2 = list(inMemoryByteStore.yield_keys())

print(len(list1))
print(len(list2))

"""
0
42
"""

from dotenv import load_dotenv

from langchain_openai import OpenAIEmbeddings

from langchain.storage import InMemoryByteStore

from langchain.embeddings import CacheBackedEmbeddings

from langchain_community.document_loaders import TextLoader

from langchain_text_splitters import CharacterTextSplitter

from langchain_community.vectorstores import FAISS

load_dotenv()

openAIEmbeddings = OpenAIEmbeddings()

inMemoryByteStore = InMemoryByteStore()

cacheBackedEmbeddings = CacheBackedEmbeddings.from_bytes_store(openAIEmbeddings, inMemoryByteStore, namespace = openAIEmbeddings.model)

textLoader = TextLoader("state_of_the_union.txt")

documentList = textLoader.load()

characterTextSplitter = CharacterTextSplitter(chunk_size = 1000, chunk_overlap = 0)

splitDocumentList = characterTextSplitter.split_documents(documentList)

list1 = list(inMemoryByteStore.yield_keys())

faiss = FAISS.from_documents(splitDocumentList, cacheBackedEmbeddings)

list2 = list(inMemoryByteStore.yield_keys())

print(len(list1))

print(len(list2))

"""

▶

[태그:] FAISS

[PYTHON/LANGCHAIN] RunnableWithMessageHistory 클래스 : invoke 메소드를 사용해 채팅하기 1

[PYTHON/LANGCHAIN] create_retriever_tool 함수 : FAISS 벡터 스토어 검색 도구 만들기 2

[PYTHON/LANGCHAIN] FAISS 클래스 : from_documents 정적 메소드를 사용해 FAISS 객체 만들기

[PYTHON/LANGCHAIN] TimeWeightedVectorStoreRetriever 클래스 : mock_now 함수를 사용해 가상 시간 설정하기

[PYTHON/LANGCHAIN] TimeWeightedVectorStoreRetriever 클래스 : 높은 감쇠율로 문서 검색하기

[PYTHON/LANGCHAIN] TimeWeightedVectorStoreRetriever 클래스 : 낮은 감쇠율로 문서 검색하기

[PYTHON/LANGCHAIN] TimeWeightedVectorStoreRetriever 클래스 : 생성자에서 vectorstore/decay_rate 인자를 사용해 TimeWeightedVectorStoreRetriever 객체 만들기

[PYTHON/LANGCHAIN] VectorStoreRetriever 클래스 : configurable_fields 메소드에서 search_kwargs 인자에 ConfigurableField 객체 설정하기

[PYTHON/LANGCHAIN] EnsembleRetriever 클래스 : 생성자에서 retrievers/weights 인자를 사용해 EnsembleRetriever 객체 만들기

[PYTHON/LANGCHAIN] ContextualCompressionRetriever 클래스 : DocumentCompressorPipeline 객체를 사용해 컨텍스트 압축 검색하기

[PYTHON/LANGCHAIN] ContextualCompressionRetriever 클래스 : EmbeddingsFilter 객체를 사용해 컨텍스트 압축 검색하기

[PYTHON/LANGCHAIN] FAISS 클래스 : as_retriever 메소드에서 search_kwargs 인자를 사용해 벡터 스토어 검색기 구하기 2

[PYTHON/LANGCHAIN] ContextualCompressionRetriever 클래스 : LLMListwiseRerank 객체를 사용해 컨텍스트 압축 검색하기

[PYTHON/LANGCHAIN] ContextualCompressionRetriever 클래스 : LLMChainFilter 객체를 사용해 컨텍스트 압축 검색하기

[PYTHON/LANGCHAIN] ContextualCompressionRetriever 클래스 : 생성자에서 base_compressor/base_retriever 인자를 사용해 ContextualCompressionRetriever 객체 만들기

[PYTHON/LANGCHAIN] ContextualCompressionRetriever 클래스 : LLMChainExtractor 객체를 사용해 컨텍스트 압축 검색하기

[PYTHON/LANGCHAIN] VectorStoreRetriever 클래스 : invoke 메소드 사용하기 2

[PYTHON/LANGCHAIN] FAISS 클래스 : as_retriever 메소드에서 search_type/search_kwargs 인자를 사용해 벡터 스토어 검색기 구하기

[PYTHON/LANGCHAIN] FAISS 클래스 : as_retriever 메소드에서 search_type 인자를 사용해 벡터 스토어 검색기 구하기

[PYTHON/LANGCHAIN] VectorStoreRetriever 클래스 : invoke 메소드 사용하기 1

[PYTHON/LANGCHAIN] FAISS 클래스 : as_retriever 메소드를 사용해 벡터 스토어 검색기 구하기

[PYTHON/LANGCHAIN] InMemoryByteStore 클래스 : 메모리 파일 저장소 만들기

[PYTHON/LANGCHAIN] FAISS 클래스 : similarity_search 메소드를 사용해 문자열 쿼리 유사성 기반으로 문서 리스트 구하기

[PYTHON/LANGCHAIN] from_documents 함수 : 메모리 내 벡터 저장소를 위한 FAISS 객체 만들기

[PYTHON/LANGCHAIN] RunnableSequence 클래스 : get_prompts 메소드를 사용해 체인에서 사용하는 프롬프트 템플리트 리스트 구하기