[AUR-429] Add MVP pipeline with Ingestion and QA stage (#39)

* add base Tool

* minor update test_tool

* update test dependency

* update test dependency

* Fix namespace conflict

* update test

* add base Agent Interface, add ReWoo Agent

* minor update

* update test

* fix typo

* remove unneeded print

* update rewoo agent

* add LLMTool

* update BaseAgent type

* add ReAct agent

* add ReAct agent

* minor update

* minor update

* minor update

* minor update

* update base reader with BaseComponent

* add splitter

* update agent and tool

* update vectorstores

* update load/save for indexing and retrieving pipeline

* update test_agent for more use-cases

* add missing dependency for test

* update test case for in memory vectorstore

* add TextSplitter to BaseComponent

* update type hint basetool

* add insurance mvp pipeline

* update requirements

* Remove redundant plugins param

* Mock GoogleSearch

---------

Co-authored-by: trducng <trungduc1992@gmail.com>
This commit is contained in:
Tuan Anh Nguyen Dang (Tadashi_Cin)
2023-10-05 12:31:33 +07:00
committed by GitHub
parent 2638152054
commit 79cc60e6a2
9 changed files with 389 additions and 4 deletions

View File

@@ -0,0 +1,17 @@
import hashlib
from typing import List
def filename_to_hash(filename: str) -> str:
"""
Convert filename to hash to be used as collection name for storage
"""
result = hashlib.md5(filename.encode())
return result.hexdigest()
def file_names_to_collection_name(file_name_list: List[str]) -> str:
"""
Convert list of filenames to collection name
"""
return filename_to_hash(" ".join(file_name_list))