[AUR-429] Add MVP pipeline with Ingestion and QA stage (#39)
* add base Tool * minor update test_tool * update test dependency * update test dependency * Fix namespace conflict * update test * add base Agent Interface, add ReWoo Agent * minor update * update test * fix typo * remove unneeded print * update rewoo agent * add LLMTool * update BaseAgent type * add ReAct agent * add ReAct agent * minor update * minor update * minor update * minor update * update base reader with BaseComponent * add splitter * update agent and tool * update vectorstores * update load/save for indexing and retrieving pipeline * update test_agent for more use-cases * add missing dependency for test * update test case for in memory vectorstore * add TextSplitter to BaseComponent * update type hint basetool * add insurance mvp pipeline * update requirements * Remove redundant plugins param * Mock GoogleSearch --------- Co-authored-by: trducng <trungduc1992@gmail.com>
This commit is contained in:
committed by
GitHub
parent
2638152054
commit
79cc60e6a2
17
knowledgehub/pipelines/utils.py
Normal file
17
knowledgehub/pipelines/utils.py
Normal file
@@ -0,0 +1,17 @@
|
||||
import hashlib
|
||||
from typing import List
|
||||
|
||||
|
||||
def filename_to_hash(filename: str) -> str:
|
||||
"""
|
||||
Convert filename to hash to be used as collection name for storage
|
||||
"""
|
||||
result = hashlib.md5(filename.encode())
|
||||
return result.hexdigest()
|
||||
|
||||
|
||||
def file_names_to_collection_name(file_name_list: List[str]) -> str:
|
||||
"""
|
||||
Convert list of filenames to collection name
|
||||
"""
|
||||
return filename_to_hash(" ".join(file_name_list))
|
Reference in New Issue
Block a user