Refactor the `kotaemon/pipelines` module to `kotaemon/indices`. Create the VectorIndex.
Note: currently I place `qa` to be inside `kotaemon/indices` since at the moment we only have `qa` in RAG. At the same time, I think `qa` can be an independent module in `kotaemon/qa`. Since this can be changed later, I still go at the 1st option for now to observe if we can change it later.
* enforce Document as IO
* Separate rerankers, splitters and extractors (#85)
* partially refractor importing
* add text to embedding outputs
---------
Co-authored-by: Nguyen Trung Duc (john) <trungduc1992@gmail.com>
* add rerankers in retrieving pipeline
* update example MVP pipeline
* add citation pipeline and function call interface
* change return type of QA and AgentPipeline to Document
* Move splitter into indexing module
* Rename post_processing module to parsers
* Migrate LLM-specific composite pipelines into llms module
This change moves the `splitters` module into `indexing` module. The `indexing` module will be created soon, to house `indexing`-related components.
This change renames `post_processing` module into `parsers` module. Post-processing is a generic term which provides very little information. In the future, we will add other extractors into the `parser` module, like Metadata extractor...
This change migrates the composite elements into `llms` module. These elements heavily assume that the internal nodes are llm-specific. As a result, migrating these elements into `llms` module will make them more discoverable, and simplify code base structure.
Since the only usage of prompt is within LLMs, it is reasonable to keep it within the LLM module. This way, it would be easier to discover module, and make the code base less complicated.
Changes:
* Move prompt components into llms
* Bump version 0.3.1
* Make pip install dependencies in eager mode
---------
Co-authored-by: ian <ian@cinnamon.is>
This change speeds up OCR extraction by allowing bypassing OCR for texts that are irrelevant (not in table).
---------
Co-authored-by: Nguyen Trung Duc (john) <trungduc1992@gmail.com>
This change remove `BaseComponent`'s:
- run_raw
- run_batch_raw
- run_document
- run_batch_document
- is_document
- is_batch
Each component is expected to support multiple types of inputs and a single type of output. Since we want the component to work out-of-the-box with both standardized and customized use cases, supporting multiple types of inputs are expected. At the same time, to reduce the complexity of understanding how to use a component, we restrict a component to only have a single output type.
To accommodate these changes, we also refactor some components to remove their run_raw, run_batch_raw... methods, and to decide the common output interface for those components.
Tests are updated accordingly.
Commit changes:
* Add kwargs to vector store's query
* Simplify the BaseComponent
* Update tests
* Remove support for Python 3.8 and 3.9
* Bump version 0.3.0
* Fix github PR caching still use old environment after bumping version
---------
Co-authored-by: ian <ian@cinnamon.is>
By allowing specifying the UI outputs in the code, any time user runs `kh export ...`, that outputs in the code will be included in the UI YAML file. Otherwise, any time the user runs `kh export ...`, the output section in the UI YAML file will be reset to the default output.
* add base Tool
* minor update test_tool
* update test dependency
* update test dependency
* Fix namespace conflict
* update test
* add base Agent Interface, add ReWoo Agent
* minor update
* update test
* fix typo
* remove unneeded print
* update rewoo agent
* add LLMTool
* update BaseAgent type
* add ReAct agent
* add ReAct agent
* minor update
* minor update
* minor update
* minor update
* update base reader with BaseComponent
* add splitter
* update agent and tool
* update vectorstores
* update load/save for indexing and retrieving pipeline
* update test_agent for more use-cases
* add missing dependency for test
* update test case for in memory vectorstore
* add TextSplitter to BaseComponent
* update type hint basetool
---------
Co-authored-by: trducng <trungduc1992@gmail.com>
* add test case for Chroma save/load
* minor name change
* add delete_collection support for chroma
* move save load to chroma
---------
Co-authored-by: Nguyen Trung Duc (john) <john@cinnamon.is>
This CL implements:
- The logic to export log to Excel.
- Route the export logic in the UI.
- Demonstrate this functionality in `./examples/promptui` project.
From pipeline > config > UI. Provide example project for promptui
- Pipeline to config: `kotaemon.contribs.promptui.config.export_pipeline_to_config`. The config follows schema specified in this document: https://cinnamon-ai.atlassian.net/wiki/spaces/ATM/pages/2748711193/Technical+Detail. Note: this implementation exclude the logs, which will be handled in AUR-408.
- Config to UI: `kotaemon.contribs.promptui.build_from_yaml`
- Example project is located at `examples/promptui/`
* [AUR-362] Add In-memory vector store
* [AUR-362] fix delete fun input format
* [AUR-362] revise persist and from persist path to save and load
* [AUR-362] revise simple.py to in_memory.py
Document store handles storing and indexing Documents. It supports the following interfaces:
- add: add 1 or more documents into document store
- get: get a list of documents
- get_all: get all documents in a document store
- delete: delete 1 or more document
- save: persist a document store into disk
- load: load a document store from disk
