* enforce Document as IO
* Separate rerankers, splitters and extractors (#85)
* partially refractor importing
* add text to embedding outputs
---------
Co-authored-by: Nguyen Trung Duc (john) <trungduc1992@gmail.com>
* add rerankers in retrieving pipeline
* update example MVP pipeline
* add citation pipeline and function call interface
* change return type of QA and AgentPipeline to Document
* Move splitter into indexing module
* Rename post_processing module to parsers
* Migrate LLM-specific composite pipelines into llms module
This change moves the `splitters` module into `indexing` module. The `indexing` module will be created soon, to house `indexing`-related components.
This change renames `post_processing` module into `parsers` module. Post-processing is a generic term which provides very little information. In the future, we will add other extractors into the `parser` module, like Metadata extractor...
This change migrates the composite elements into `llms` module. These elements heavily assume that the internal nodes are llm-specific. As a result, migrating these elements into `llms` module will make them more discoverable, and simplify code base structure.
Since the only usage of prompt is within LLMs, it is reasonable to keep it within the LLM module. This way, it would be easier to discover module, and make the code base less complicated.
Changes:
* Move prompt components into llms
* Bump version 0.3.1
* Make pip install dependencies in eager mode
---------
Co-authored-by: ian <ian@cinnamon.is>
This change speeds up OCR extraction by allowing bypassing OCR for texts that are irrelevant (not in table).
---------
Co-authored-by: Nguyen Trung Duc (john) <trungduc1992@gmail.com>
This change remove `BaseComponent`'s:
- run_raw
- run_batch_raw
- run_document
- run_batch_document
- is_document
- is_batch
Each component is expected to support multiple types of inputs and a single type of output. Since we want the component to work out-of-the-box with both standardized and customized use cases, supporting multiple types of inputs are expected. At the same time, to reduce the complexity of understanding how to use a component, we restrict a component to only have a single output type.
To accommodate these changes, we also refactor some components to remove their run_raw, run_batch_raw... methods, and to decide the common output interface for those components.
Tests are updated accordingly.
Commit changes:
* Add kwargs to vector store's query
* Simplify the BaseComponent
* Update tests
* Remove support for Python 3.8 and 3.9
* Bump version 0.3.0
* Fix github PR caching still use old environment after bumping version
---------
Co-authored-by: ian <ian@cinnamon.is>
* add base Tool
* minor update test_tool
* update test dependency
* update test dependency
* Fix namespace conflict
* update test
* add base Agent Interface, add ReWoo Agent
* minor update
* update test
* fix typo
* remove unneeded print
* update rewoo agent
* add LLMTool
* update BaseAgent type
* add ReAct agent
* add ReAct agent
* minor update
* minor update
* minor update
* minor update
* update base reader with BaseComponent
* add splitter
* update agent and tool
* update vectorstores
* update load/save for indexing and retrieving pipeline
* update test_agent for more use-cases
* add missing dependency for test
* update test case for in memory vectorstore
* add TextSplitter to BaseComponent
* update type hint basetool
---------
Co-authored-by: trducng <trungduc1992@gmail.com>
From pipeline > config > UI. Provide example project for promptui
- Pipeline to config: `kotaemon.contribs.promptui.config.export_pipeline_to_config`. The config follows schema specified in this document: https://cinnamon-ai.atlassian.net/wiki/spaces/ATM/pages/2748711193/Technical+Detail. Note: this implementation exclude the logs, which will be handled in AUR-408.
- Config to UI: `kotaemon.contribs.promptui.build_from_yaml`
- Example project is located at `examples/promptui/`
Design the base interface of vector store, and apply it to the Chroma Vector Store (wrapped around llama_index's implementation). Provide the pipelines to populate and retrieve from vector store.
