Move LLM-related components into LLM module (#74)

Browse Source

* Move splitter into indexing module
* Rename post_processing module to parsers
* Migrate LLM-specific composite pipelines into llms module

This change moves the `splitters` module into `indexing` module. The `indexing` module will be created soon, to house `indexing`-related components.

This change renames `post_processing` module into `parsers` module. Post-processing is a generic term which provides very little information. In the future, we will add other extractors into the `parser` module, like Metadata extractor...

This change migrates the composite elements into `llms` module. These elements heavily assume that the internal nodes are llm-specific. As a result, migrating these elements into `llms` module will make them more discoverable, and simplify code base structure.

...

This commit is contained in:

Nguyen Trung Duc (john)

2023-11-15 16:26:53 +07:00

committed by GitHub

parent 9945afdf6f

commit f8b8d86d4e

13 changed files with 41 additions and 35 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

0

knowledgehub/indexing/__init__.py Normal file