Public/kotaemon
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
315 Commits 1 Branch 0 Tags 15 MiB
6f4acc979c
Commit Graph

177 Commits

Author SHA1 Message Date
cin-klein
66e565649e
feat: integrate nano-graphrag (#433) 
* add nano graph-rag

* ignore entities for relevant context reference

* refactor and add local model as default nano-graphrag

* feat: add kotaemon llm & embedding integration with nanographrag

* fix: add env var for nano GraphRAG

---------

Co-authored-by: Tadashi <tadashi@cinnamon.is>
 2024-10-30 15:32:30 +07:00
Khoi-Nguyen Nguyen-Ngoc
19b386b51e
fix: pin python-multipart version to avoid yanking issues with micropip (#436) 2024-10-28 15:13:47 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
f2f192ed72
feat: add toggle dark mode button on main Chat UI (#423) 
* feat: add toggle dark mode button on main UI

* docs: update docs
 2024-10-22 18:48:18 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
764fe595f4
feat: add file grouping feature (#416) bump:patch 2024-10-21 12:47:18 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
e6fa1af404
feat: add mindmap visualization (#405) bump:minor 2024-10-17 14:35:28 +07:00
a652
4764b0e82a
fix: update adobe_loader (#399) bump:patch 
* Update adobe_loader.py

fix:When initializing the Document, extra_info was not added to the metadata.

* Update adobe_loader.py

Change the method of checking whether extra_info is None.
 2024-10-16 11:01:55 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
b113efc855
feat: add web URL loader & refine indexing logics (#397) 
* feat: add web URL loader & refine indexing logics

* fix: comfort mypy
 2024-10-15 22:42:24 +07:00
ronchengang
8188760f32
feat: allow to use customized GraphRAG settings.yaml (#387) bump:patch 
* allow to use customized GraphRAG settings.yaml

* adjust import style

* fix typo

* Added GraphRAG original documentation reference.

* feat: allow to use customized GraphRAG settings.yaml
(#387)

---------

Co-authored-by: Chen, Ron Gang <git@git.com>
 2024-10-14 21:18:34 +07:00
Tadashi
f0f3b4b23e
docs: update README #none 2024-10-14 10:16:43 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
15c7916ad8
fix: improve GRAPHRAG key passing (#384) #none 2024-10-11 12:01:06 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
6da9db489f
fix: add optional graphrag toggle in dockerfile (#377) 
* fix: toggle graphrag install in Docker build

* fix: update Dockerfile

* fix: remove unused logics in chat_fn

* fix: disable duckduckgo test due to API limit
 2024-10-10 16:09:57 +07:00
Tadashi
3ff6af8acf
fix: optimize chat suggestion logic 2024-10-10 14:44:50 +07:00
ronchengang
ad34395d0b
update output path logic since GraphRAG has changed the storage config value in the latest release (#374) bump:patch 
Co-authored-by: Chen, Ron Gang <git@git.com>
 2024-10-10 11:20:02 +07:00
KennyWu
49a083fd9f
feat: tweak the 'Chat suggestion' feature to tie it to conversations (#341) #none 
Signed-off-by: Kennywu <jdlow@live.cn>
 2024-10-10 11:02:04 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
dfd00fe752
fix: vastly improve chat UI responsiveness by reordering Gradio events (#360) bump:patch 2024-10-04 17:15:49 +07:00
taprosoft
76ab3fdd90 fix: check empty Cohere key in rerank 2024-10-01 09:37:09 +00:00
Tadashi
a424a630f2
fix: pass .env.example to Docker and release package bump:patch 2024-10-01 14:49:57 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
7ac8f0329a
fix: improve Download UI bump:minor (#352) 
* fix: rerank test ui

* fix: improve download all UI
 2024-10-01 12:03:19 +07:00
KennyWu
53530e296f
feat: support TEI embedding service, configurable reranking model (#287) 
* feat: add support for TEI embedding service, allow reranking model to be configurable.

Signed-off-by: Kennywu <jdlow@live.cn>

* fix: add cohere default reranking model

* fix: comfort pre-commit

---------

Signed-off-by: Kennywu <jdlow@live.cn>
Co-authored-by: wujiaye <wujiaye@bluemoon.com.cn>
Co-authored-by: Tadashi <tadashi@cinnamon.is>
 2024-09-30 22:00:00 +07:00
Mikhail Khludnev
2e3c17b256
fix: convert graphrag input path to str (#237) #none 
I noticed type cast error in pycharm debug (it intercept forking programs). Anyway, this change obeys types. Although, this conversions happens implicitly.

Co-authored-by: Tadashi <tadashi@cinnamon.is>
 2024-09-29 23:02:23 +07:00
Pedro Lima
aac6233412
feat: button to delete all files in index (#320) #none 
* button to delete all files in index

* code formatting

---------

Co-authored-by: Tadashi <tadashi@cinnamon.is>
 2024-09-29 22:55:51 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
b7e81e61dd
fix: remove duplicated deps (#344) #none 2024-09-29 22:38:02 +07:00
Ben Dykstra
f7b6f313b5
fix: update setup instructions (#144) #none 
* activate directory to gitignore

* add my custom env to gitignore, will have to change that

* add unstructured to kotaemon pyproject.toml

* add .env to gitignore

* remove .env from tracking

* make changes to the run_macos script, update readme with more detailed instructions

* remove my personal changes from gitignore

* remove line from run_macos script

* remove option for not installing miniconda for non technical users, mark docker dependency as optional

* docs: update demo URL

* gitignore changes

* merge .env.example

* revert changes to run_macos.sh

* unstructured to advanced dependencies

* add link to unstructured system dependencies

* remove api key

* fix: skip tests when unstructured pdf not installed

* chore: loosen unstructured package version in pyproject.toml

* chore: correct syntax

---------

Co-authored-by: Tadashi <tadashi@cinnamon.is>
Co-authored-by: cin-albert <albert@cinnamon.is>
 2024-09-29 22:26:02 +07:00
taprosoft
00df123309 fix: fix vectorstore tests #none 2024-09-27 04:18:11 +00:00
saidmukhamad
94cc3a96c2
fix: add langchain google dependency (#329) 
* add-gemini-deps

* uncomment gemeni flow settings
 2024-09-27 11:15:42 +07:00
Tadashi
79b309396b
fix: update default cohere embedding models bump:patch 2024-09-25 11:10:09 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
88d577b0cc
feat: add first setup screen for LLM & Embedding models (#314) (bump:minor) 
* fix: utf-8 txt reader

* fix: revise vectorstore import and make it optional

* feat: add cohere chat model with tool call support

* fix: simplify citation pipeline

* fix: improve citation logic

* fix: improve decompose func call

* fix: revise question rewrite prompt

* fix: revise chat box default placeholder

* fix: add key from ktem to cohere rerank

* fix: conv name suggestion

* fix: ignore default key cohere rerank

* fix: improve test connection UI

* fix: reorder requirements

* feat: add first setup screen

* fix: update requirements

* fix: vectorstore tests

* fix: update cohere version

* fix: relax langchain core version

* fix: add demo mode

* fix: update flowsettings

* fix: typo

* fix: fix bool env passing
 2024-09-22 16:32:23 +07:00
Khoi-Nguyen Nguyen-Ngoc
a865e2b095
feat: modify base dependencies + remove unnecessary packages in lite docker (#310) 
* feat: update base/adv dependencies

* feat: update Dockerfile

* ci: update free disk for docker build
 2024-09-21 12:11:58 +07:00
Quang (Albert)
7762190d05
feat: add local theme (#288) 
* feat: add local theme instead of from hub

* chore: add credit

* fix: typo
 2024-09-17 19:03:39 +07:00
Anush
e2bd78e9c4
feat: Qdrant vectorstore support (#260) 
* feat: Qdrant vectorstore support

* chore: review changes

* docs: Updated README.md
 2024-09-16 04:17:36 +07:00
kan_cin
d3fd75297f
feat: add multi-stages docker and support platform arm (#274) 
* feat: add multi-stages docker and support platform arm

* refactor: pre-commit

* fix: raise ImportError (fastembed) instead of auto install

* feat: add dependencies for local llm

* feat: free disk

* feat: update README

* feat: update README

* chore: fix typo

---------

Co-authored-by: cin-niko <niko@cinnamon.is>
 2024-09-12 20:25:03 +07:00
mst
73a476979e
fix: change column type to string for relation_type (#272) #none 2024-09-11 20:47:03 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
96d2086017
fix: add guidance parameters for LC wrapper models (#255) 
* fix: add docstring to LC wrapper models

* fix: fix metadata passing with LC embedding wrapper
 2024-09-09 14:15:34 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
b06c4777a3
fix: add PDFJS download to Windows setup (#249) 2024-09-08 21:22:01 +07:00
kan_cin
dbb6bb275f
feat: add test connection for edit spec (#239) 2024-09-08 10:55:13 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
069f0f3c83
feat: expose Cohere and HF embedding support on UI (#236) 2024-09-06 18:18:19 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
ef7e91fcae
fix: update requirements (#230) 2024-09-06 09:36:21 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
e2ed3564ce
fix: limit fastapi version (#229) 2024-09-06 09:23:26 +07:00
Tadashi
318895b287
fix: disable default install for anthropic 2024-09-05 23:18:53 +07:00
Tadashi
3267e6c654
fix: disable default install for google-genai package 2024-09-05 23:08:28 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
05245f501c
feat: add support for Gemini, Claude through Langchain (#225) (bump:patch) 2024-09-05 21:58:20 +07:00
ChengZi
772186b6e5
feat: support milvus vector db (#188) #none 
Signed-off-by: ChengZi <chen.zhang@zilliz.com>
 2024-09-04 20:22:50 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
76f2652d2a
fix: re-enable tests and fix legacy test interface (#208) 
* fix: re-enable tests and fix legacy test interface

* fix: skip llamacpp based on installed status

* fix: minor fix
 2024-09-04 12:37:39 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
607867d7e6
feat: add markdown file support (#202) 
* feat: add support for .md

* fix: disable download all on private collection
 2024-09-03 23:15:26 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
35b2927e5c
fix: update app version resolver in flowsettings (#180) (bump:patch) 2024-09-02 17:42:39 +07:00
kan_cin
041d229282
feat: add test connection feature (#166) 
* feat: add test connection feature

* fix: typo
 2024-09-01 08:22:36 +07:00
Quang (Albert)
4b2b334d2c
fix: refine kotaemon/pyproject.toml (#153) 2024-08-30 23:02:14 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
d880294153
fix: pwd change in setttings (#147) 2024-08-29 13:41:12 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
bb56ef4f8e
chore: update workflow (#124) 2024-08-26 09:52:16 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
2570e11501
feat: merge develop (#123) 
* Support hybrid vector retrieval

* Enable figures and table reading in Azure DI

* Retrieve with multi-modal

* Fix mixing up table

* Add txt loader

* Add Anthropic Chat

* Raising error when retrieving help file

* Allow same filename for different people if private is True

* Allow declaring extra LLM vendors

* Show chunks on the File page

* Allow elasticsearch to get more docs

* Fix Cohere response (#86)

* Fix Cohere response

* Remove Adobe pdfservice from dependency

kotaemon doesn't rely more pdfservice for its core functionality,
and pdfservice uses very out-dated dependency that causes conflict.

---------

Co-authored-by: trducng <trungduc1992@gmail.com>

* Add confidence score (#87)

* Save question answering data as a log file

* Save the original information besides the rewritten info

* Export Cohere relevance score as confidence score

* Fix style check

* Upgrade the confidence score appearance (#90)

* Highlight the relevance score

* Round relevance score. Get key from config instead of env

* Cohere return all scores

* Display relevance score for image

* Remove columns and rows in Excel loader which contains all NaN (#91)

* remove columns and rows which contains all NaN

* back to multiple joiner options

* Fix style

---------

Co-authored-by: linhnguyen-cinnamon <cinmc0019@CINMC0019-LinhNguyen.local>
Co-authored-by: trducng <trungduc1992@gmail.com>

* Track retriever state

* Bump llama-index version 0.10

* feat/save-azuredi-mhtml-to-markdown (#93)

* feat/save-azuredi-mhtml-to-markdown

* fix: replace os.path to pathlib change theflow.settings

* refactor: base on pre-commit

* chore: move the func of saving content markdown above removed_spans

---------

Co-authored-by: jacky0218 <jacky0218@github.com>

* fix: losing first chunk (#94)

* fix: losing first chunk.

* fix: update the method of preventing losing chunks

---------

Co-authored-by: jacky0218 <jacky0218@github.com>

* fix: adding the base64 image in markdown (#95)

* feat: more chunk info on UI

* fix: error when reindexing files

* refactor: allow more information exception trace when using gpt4v

* feat: add excel reader that treats each worksheet as a document

* Persist loader information when indexing file

* feat: allow hiding unneeded setting panels

* feat: allow specific timezone when creating conversation

* feat: add more confidence score (#96)

* Allow a list of rerankers

* Export llm reranking score instead of filter with boolean

* Get logprobs from LLMs

* Rename cohere reranking score

* Call 2 rerankers at once

* Run QA pipeline for each chunk to get qa_score

* Display more relevance scores

* Define another LLMScoring instead of editing the original one

* Export logprobs instead of probs

* Call LLMScoring

* Get qa_score only in the final answer

* feat: replace text length with token in file list

* ui: show index name instead of id in the settings

* feat(ai): restrict the vision temperature

* fix(ui): remove the misleading message about non-retrieved evidences

* feat(ui): show the reasoning name and description in the reasoning setting page

* feat(ui): show version on the main windows

* feat(ui): show default llm name in the setting page

* fix(conf): append the result of doc in llm_scoring (#97)

* fix: constraint maximum number of images

* feat(ui): allow filter file by name in file list page

* Fix exceeding token length error for OpenAI embeddings by chunking then averaging (#99)

* Average embeddings in case the text exceeds max size

* Add docstring

* fix: Allow empty string when calling embedding

* fix: update trulens LLM ranking score for retrieval confidence, improve citation (#98)

* Round when displaying not by default

* Add LLMTrulens reranking model

* Use llmtrulensscoring in pipeline

* fix: update UI display for trulen score

---------

Co-authored-by: taprosoft <tadashi@cinnamon.is>

* feat: add question decomposition & few-shot rewrite pipeline (#89)

* Create few-shot query-rewriting. Run and display the result in info_panel

* Fix style check

* Put the functions to separate modules

* Add zero-shot question decomposition

* Fix fewshot rewriting

* Add default few-shot examples

* Fix decompose question

* Fix importing rewriting pipelines

* fix: update decompose logic in fullQA pipeline

---------

Co-authored-by: taprosoft <tadashi@cinnamon.is>

* fix: add encoding utf-8 when save temporal markdown in vectorIndex (#101)

* fix: improve retrieval pipeline and relevant score display (#102)

* fix: improve retrieval pipeline by extending first round top_k with multiplier

* fix: minor fix

* feat: improve UI default settings and add quick switch option for pipeline

* fix: improve agent logics (#103)

* fix: improve agent progres display

* fix: update retrieval logic

* fix: UI display

* fix: less verbose debug log

* feat: add warning message for low confidence

* fix: LLM scoring enabled by default

* fix: minor update logics

* fix: hotfix image citation

* feat: update docx loader for handle merged table cells + handle zip file upload (#104)

* feat: update docx loader for handle merged table cells

* feat: handle zip file

* refactor: pre-commit

* fix: escape text in download UI

* feat: optimize vector store query db (#105)

* feat: optimize vector store query db

* feat: add file_id to chroma metadatas

* feat: remove unnecessary logs and update migrate script

* feat: iterate through file index

* fix: remove unused code

---------

Co-authored-by: taprosoft <tadashi@cinnamon.is>

* fix: add openai embedidng exponential back-off

* fix: update import download_loader

* refactor: codespell

* fix: update some default settings

* fix: update installation instruction

* fix: default chunk length in simple QA

* feat: add share converstation feature and enable retrieval history (#108)

* feat: add share converstation feature and enable retrieval history

* fix: update share conversation UI

---------

Co-authored-by: taprosoft <tadashi@cinnamon.is>

* fix: allow exponential backoff for failed OCR call (#109)

* fix: update default prompt when no retrieval is used

* fix: create embedding for long image chunks

* fix: add exception handling for additional table retriever

* fix: clean conversation & file selection UI

* fix: elastic search with empty doc_ids

* feat: add thumbnail PDF reader for quick multimodal QA

* feat: add thumbnail handling logic in indexing

* fix: UI text update

* fix: PDF thumb loader page number logic

* feat: add quick indexing pipeline and update UI

* feat: add conv name suggestion

* fix: minor UI change

* feat: citation in thread

* fix: add conv name suggestion in regen

* chore: add assets for usage doc

* chore: update usage doc

* feat: pdf viewer (#110)

* feat: update pdfviewer

* feat: update missing files

* fix: update rendering logic of infor panel

* fix: improve thumbnail retrieval logic

* fix: update PDF evidence rendering logic

* fix: remove pdfjs built dist

* fix: reduce thumbnail evidence count

* chore: update gitignore

* fix: add js event on chat msg select

* fix: update css for viewer

* fix: add env var for PDFJS prebuilt

* fix: move language setting to reasoning utils

---------

Co-authored-by: phv2312 <kat87yb@gmail.com>
Co-authored-by: trducng <trungduc1992@gmail.com>

* feat: graph rag (#116)

* fix: reload server when add/delete index

* fix: rework indexing pipeline to be able to disable vectorstore and splitter if needed

* feat: add graphRAG index with plot view

* fix: update requirement for graphRAG and lighten unnecessary packages

* feat: add knowledge network index (#118)

* feat: add Knowledge Network index

* fix: update reader mode setting for knet

* fix: update init knet

* fix: update collection name to index pipeline

* fix: missing req

---------

Co-authored-by: jeff52415 <jeff.yang@cinnamon.is>

* fix: update info panel return for graphrag

* fix: retriever setting graphrag

* feat: local llm settings (#122)

* feat: expose context length as reasoning setting to better fit local models

* fix: update context length setting for agents

* fix: rework threadpool llm call

* fix: fix improve indexing logic

* fix: fix improve UI

* feat: add lancedb

* fix: improve lancedb logic

* feat: add lancedb vectorstore

* fix: lighten requirement

* fix: improve lanceDB vs

* fix: improve UI

* fix: openai retry

* fix: update reqs

* fix: update launch command

* feat: update Dockerfile

* feat: add plot history

* fix: update default config

* fix: remove verbose print

* fix: update default setting

* fix: update gradio plot return

* fix: default gradio tmp

* fix: improve lancedb docstore

* fix: fix question decompose pipeline

* feat: add multimodal reader in UI

* fix: udpate docs

* fix: update default settings & docker build

* fix: update app startup

* chore: update documentation

* chore: update README

* chore: update README

---------

Co-authored-by: trducng <trungduc1992@gmail.com>

* chore: update README

* chore: update README

---------

Co-authored-by: trducng <trungduc1992@gmail.com>
Co-authored-by: cin-ace <ace@cinnamon.is>
Co-authored-by: Linh Nguyen <70562198+linhnguyen-cinnamon@users.noreply.github.com>
Co-authored-by: linhnguyen-cinnamon <cinmc0019@CINMC0019-LinhNguyen.local>
Co-authored-by: cin-jacky <101088014+jacky0218@users.noreply.github.com>
Co-authored-by: jacky0218 <jacky0218@github.com>
Co-authored-by: kan_cin <kan@cinnamon.is>
Co-authored-by: phv2312 <kat87yb@gmail.com>
Co-authored-by: jeff52415 <jeff.yang@cinnamon.is>
 2024-08-26 08:50:37 +07:00
trducng
ebf1315569
(pump:minor) Allow the indexing pipeline to report the indexing progress onto the UI (#81) 
* Turn the file indexing event to generator to report progress

* Fix React text's trimming function

* Refactor delete file into a method
 2024-05-25 22:09:41 +07:00
trducng
56dfc8fb53
Allow the application name to be configurable in settings (#80) 
* Make app name configurable

* Use app name in browser tab
 2024-05-20 22:37:24 +07:00
trducng (john)
04e602161b
Fix Yaml datetime format (#79) 2024-05-20 17:36:14 +07:00
trducng (john)
5ca3c25404
Avoid empty chat message (#78) 2024-05-20 16:20:50 +07:00
ian_Cin
b2296cfcdf
(bump:patch) Feat: Show app version in the Help page (#68) 
* typo

* show version in the Help page

* update docs

* pump duckduckgo-search

* allow app version to be set by env var
 2024-05-16 14:27:51 +07:00
ian_Cin
a122dc0a94
(bump:patch) Fix: llama-cpp-python security bug and setup local latest branch in github action (#66) 
* update llama-cpp-python version in response to https://github.com/Cinnamon/kotaemon/security/dependabot/1

* setup local latest branch in github action
 2024-05-15 17:57:37 +07:00
ian_Cin
654501e01c
(bump:minor) Feat: Add mechanism for user-site update and auto creating releases (#56) 
* move flowsettings.py and launch.py to root

* update docs

* sync sub package versions

* rename launch.py to app.py and make run scripts work with installation package

* add update scripts

* auto version for root package

* rename authors and update doc dir

* Update auto-bump-and-release.yaml to trigger on push to main branch

* latest as branch instead of tag

* pin deps versions

* cache the changelogs
 2024-05-15 16:34:50 +07:00
ian_Cin
eb198e0ff3
fix bug in delete file, remove file delete confirmation (#59) 2024-05-09 16:21:56 +07:00
Albert
466adf2d94
Feat/Add ReAct and ReWOO Reasoning Pipelines (#43) 
* Add ReactAgentPipeline by wrapping the ReactAgent

* Implement stream processing for ReactAgentPipeline and RewooAgentPipeline

* Fix highlight_citation in Rewoo and remove highlight_citation from React

* Fix importing ktem.llms inside kotaemon

* fix: Change Rewoo::solver's output to LLMInterface instead of plain text

* Add more user_settings to the RewooAgentPipeline

* Fix LLMTool

* Add more user_settings to the ReactAgentPipeline

* Minor fix

* Stream the react agent immediately

* Yield the Rewoo progress to info panel

* Hide the agent in flowsettings

* Remove redundant comments

---------

Co-authored-by: trducng <trungduc1992@gmail.com>
 2024-05-09 16:06:24 +07:00
Duc Nguyen (john)
ec11b54ff2
Add Azure AI Document Intelligence loader (#52) 
* Add azureai document intelligence loader

* Add load_data interface to Azure DI

* Bump version

* Access azure credentials from environment variables
 2024-04-29 14:49:55 +07:00
Duc Nguyen (john)
a8725710af
Allow users to select reasoning pipeline. Fix small issues with user UI, cohere name (#50) 
* Fix user page

* Allow changing LLM in reasoning pipeline

* Fix CohereEmbedding name
 2024-04-25 17:18:12 +07:00
Duc Nguyen (john)
e29bec6275
Allow file index to be private (#45) 
* Fix breaking reranker

* Allow private file index

* Avoid setting default to 1 when user management is enabled
 2024-04-25 14:24:35 +07:00
Duc Nguyen (john)
456f020caf
Enable MHTML reader (#44) 
* Enable mhtml loader

* Use default supported file types

* Add tests and bump version
 2024-04-23 14:16:24 +07:00
Duc Nguyen (john)
fbe983ccb3
Add relevant chat context when query the index (#42) 
* Add context for query

* Add older messages in the chat

* Update the indexing

* Make some hard-code values configurable

* Remove hard-code values
 2024-04-22 14:32:30 +07:00
Duc Nguyen (john)
749c9e5641
Remove redundant attributes in the file index (#41) 2024-04-20 18:21:32 +07:00
Duc Nguyen (john)
c6045bcb9f
Update the Citation pipeline according to new OpenAI function call interface (#40) 2024-04-20 01:12:23 +07:00
Duc Nguyen (john)
1b2082a140
Allow file selector to be disabled (#36) 
* Allow file selector to be disabled

* Update docs and variable names
 2024-04-16 18:43:56 +07:00
ian_Cin
e19893a509
fix typo (#35) 2024-04-15 23:16:32 +07:00
ian_Cin
5286ff48bc
Fix info panel overflow (#33) 
* update chatbot placeholder

* fix chat info panel overflow bug

* set azure_endpoint to required in AzureChatOpenAI

* update screenshots
 2024-04-14 09:34:14 +07:00
ian_Cin
8985963e1e
Setup app data dir (#32) 
* setup local data dir

* update readme

* update chat panel

* update help page
 2024-04-13 23:26:06 +07:00
Duc Nguyen (john)
0417610d3e
Refactor reasoning pipeline (#31) 
* Move the text rendering out for reusability

* Refactor common operations in the reasoning pipeline

* Add run method

* Provide dedicated method for invoke
 2024-04-13 23:13:04 +07:00
ian_Cin
4022af7e9b
allow LlamaCppChat to auto download model from hf hub (#29) 2024-04-13 18:57:04 +07:00
Duc Nguyen (john)
917fb0a082
Treat index id as auto-generated field (#27) 
* Treat index id as auto-generated field

* fix Can't create index: KeyError: 'embedding' #28

* udpate docs

* Update requirement

* Use lighter default local embedding model

---------

Co-authored-by: ian <ian@cinnamon.is>
 2024-04-13 18:29:37 +07:00
Duc Nguyen (john)
66905d39c4
Allow adding, updating and deleting indices (#24) 
* Allow adding indices

* Allow deleting indices

* Allow updating the indices

* When there are multiple indices, group them below Indices tab

* Update elem classes
 2024-04-12 15:41:09 +07:00
Duc Nguyen (john)
5ce6bac03d
Allow listing indices (#22) 2024-04-11 16:28:04 +07:00
Duc Nguyen (john)
3ed50b0f10
Improve LLMs and Embedding models resources experience (#21) 
* Fix inconsistent default values
* Disallow LLM's empty name. Handle LLM creation error on UI
 2024-04-11 07:50:53 +07:00
Duc Nguyen (john)
f3e82b2e70
Put the preparation step in FileIndex to on_start (#20) 2024-04-10 19:30:45 +07:00
ian_Cin
b507eef541
Improve manuals (#19) 
* Rename Admin -> Resources
* Improve ui
* Update docs
 2024-04-10 17:04:04 +07:00
Duc Nguyen (john)
7b3307e3c4
Provide embedding manager (#16) 
* Provide the Embedding management UI

* Update Fastembed documentation

* Add validation when adding / updating embeddings

* Stop using the old ktem embeddings manager

* Set default local embedding models

* Move the local embeddings below in flowsettings

* Update flowsettings
 2024-04-10 15:11:44 +07:00
Duc Nguyen (john)
ed10020ea3
Refactor embeddings and provide vanilla OpenAI-based embeddings (#11) 
* Prepend all Langchain-based embeddings with LC

* Provide vanilla OpenAI embeddings

* Add test for AzureOpenAIEmbeddings and OpenAIEmbeddings

* Fix disallowed empty string

* Use OpenAIEmbeddings in flowsettings

---------

Co-authored-by: ian_Cin <ian@cinnamon.is>
 2024-04-09 15:07:59 +07:00
Duc Nguyen (john)
e75354d410
Enable fastembed as a local embedding vendor (#12) 
* Prepend all Langchain-based embeddings with LC

* Provide vanilla OpenAI embeddings

* Add test for AzureOpenAIEmbeddings and OpenAIEmbeddings

* Incorporate fastembed

---------

Co-authored-by: ian_Cin <ian@cinnamon.is>
 2024-04-09 01:44:34 +07:00
ian_Cin
8001c86b16
Feat/new UI (#13) 
* new custom theme

* improve css: scrollbar, header, tabs and buttons

* update settings tab

* open file index selector by default

* update chat control panel

* update chat panel

* update file index page

* cap gradio<=4.22.0

* rename admin page

* adjust UI

* update flowsettings

* auto start in browser

* change colour for edit LLM page's button
 2024-04-08 22:23:00 +07:00
Duc Nguyen (john)
a203fc0f7c
Allow users to add LLM within the UI (#6) 
* Rename AzureChatOpenAI to LCAzureChatOpenAI
* Provide vanilla ChatOpenAI and AzureChatOpenAI
* Remove the highest accuracy, lowest cost criteria

These criteria are unnecessary. The users, not pipeline creators, should choose
which LLM to use. Furthermore, it's cumbersome to input this information,
really degrades user experience.

* Remove the LLM selection in simple reasoning pipeline
* Provide a dedicated stream method to generate the output
* Return placeholder message to chat if the text is empty
 2024-04-06 11:53:17 +07:00
Duc Nguyen (john)
e187e23dd1
Improve UX (#9) 
* Go to chat tab when resignin
* Allow placeholder message configurable
* Hide setting tabs if there aren't any settings
 2024-04-04 15:39:24 +07:00
ian_Cin
ecf09b275f
Fix UI bugs (#8) 
* Auto create conversation when the user starts

* Add conversation rename rule check

* Fix empty name during save

* Confirm deleting conversation

* Show warning if users don't select file when upload files in the File Index

* Feedback when user uploads duplicated file

* Limit the file types

* Fix valid username

* Allow login when username with leading and trailing whitespaces

* Improve the user

* Disable admin panel for non-admnin user

* Refresh user lists after creating/deleting users

* Auto logging in

* Clear admin information upon signing out

* Fix unable to receive uploaded filename that include special characters, like !@#$%^&*().pdf

* Set upload validation for FileIndex

* Improve user management UI/UIX

* Show extraction error when indexing file

* Return selected user -1 when signing out

* Fix default supported file types in file index

* Validate changing password

* Allow the selector to contain mulitple gradio components

* A more tolerable placeholder screen

* Allow chat suggestion box

* Increase concurrency limit

* Make adobe loader optional

* Use BaseReasoning

---------

Co-authored-by: trducng <trungduc1992@gmail.com>
 2024-04-03 16:33:54 +07:00
ian_Cin
43a18ba070
Feat/regenerate answer (#7) 
* Add regen button and repharasing question on regen

* Stop appending regen messages to history, allow only one

* Add dynamic conversation state

* Allow reasoning pipeline to manipulate state

---------

Co-authored-by: albert <albert@cinnamon.is>
Co-authored-by: Duc Nguyen (john) <trungduc1992@gmail.com>
 2024-04-03 15:37:55 +07:00
ian_Cin
e67a25c0bd
Feat/add multimodal loader (#5) 
* Add Adobe reader as the multimodal loader

* Allow FullQAPipeline to reasoning on figures

* fix: move the adobe import to avoid ImportError, notify users whenever they run the AdobeReader

---------

Co-authored-by: cin-albert <albert@cinnamon.is>
 2024-04-03 14:52:40 +07:00
ian_Cin
a3bf728400
Update various docs (#4) 
* rename cli tool

* remove redundant docs

* update docs

* update macos instructions

* add badges
 2024-03-29 19:47:03 +07:00
ian
14482e9755 bug fix: settings are not persistent 2024-03-28 16:36:05 +07:00
ian
f9cc40ca25 improve llm selection for simple reasoning pipeline 2024-03-28 16:35:13 +07:00
ian
e3498a4958 rename ktem test dir 2024-03-28 16:27:05 +07:00
ian
c1b1371a68 enable config through .env 2024-03-27 19:04:48 +07:00
ian
da86fa463f rename test dir 2024-03-27 18:56:06 +07:00
ian_Cin
d22ae88c7a make default installation faster (#2) 
* remove cohere as default

* refractor dependencies

* use llama-index pdf reader as default (pypdf)

* fix some lazy docstring

* update install scripts

* minor fix
 2024-03-21 22:48:20 +07:00
ian_Cin
a8f92b3f9e post migrate cleanup 2024-03-18 23:10:20 +07:00
ian_Cin
df12dec732 Feat/local endpoint llm (#148) 
* serve local model in a different process from the app
---------

Co-authored-by: albert <albert@cinnamon.is>
Co-authored-by: trducng <trungduc1992@gmail.com>
 2024-03-15 16:17:33 +07:00
Duc Nguyen (john)
2950e6ed02 Improve behavior of simple reasoning (#157) 
* Add base reasoning implementation

* Provide explicit async and streaming capability

* Allow refreshing the information panel
 2024-03-12 13:03:38 +07:00
Duc Nguyen (john)
cb01d27d19 Fix integrating indexing and retrieval pipelines to FileIndex (#155) 
* Add docs for settings
* Add mdx_truly_sane_lists to doc requirements
 2024-03-10 16:41:42 +07:00
trducng
2b3571e892 Fix subscribing sign-in/out 2024-03-08 13:38:29 +07:00
Duc Nguyen (john)
4f356f7f9a Provide dedicated page for login (#153) 2024-03-08 08:06:51 +07:00