Best docs Cinnamon will probably ever have (#105)

docs/contributing.md

@@ -0,0 +1,165 @@
+# Getting started
+
+## Setup
+
+- Create conda environment (suggest 3.10)
+
+  ```shell
+  conda create -n kotaemon python=3.10
+  conda activate kotaemon
+  ```
+
+- Clone the repo
+
+  ```shel
+  git clone git@github.com:Cinnamon/kotaemon.git
+  cd kotaemon
+  ```
+
+- Install all
+
+  ```shell
+  pip install -e ".[dev]"
+  ```
+
+- Pre-commit
+
+  ```shell
+  pre-commit install
+  ```
+
+- Test
+
+  ```shell
+  pytest tests
+  ```
+
+## Credential sharing
+
+This repo uses [git-secret](https://sobolevn.me/git-secret/) to share credentials, which
+internally uses `gpg` to encrypt and decrypt secret files.
+
+This repo uses `python-dotenv` to manage credentials stored as environment variable.
+Please note that the use of `python-dotenv` and credentials are for development
+purposes only. Thus, it should not be used in the main source code (i.e. `kotaemon/` and `tests/`), but can be used in `examples/`.
+
+### Install git-secret
+
+Please follow the [official guide](https://sobolevn.me/git-secret/installation) to install git-secret.
+
+For Windows users, see [For Windows users](#for-windows-users).
+
+For users who don't have sudo privilege to install packages, follow the `Manual Installation` in the [official guide](https://sobolevn.me/git-secret/installation) and set `PREFIX` to a path that you have access to. And please don't forget to add `PREFIX` to your `PATH`.
+
+### Gaining access to credientials
+
+In order to gain access to the secret files, you must provide your gpg public file to anyone who has access and ask them to ask your key to the keyring. For a quick tutorial on generating your gpg key pair, you can refer to the `Using gpg` section from the [git-secret main page](https://sobolevn.me/git-secret/).
+
+### Decrypt the secret file
+
+The credentials are encrypted in the `.env.secret` file. To print the decrypted content to stdout, run
+
+```shell
+git-secret cat [filename]
+```
+
+Or to get the decrypted `.env` file, run
+
+```shell
+git-secret reveal [filename]
+```
+
+### For Windows users
+
+git-secret is currently not available for Windows, thus the easiest way is to use it in WSL (please use the latest version of WSL2). From there you have 2 options:
+
+1. Using the gpg of WSL.
+
+   This is the most straight-forward option since you would use WSL just like any other unix environment. However, the downside is that you have to make WSL your main environment, which means WSL must have write permission on your repo. To achieve this, you must either:
+
+   - Clone and store your repo inside WSL's file system.
+   - Provide WSL with necessary permission on your Windows file system. This can be achieve by setting `automount` options for WSL. To do that, add these content to `/etc/wsl.conf` and then restart your sub-system.
+
+     ```shell
+     [automount]
+     options = "metadata,umask=022,fmask=011"
+     ```
+
+     This enables all permissions for user owner.
+
+2. Using the gpg of Windows but with git-secret from WSL.
+
+   For those who use Windows as the main environment, having to switch back and forth between Windows and WSL will be inconvenient. You can instead stay within your Windows environment and apply some tricks to use `git-secret` from WSL.
+
+   - Install and setup `gpg` on Windows.
+   - Install `git-secret` on WSL. Now in Windows, you can invoke `git-secret` using `wsl git-secret`.
+   - Alternatively you can setup alias in CMD to shorten the syntax. Please refer to [this SO answer](https://stackoverflow.com/a/65823225) for the instruction. Some recommended aliases are:
+
+     ```bat
+     @echo off
+
+     :: Commands
+     DOSKEY ls=dir /B $*
+     DOSKEY ll=dir /a $*
+     DOSKEY git-secret=wsl git-secret $*
+     DOSKEY gs=wsl git-secret $*
+     ```
+
+     Now you can invoke `git-secret` in CMD using `git-secret` or `gs`.
+
+     - For Powershell users, similar behaviours can be achieved using `Set-Alias` and `profile.ps1`. Please refer this [SO thread](https://stackoverflow.com/questions/61081434/how-do-i-create-a-permanent-alias-file-in-powershell-core) as an example.
+
+# PR guideline
+
+## Common conventions
+
+- Review should be done as soon as possible (within 2 business days).
+- PR title: [ticket] One-line description (example: [AUR-385, AUR-388] Declare BaseComponent and decide LLM call interface).
+- [Encouraged] Provide a quick description in the PR, so that:
+  - Reviewers can quickly understand the direction of the PR.
+  - It will be included in the commit message when the PR is merged.
+
+## Environment caching on PR
+
+- To speed up CI, environments are cached based on the version specified in `__init__.py`.
+- Since dependencies versions in `setup.py` are not pinned, you need to pump the version in order to use a new environment. That environment will then be cached and used by your subsequence commits within the PR, until you pump the version again
+- The new environment created during your PR is cached and will be available to others once the PR is merged.
+- If you are experimenting with new dependencies and want a fresh environment every time, add `[ignore cache]` in your commit message. The CI will create a fresh environment to run your commit and then discard it.
+- If your PR include updated dependencies, the recommended workflow would be:
+  - Doing development as usual.
+  - When you want to run the CI, push a commit with the message containing `[ignore cache]`.
+  - Once the PR is final, pump the version in `__init__.py` and push a final commit not containing `[ignore cache]`.
+
+Examples: https://github.com/Cinnamon/kotaemon/pull/2
+
+## Merge PR guideline
+
+- Use squash and merge option
+- 1st line message is the PR title.
+- The text area is the PR description.
+
+![image](https://github.com/Cinnamon/kotaemon/assets/35283585/e2593010-d7ef-46e3-8719-6fcae0315b5d)
+![image](https://github.com/Cinnamon/kotaemon/assets/35283585/bfe6a117-85cd-4dd4-b432-197c791a9901)
+
+## Develop pipelines
+
+- Nodes
+- Params
+- Run function
+
+```
+from kotaemon.base import BaseComponent
+
+class Pipeline(BaseComponent):
+   llm: AzureOpenAIEmbedding
+   doc_store: BaseDocumentStore
+
+   def run(self, input1, input2) -> str:
+      input = input1 + input2
+      output = self.llm(input)
+      self.doc_store.add(output)
+      return output
+
+pipeline = Pipeline(llm=AzureOpenAILLM(), doc_store=InMemoryDocstore())
+output = pipeline("this is text1", "this is text2")
+```

docs/create-a-component.md

@@ -0,0 +1,71 @@
+# Creating a component
+
+A fundamental concept in kotaemon is "component".
+
+Anything that isn't data or data structure is a "component". A component can be
+thought of as a step within a pipeline. It takes in some input, processes it,
+and returns an output, just the same as a Python function! The output will then
+become an input for the next component in a pipeline. In fact, a pipeline is just
+a component. More appropriately, a nested component: a component that makes use of one or more other components in
+the processing step. So in reality, there isn't a difference between a pipeline
+and a component! Because of that, in kotaemon, we will consider them the
+same as "component".
+
+To define a component, you will:
+
+1. Create a class that subclasses from `kotaemon.base.BaseComponent`
+2. Declare init params with type annotation
+3. Declare nodes (nodes are just other components!) with type annotation
+4. Implement the processing logic in `run`.
+
+The syntax of a component is as follow:
+
+```python
+from kotaemon.base import BaseComponent
+from kotaemon.llms import AzureChatOpenAI
+from kotaemon.parsers import RegexExtractor
+
+
+class FancyPipeline(BaseComponent):
+    param1: str = "This is param1"
+    param2: int = 10
+    param3: float
+
+    node1: BaseComponent    # this is a node because of BaseComponent type annotation
+    node2: AzureChatOpenAI  # this is also a node because AzureChatOpenAI subclasses BaseComponent
+    node3: RegexExtractor   # this is also a node bceause RegexExtractor subclasses BaseComponent
+
+    def run(self, some_text: str):
+        prompt = (self.param1 + some_text) * int(self.param2 + self.param3)
+        llm_pred = self.node2(prompt).text
+        matches = self.node3(llm_pred)
+        return matches
+```
+
+Then this component can be used as follow:
+
+```python
+llm = AzureChatOpenAI(endpoint="some-endpont")
+extractor = RegexExtractor(pattern=["yes", "Yes"])
+
+component = FancyPipeline(
+    param1="Hello"
+    param3=1.5
+    node1=llm,
+    node2=llm,
+    node3=extractor
+)
+component("goodbye")
+```
+
+This way, we can define each operation as a reusable component, and use them to
+compose larger reusable components!
+
+## Benefits of component
+
+By defining a component as above, we formally encapsulate all the necessary
+information inside a single class. This introduces several benefits:
+
+1. Allow tools like promptui to inspect the inner working of a component in
+   order to automatically generate the promptui.
+2. Allow visualizing a pipeline for debugging purpose.

docs/data-components.md

@@ -0,0 +1,32 @@
+The data & data structure components include:
+
+- The `Document` class.
+- The document store.
+- The vector store.
+
+### Data Loader
+
+- PdfLoader
+- Layout-aware with table parsing PdfLoader
+
+  - MathPixLoader: To use this loader, you need MathPix API key, refer to [mathpix docs](https://docs.mathpix.com/#introduction) for more information
+  - OCRLoader: This loader uses lib-table and Flax pipeline to perform OCR and read table structure from PDF file (TODO: add more info about deployment of this module).
+  - Output:
+
+    - Document: text + metadata to identify whether it is table or not
+
+      ```
+      - "source": source file name
+      - "type": "table" or "text"
+      - "table_origin": original table in markdown format (to be feed to LLM or visualize using external tools)
+      - "page_label": page number in the original PDF document
+      ```
+
+### Document Store
+
+- InMemoryDocumentStore
+
+### Vector Store
+
+- ChromaVectorStore
+- InMemoryVectorStore

docs/extra/css/code_select.css

@@ -0,0 +1,5 @@
+.language-pycon .gp,
+.language-pycon .go {
+  /* Generic.Prompt,  Generic.Output */
+  user-select: none;
+}

docs/index.md

@@ -0,0 +1 @@
+--8<-- "README.md"

docs/overview.md

@@ -0,0 +1,75 @@
+## Introduction
+
+`kotaemon` library focuses on the AI building blocks to implement the Kotaemon. It can be used in both client project and in product development. It consists of base interfaces, core components and a list of utilities:
+
+- Base interfaces: `kotaemon` defines the base interface of a component in a pipeline. A pipeline is also a component. By clearly define this interface, a pipeline of steps can be easily constructed and orchestrated.
+- Core components: `kotaemon` implements (or wraps 3rd-party libraries like Langchain, llama-index,... when possible) commonly used components in kotaemon use cases. Some of these components are: LLM, vector store, document store, retriever... For a detailed list and description of these components, please refer to the [Pipeline Components](Pipeline-Components) and [Data & Data Structure Components](Data-&-Data-Structure-Components) sections.
+- List of utilities: `lib-knowledge` provides utilities and tools that are usually needed in client project. For example, it provides a prompt engineering UI for AI developers in a project to quickly create a prompt engineering tool for DMs and QALs. It also provides a command to quickly spin up a project code base. For a full list and description of these utilities, please refer to the [Utilities](Utilities) section.
+
+```mermaid
+mindmap
+  root((kotaemon))
+    Base Interfaces
+      Document
+      LLMInterface
+      RetrievedDocument
+      BaseEmbeddings
+      BaseChat
+      BaseCompletion
+      ...
+    Core Components
+      LLMs
+        AzureOpenAI
+        OpenAI
+      Embeddings
+        AzureOpenAI
+        OpenAI
+        HuggingFaceEmbedding
+      VectorStore
+        InMemoryVectorstore
+        ChromaVectorstore
+      Agent
+      Tool
+      DocumentStore
+      ...
+    Utilities
+      Scaffold project
+      PromptUI
+      Documentation Support
+```
+
+## Expected benefit
+
+Before `kotaemon`:
+
+- Starting everything from scratch.
+- Knowledge and expertise is fragmented.
+- Nothing to reuse.
+- No way to collaborate between tech and non-tech experts.
+
+`kotaemon` expects to completely revolutionize the way we are building LLM-related projects. It helps the company side-steps those issues by:
+
+- Standardize the interface to (1) make building LLM pipeline clearer (2) more reasonable to integrate pipelines between different projects.
+- Centralize LLM-related technical components into 1 place. Avoid fragmented technology development. Easy to find the LLM-related technology inside the company.
+- Centralize bug fixes and improvements in 1 place.
+- Reduce boilerplate code during project development.
+- Lightning fast prototyping.
+
+## Install
+
+The kotaemon can be installed from source with:
+
+```
+pip install kotaemon@git+ssh://git@github.com/Cinnamon/kotaemon.git
+```
+
+or from Cinnamon's internal python package index:
+
+```
+pip install kotaemon --extra-index-url https://ian_devpi.promptui.dm.cinnamon.is/root/packages
+```
+
+## Example use cases
+
+- Start a project from scratch: `kh start-project`
+- Run prompt engineering UI tool: `kh promptui export`, then `kh promptui run`.

docs/scripts/generate_examples_docs.py

@@ -0,0 +1,65 @@
+# import shutil
+from pathlib import Path
+from typing import Any, Iterable
+
+import mkdocs_gen_files
+
+# get the root source code directory
+doc_dir_name = "docs"
+doc_dir = Path(__file__)
+while doc_dir.name != doc_dir_name and doc_dir != doc_dir.parent:
+    doc_dir = doc_dir.parent
+
+if doc_dir == doc_dir.parent:
+    raise ValueError(f"root_name ({doc_dir_name}) not in path ({str(Path(__file__))}).")
+
+
+def generate_docs_for_examples_readme(
+    examples_dir: Path, target_doc_folder: str, ignored_modules: Iterable[Any] = []
+):
+    if not examples_dir.is_dir():
+        raise ModuleNotFoundError(str(examples_dir))
+
+    nav = mkdocs_gen_files.Nav()
+
+    for path in sorted(examples_dir.rglob("*README.md")):
+        # ignore modules with name starts with underscore (i.e. __init__)
+        if path.name.startswith("_") or path.name.startswith("test"):
+            continue
+
+        module_path = path.parent.relative_to(examples_dir).with_suffix("")
+        doc_path = path.parent.relative_to(examples_dir).with_suffix(".md")
+        full_doc_path = Path(target_doc_folder, doc_path)
+
+        parts = list(module_path.parts)
+        identifier = ".".join(parts)
+
+        if "tests" in parts:
+            continue
+
+        ignore = False
+        for each_module in ignored_modules:
+            if identifier.startswith(each_module):
+                ignore = True
+                break
+        if ignore:
+            continue
+
+        nav_titles = [name.replace("_", " ").title() for name in parts]
+        nav[nav_titles] = doc_path.as_posix()
+
+        with mkdocs_gen_files.open(full_doc_path, "w") as f:
+            f.write(f'--8<-- "{path.relative_to(examples_dir.parent)}"')
+
+        mkdocs_gen_files.set_edit_path(
+            full_doc_path, Path("..") / path.relative_to(examples_dir.parent)
+        )
+
+    with mkdocs_gen_files.open(f"{target_doc_folder}/NAV.md", "w") as nav_file:
+        nav_file.writelines(nav.build_literate_nav())
+
+
+generate_docs_for_examples_readme(
+    examples_dir=doc_dir.parent / "examples",
+    target_doc_folder="examples",
+)

docs/scripts/generate_reference_docs.py

@@ -0,0 +1,75 @@
+# import shutil
+from pathlib import Path
+from typing import Any, Iterable
+
+import mkdocs_gen_files
+
+# get the root source code directory
+doc_dir_name = "docs"
+doc_dir = Path(__file__)
+while doc_dir.name != doc_dir_name and doc_dir != doc_dir.parent:
+    doc_dir = doc_dir.parent
+
+if doc_dir == doc_dir.parent:
+    raise ValueError(f"root_name ({doc_dir_name}) not in path ({str(Path(__file__))}).")
+
+
+def generate_docs_for_src_code(
+    code_dir: Path, target_doc_folder: str, ignored_modules: Iterable[Any] = []
+):
+    if not code_dir.is_dir():
+        raise ModuleNotFoundError(str(code_dir))
+
+    nav = mkdocs_gen_files.Nav()
+
+    for path in sorted(code_dir.rglob("*.py")):
+        # ignore modules with name starts with underscore (i.e. __init__)
+        # if path.name.startswith("_") or path.name.startswith("test"):
+        #     continue
+
+        module_path = path.relative_to(code_dir).with_suffix("")
+        doc_path = path.relative_to(code_dir).with_suffix(".md")
+        full_doc_path = Path(target_doc_folder, doc_path)
+
+        parts = list(module_path.parts)
+
+        if parts[-1] == "__init__":
+            doc_path = doc_path.with_name("index.md")
+            full_doc_path = full_doc_path.with_name("index.md")
+            parts.pop()
+
+        if not parts:
+            continue
+
+        if "tests" in parts:
+            continue
+
+        identifier = ".".join(parts)
+        ignore = False
+        for each_module in ignored_modules:
+            if identifier.startswith(each_module):
+                ignore = True
+                break
+        if ignore:
+            continue
+
+        nav_titles = [name.replace("_", " ").title() for name in parts]
+        nav[nav_titles] = doc_path.as_posix()
+
+        with mkdocs_gen_files.open(full_doc_path, "w") as f:
+            f.write(f"::: {identifier}")
+
+        # this method works in docs folder
+        mkdocs_gen_files.set_edit_path(
+            full_doc_path, Path("..") / path.relative_to(code_dir.parent)
+        )
+
+    with mkdocs_gen_files.open(f"{target_doc_folder}/NAV.md", "w") as nav_file:
+        nav_file.writelines(nav.build_literate_nav())
+
+
+generate_docs_for_src_code(
+    code_dir=doc_dir.parent / "kotaemon",
+    target_doc_folder="reference",
+    ignored_modules={"contribs"},
+)

docs/theme/assets/pymdownx-extras/material-extra-theme-TVq-kNRT.js

@@ -0,0 +1,2 @@
+!function(){"use strict";var e;e=function(e){"true"===localStorage.getItem("data-md-prefers-color-scheme")&&document.querySelector("body").setAttribute("data-md-color-scheme",e.matches?"dracula":"default")},new MutationObserver((function(t){t.forEach((function(t){if("childList"===t.type&&t.addedNodes.length)for(var a=0;a<t.addedNodes.length;a++){var r=t.addedNodes[a];if(1===r.nodeType&&"body"===r.tagName.toLowerCase()){d=r,o=void 0,c=void 0,l=void 0,o="not all"!==window.matchMedia("(prefers-color-scheme)").media,c=localStorage.getItem("data-md-color-scheme"),l=localStorage.getItem("data-md-prefers-color-scheme"),c||(c="dracula"),l||(l="false"),"true"===l&&o?c=window.matchMedia("(prefers-color-scheme: dark)").matches?"dracula":"default":l="false",d.setAttribute("data-md-prefers-color-scheme",l),d.setAttribute("data-md-color-scheme",c),o&&window.matchMedia("(prefers-color-scheme: dark)").addListener(e);break}}var d,o,c,l}))})).observe(document.querySelector("html"),{childList:!0}),window.toggleScheme=function(){var e=document.querySelector("body"),t="not all"!==window.matchMedia("(prefers-color-scheme)").media,a=e.getAttribute("data-md-color-scheme"),r=e.getAttribute("data-md-prefers-color-scheme");t&&"default"===a&&"true"!==r?(r="true",a=window.matchMedia("(prefers-color-scheme: dark)").matches?"dracula":"default"):t&&"true"===r?(r="false",a="dracula"):"dracula"===a?(r="false",a="default"):(r="false",a="dracula"),localStorage.setItem("data-md-prefers-color-scheme",r),e.setAttribute("data-md-prefers-color-scheme",r),e.setAttribute("data-md-color-scheme",a)}}();
+//# sourceMappingURL=material-extra-theme-TVq-kNRT.js.map

docs/theme/assets/pymdownx-extras/material-extra-theme-TVq-kNRT.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"material-extra-theme-TVq-kNRT.js","sources":["material-extra-theme.js"],"sourcesContent":["(() => {\n\n  const preferToggle = e => {\n    if (localStorage.getItem(\"data-md-prefers-color-scheme\") === \"true\") {\n      document.querySelector(\"body\").setAttribute(\"data-md-color-scheme\", (e.matches) ? \"dracula\" : \"default\")\n    }\n  }\n\n  const setupTheme = body => {\n    const preferSupported = window.matchMedia(\"(prefers-color-scheme)\").media !== \"not all\"\n    let scheme = localStorage.getItem(\"data-md-color-scheme\")\n    let prefers = localStorage.getItem(\"data-md-prefers-color-scheme\")\n\n    if (!scheme) {\n      scheme = \"dracula\"\n    }\n    if (!prefers) {\n      prefers = \"false\"\n    }\n\n    if (prefers === \"true\" && preferSupported) {\n      scheme = (window.matchMedia(\"(prefers-color-scheme: dark)\").matches) ? \"dracula\" : \"default\"\n    } else {\n      prefers = \"false\"\n    }\n\n    body.setAttribute(\"data-md-prefers-color-scheme\", prefers)\n    body.setAttribute(\"data-md-color-scheme\", scheme)\n\n    if (preferSupported) {\n      const matchListener = window.matchMedia(\"(prefers-color-scheme: dark)\")\n      matchListener.addListener(preferToggle)\n    }\n  }\n\n  const observer = new MutationObserver(mutations => {\n    mutations.forEach(mutation => {\n      if (mutation.type === \"childList\") {\n        if (mutation.addedNodes.length) {\n          for (let i = 0; i < mutation.addedNodes.length; i++) {\n            const el = mutation.addedNodes[i]\n\n            if (el.nodeType === 1 && el.tagName.toLowerCase() === \"body\") {\n              setupTheme(el)\n              break\n            }\n          }\n        }\n      }\n    })\n  })\n\n  observer.observe(document.querySelector(\"html\"), {childList: true})\n})()\n\nwindow.toggleScheme = () => {\n  const body = document.querySelector(\"body\")\n  const preferSupported = window.matchMedia(\"(prefers-color-scheme)\").media !== \"not all\"\n  let scheme = body.getAttribute(\"data-md-color-scheme\")\n  let prefer = body.getAttribute(\"data-md-prefers-color-scheme\")\n\n  if (preferSupported && scheme === \"default\" && prefer !== \"true\") {\n    prefer = \"true\"\n    scheme = (window.matchMedia(\"(prefers-color-scheme: dark)\").matches) ? \"dracula\" : \"default\"\n  } else if (preferSupported && prefer === \"true\") {\n    prefer = \"false\"\n    scheme = \"dracula\"\n  } else if (scheme === \"dracula\") {\n    prefer = \"false\"\n    scheme = \"default\"\n  } else {\n    prefer = \"false\"\n    scheme = \"dracula\"\n  }\n  localStorage.setItem(\"data-md-prefers-color-scheme\", prefer)\n  body.setAttribute(\"data-md-prefers-color-scheme\", prefer)\n  body.setAttribute(\"data-md-color-scheme\", scheme)\n}\n"],"names":["preferToggle","e","localStorage","getItem","document","querySelector","setAttribute","matches","MutationObserver","mutations","forEach","mutation","type","addedNodes","length","i","el","nodeType","tagName","toLowerCase","body","preferSupported","scheme","prefers","window","matchMedia","media","addListener","observe","childList","toggleScheme","getAttribute","prefer","setItem"],"mappings":"yBAAA,IAEQA,IAAe,SAAAC,GAC0C,SAAzDC,aAAaC,QAAQ,iCACvBC,SAASC,cAAc,QAAQC,aAAa,uBAAyBL,EAAEM,QAAW,UAAY,YA+BjF,IAAIC,kBAAiB,SAAAC,GACpCA,EAAUC,SAAQ,SAAAC,GAChB,GAAsB,cAAlBA,EAASC,MACPD,EAASE,WAAWC,OACtB,IAAK,IAAIC,EAAI,EAAGA,EAAIJ,EAASE,WAAWC,OAAQC,IAAK,CACnD,IAAMC,EAAKL,EAASE,WAAWE,GAE/B,GAAoB,IAAhBC,EAAGC,UAA+C,SAA7BD,EAAGE,QAAQC,cAA0B,CAlCrDC,EAmCIJ,EAlCfK,SACFC,SACAC,SAFEF,EAAwE,YAAtDG,OAAOC,WAAW,0BAA0BC,MAChEJ,EAASpB,aAAaC,QAAQ,wBAC9BoB,EAAUrB,aAAaC,QAAQ,gCAE9BmB,IACHA,EAAS,WAENC,IACHA,EAAU,SAGI,SAAZA,GAAsBF,EACxBC,EAAUE,OAAOC,WAAW,gCAAgClB,QAAW,UAAY,UAEnFgB,EAAU,QAGZH,EAAKd,aAAa,+BAAgCiB,GAClDH,EAAKd,aAAa,uBAAwBgB,GAEtCD,GACoBG,OAAOC,WAAW,gCAC1BE,YAAY3B,GAalB,KACF,CACF,CAtCW,IAAAoB,EACXC,EACFC,EACAC,CAsCJ,GACF,IAESK,QAAQxB,SAASC,cAAc,QAAS,CAACwB,WAAW,IAG/DL,OAAOM,aAAe,WACpB,IAAMV,EAAOhB,SAASC,cAAc,QAC9BgB,EAAwE,YAAtDG,OAAOC,WAAW,0BAA0BC,MAChEJ,EAASF,EAAKW,aAAa,wBAC3BC,EAASZ,EAAKW,aAAa,gCAE3BV,GAA8B,YAAXC,GAAmC,SAAXU,GAC7CA,EAAS,OACTV,EAAUE,OAAOC,WAAW,gCAAgClB,QAAW,UAAY,WAC1Ec,GAA8B,SAAXW,GAC5BA,EAAS,QACTV,EAAS,WACW,YAAXA,GACTU,EAAS,QACTV,EAAS,YAETU,EAAS,QACTV,EAAS,WAEXpB,aAAa+B,QAAQ,+BAAgCD,GACrDZ,EAAKd,aAAa,+BAAgC0B,GAClDZ,EAAKd,aAAa,uBAAwBgB,EAC5C"}

docs/theme/main.html

@@ -0,0 +1,6 @@
+{% extends "base.html" %}
+
+{% block libs %}
+{{ super() }}
+{% include "partials/libs.html" ignore missing %}
+{% endblock %}

docs/theme/partials/footer.html

@@ -0,0 +1,49 @@
+
+{% import "partials/language.html" as lang with context %}
+<footer class="md-footer">
+  {% if page.previous_page or page.next_page %}
+    <nav
+      class="md-footer__inner md-grid"
+      aria-label="{{ lang.t('footer.title') }}"
+    >
+      {% if page.previous_page %}
+        <a
+          href="{{ page.previous_page.url | url }}"
+          class="md-footer__link md-footer__link--prev"
+          rel="prev"
+        >
+          <div class="md-footer__button md-icon">
+            {% include ".icons/material/arrow-left.svg" %}
+          </div>
+          <div class="md-footer__title">
+            <div class="md-ellipsis">
+              <span class="md-footer__direction">
+                {{ lang.t("footer.previous") }}
+              </span>
+              {{ page.previous_page.title }}
+            </div>
+          </div>
+        </a>
+      {% endif %}
+      {% if page.next_page %}
+        <a
+          href="{{ page.next_page.url | url }}"
+          class="md-footer__link md-footer__link--next"
+          rel="next"
+        >
+          <div class="md-footer__title">
+            <div class="md-ellipsis">
+              <span class="md-footer__direction">
+                {{ lang.t("footer.next") }}
+              </span>
+              {{ page.next_page.title }}
+            </div>
+          </div>
+          <div class="md-footer__button md-icon">
+            {% include ".icons/material/arrow-right.svg" %}
+          </div>
+        </a>
+      {% endif %}
+    </nav>
+  {% endif %}
+</footer>

docs/theme/partials/header.html

@@ -0,0 +1,88 @@
+
+{% set site_url = config.site_url | d(nav.homepage.url, true) | url %}
+{% if not config.use_directory_urls and site_url[0] == site_url[-1] == "." %}
+  {% set site_url = site_url ~ "/index.html" %}
+{% endif %}
+<header class="md-header" data-md-component="header">
+  <nav
+    class="md-header__inner md-grid"
+    aria-label="{{ lang.t('header.title') }}"
+  >
+    <a
+      href="{{ site_url }}"
+      title="{{ config.site_name | e }}"
+      class="md-header__button md-logo"
+      aria-label="{{ config.site_name }}"
+    >
+      {% include "partials/logo.html" %}
+    </a>
+    <label class="md-header__button md-icon" for="__drawer">
+      {% include ".icons/material/menu" ~ ".svg" %}
+    </label>
+    <div class="md-header__title" data-md-component="header-title">
+      <div class="md-header__ellipsis">
+        <div class="md-header__topic">
+          <span class="md-ellipsis">
+            {{ config.site_name }}
+          </span>
+        </div>
+        <div class="md-header__topic" data-md-component="header-topic">
+          <span class="md-ellipsis">
+            {% if page and page.meta and page.meta.title %}
+              {{ page.meta.title }}
+            {% else %}
+              {{ page.title }}
+            {% endif %}
+          </span>
+        </div>
+      </div>
+    </div>
+    <div class="md-header__options">
+      <div class="md-header-nav__scheme md-header-nav__button md-source__icon md-icon">
+          <a
+            href="javascript:toggleScheme();"
+            title="Light mode"
+            class="light-mode"
+          >
+          {% set icon = "material/weather-sunny" %}
+          {% include ".icons/" ~ icon ~ ".svg" %}
+          </a>
+          <a
+            href="javascript:toggleScheme();"
+            title="Dark mode"
+            class="dark-mode"
+          >
+          {% set icon = "material/weather-night" %}
+          {% include ".icons/" ~ icon ~ ".svg" %}
+          </a>
+          <a
+            href="javascript:toggleScheme();"
+            title="System preference"
+            class="system-mode"
+          >
+          {% set icon = "material/theme-light-dark" %}
+          {% include ".icons/" ~ icon ~ ".svg" %}
+          </a>
+          <!-- <a
+            href="javascript:toggleScheme();"
+            title="Unknown scheme"
+            class="unknown-mode"
+          >
+          {% set icon = "material/help-circle" %}
+          {% include ".icons/" ~ icon ~ ".svg" %}
+          </a> -->
+      </div>
+    </div>
+    {% if "material/search" in config.plugins %}
+      <label class="md-header__button md-icon" for="__search">
+        {% include ".icons/material/magnify.svg" %}
+      </label>
+      {% include "partials/search.html" %}
+    {% endif %}
+    {% if config.repo_url %}
+      <div class="md-header__source">
+        {% include "partials/source.html" %}
+      </div>
+    {% endif %}
+  </nav>
+</header>

docs/theme/partials/libs.html

@@ -0,0 +1,2 @@
+<script src="{{ 'assets/pymdownx-extras/material-extra-theme-TVq-kNRT.js' | url }}" type="text/javascript"></script>
+<script src="{{ 'assets/pymdownx-extras/material-extra-3rdparty-E-i8w1WA.js' | url }}" type="text/javascript"></script>

docs/ultilities.md

@@ -0,0 +1,172 @@
+Utilities detail can be referred in the sub-pages of this section.
+
+## Prompt engineering UI
+
+![chat-ui](https://github.com/Cinnamon/kotaemon/assets/35283585/ac8f9aac-d853-4571-a48b-d866a99eaf3e)
+
+**_Important:_** despite the name prompt engineering UI, this tool allows DMs to test any kind of parameters that are exposed by AIRs. Prompt is one kind of param. There can be other type of params that DMs can tweak (e.g. top_k, temperature...).
+
+**_Note:_** For hands-on examination of how to use prompt engineering UI, refer `./examples/promptui` and `./examples/example2/`
+
+In client projects, AI developers typically build the pipeline. However, for LLM projects requiring Japanese and domain expertise in prompt creation, non-technical team members (DM, BizDev, and QALs) can be more effective. To facilitate this, "xxx" offers a user-friendly prompt engineering UI that AI developers integrate into their pipelines. This enables non-technical members to adjust prompts and parameters, run experiments, and export results for optimization.
+
+As of Sept 2023, there are 2 kinds of prompt engineering UI:
+
+- Simple pipeline: run one-way from start to finish.
+- Chat pipeline: interactive back-and-forth.
+
+### Simple pipeline
+
+For simple pipeline, the supported client project workflow looks as follow:
+
+1. [AIR] Build pipeline
+2. [AIR] Export pipeline to config: `$ kh promptui export <module.path.piplineclass> --output <path/to/config/file.yml>`
+3. [AIR] Customize the config
+4. [AIR] Spin up prompt engineering UI: `$ kh promptui run <path/to/config/file.yml>`
+5. [DM] Change params, run inference
+6. [DM] Export to Excel
+7. [DM] Select the set of params that achieve the best output
+
+The prompt engineering UI prominently involves from step 2 to step 7 (step 1 is normal AI tasks in project, while step 7 happens exclusively in Excel file).
+
+#### Step 2 - Export pipeline to config
+
+Command:
+
+```
+$ kh promptui export <module.path.piplineclass> --output <path/to/config/file.yml>
+```
+
+where:
+
+- `<module.path.pipelineclass>` is a dot-separated path to the pipeline. For example, if your pipeline can be accessed with `from projectA.pipelines import AnsweringPipeline`, then this value is `projectA.pipelines.AnswerPipeline`.
+- `<path/to/config/file.yml>` is the target file path that the config will be exported to. If the config file already exists, and contains information of other pipelines, the config of current pipeline will additionally be added. If it contains information of the current pipeline (in the past), the old information will be replaced.
+
+By default, all params in a pipeline (including nested params) will be export to the configuration file. For params that you do not wish to expose to the UI, you can directly remove them from the config YAML file. You can also annotate those param with `ignore_ui=True`, and they will be ignored in the config generation process. Example:
+
+```python
+class Pipeline(BaseComponent):
+    param1: str = Param(default="hello")
+    param2: str = Param(default="goodbye", ignore_ui=True)
+```
+
+Declared as above, and `param1` will show up in the config YAML file, while `param2` will not.
+
+#### Step 3 - Customize the config
+
+AIR can further edit the config file in this step to get the most suitable UI (step 4) with their tasks. The exported config will have this overall schema:
+
+```
+<module.path.pipelineclass1>:
+  params:
+    ... (Detail param information to initiate a pipeline. This corresponds to the pipeline init parameters.)
+  inputs:
+    ... (Detail the input of the pipeline e.g. a text prompt, an FNOL... This corresponds to the params of `run(...)` method.)
+  outputs:
+    ... (Detail the output of the pipeline e.g. prediction, accuracy... This is the output information we wish to see in the UI.)
+  logs:
+    ... (Detail what information should show up in the log.)
+```
+
+##### Input and params
+
+The inputs section have the overall schema as follow:
+
+```
+inputs:
+  <input-variable-name-1>:
+    component: <supported-UI-component>
+    params: # this section is optional)
+      value: <default-value>
+  <input-variable-name-2>:
+    ... # similar to above
+params:
+  <param-variable-name-1>:
+    ... # similar to those in the inputs
+```
+
+The list of supported prompt UI and their corresponding gradio UI components:
+
+```
+COMPONENTS_CLASS = {
+    "text": gr.components.Textbox,
+    "checkbox": gr.components.CheckboxGroup,
+    "dropdown": gr.components.Dropdown,
+    "file": gr.components.File,
+    "image": gr.components.Image,
+    "number": gr.components.Number,
+    "radio": gr.components.Radio,
+    "slider": gr.components.Slider,
+}
+```
+
+##### Outputs
+
+The outputs are a list of variables that we wish to show in the UI. Since in Python, the function output doesn't have variable name, so output declaration is a little bit different than input and param declaration:
+
+```
+outputs:
+  - component: <supported-UI-component>
+    step: <name-of-pipeline-step>
+    item: <jsonpath way to retrieve the info>
+  - ... # similar to above
+```
+
+where:
+
+- component: the same text string and corresponding Gradio UI as in inputs & params
+- step: the pipeline step that we wish to look fetch and show output on the UI
+- item: the jsonpath mechanism to get the targeted variable from the step above
+
+##### Logs
+
+The logs show a list of sheetname and how to retrieve the desired information.
+
+```
+logs:
+  <logname>:
+    inputs:
+      - name: <column name>
+        step: <the pipeline step that we would wish to see the input>
+        variable: <the variable in the step>
+      - ...
+    outputs:
+      - name: <column name>
+        step: <the pipeline step that we would wish to see the output>
+        item: <how to retrieve the output of that step>
+```
+
+#### Step 4 + 5 - Spin up prompt engineering UI + Perform prompt engineering
+
+Command:
+
+```
+$ kh promptui run <path/to/config/file.yml>
+```
+
+This will generate an UI as follow:
+
+![Screenshot from 2023-09-20 12-20-31](https://github.com/Cinnamon/kotaemon/assets/35283585/9ac1b95a-b667-42e7-b318-98a1b805d6df)
+
+where:
+
+- The tabs at the top of the UI corresponds to the pipeline to do prompt engineering.
+- The inputs and params tabs allow users to edit (these corresponds to the inputs and params in the config file).
+- The outputs panel holds the UI elements to show the outputs defined in config file.
+- The Run button: will execute pipeline with the supplied inputs and params, and render result in the outputs panel.
+- The Export button: will export the logs of all the run to an Excel files users to inspect for best set of params.
+
+#### Step 6 - Export to Excel
+
+Upon clicking export, the users can download Excel file.
+
+### Chat pipeline
+
+Chat pipeline workflow is different from simple pipeline workflow. In simple pipeline, each Run creates a set of output, input and params for users to compare. In chat pipeline, each Run is not a one-off run, but a long interactive session. Hence, the workflow is as follow:
+
+1. Set the desired parameters.
+2. Click "New chat" to start a chat session with the supplied parameters. This set of parameters will persist until the end of the chat session. During an ongoing chat session, changing the parameters will not take any effect.
+3. Chat and interact with the chat bot on the right panel. You can add any additional input (if any), and they will be supplied to the chatbot.
+4. During chat, the log of the chat will show up in the "Output" tabs. This is empty by default, so if you want to show the log here, tell the AI developers to configure the UI settings.
+5. When finishing chat, select your preference in the radio box. Click "End chat". This will save the chat log and the preference to disk.
+6. To compare the result of different run, click "Export" to get an Excel spreadsheet summary of different run.

docs/upload-package.md

@@ -0,0 +1,24 @@
+Devpi server endpoint (subjected to change): https://ian_devpi.promptui.dm.cinnamon.is/root/packages
+
+Install devpi-client
+
+```bash
+pip install devpi-client
+```
+
+Login to the server
+
+```bash
+devpi use <server endpoint> # set server endpoint provided above
+devpi login <user name> --password=<your password> # login
+```
+
+If you don't yet have an account, please contact Ian or John.
+
+Upload your package
+
+```bash
+devpi use <package name>\dev # choose the index to upload your package
+cd <your package directory which must contain a pyproject.toml/setup.py>
+devpi upload
+```