Skip to content

Commit

Permalink
Merge pull request #45 from wandb/replit/deployment
Browse files Browse the repository at this point in the history
feat: Replit deployments
  • Loading branch information
parambharat authored Nov 10, 2023
2 parents fd92464 + e01836d commit c50ec35
Show file tree
Hide file tree
Showing 32 changed files with 2,933 additions and 963 deletions.
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -133,4 +133,4 @@ wandb/
artifacts/
data/
.idea/

.aider
7 changes: 5 additions & 2 deletions .replit
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
run = "poetry lock --no-update && poetry install && bash run.sh"
run = "bash run.sh"
entrypoint = "main.py"
modules = ["python-3.10:v18-20230807-322e88b"]

Expand All @@ -13,4 +13,7 @@ channel = "stable-23_05"
run = ["sh", "-c", "bash run.sh"]
build = ["sh", "-c", "bash build.sh"]
deploymentTarget = "gce"
ignorePorts = true

[[ports]]
localPort=8000
externalPort=80
46 changes: 24 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,23 @@
# wandbot

A question answering bot for Weights & Biases [documentation](https://docs.wandb.ai/).
This bot is built using [llama-index](https://gpt-index.readthedocs.io/en/stable/) and openai [gpt-4](https://openai.com/research/gpt-4).
Wandbot is a question-answering bot designed specifically for Weights & Biases [documentation](https://docs.wandb.ai/).
Leveraging the power of [llama-index](https://gpt-index.readthedocs.io/en/stable/) and OpenAI's [gpt-4](https://openai.com/research/gpt-4), it provides precise and context-aware responses
using a combination of [FAISS](https://github.com/facebookresearch/faiss) for RAG and OpenAI's [gpt-4](https://openai.com/research/gpt-4) for generating responses.


## Features

- The bot utilizes retrieval augmented generation with [FAISS](https://github.com/facebookresearch/faiss) backend to retrieve relevant documents and efficiently handle user queries and provides accurate, context-aware responses
- Periodic data ingestion with report generation for continuous improvement of the bot.: Checkout the latest data ingestion report [here](https://wandb.ai/wandbot/wandbot-dev/reportlist)
- Integrated with Discord and Slack, allowing seamless integration into popular collaboration platforms.
- Logging and analysis with Weights & Biases Tables for performance monitoring and continuous improvement.: Checkout the workspace for more details [here](https://wandb.ai/wandbot/wandbot_public)
- Uses a fallback mechanism for model selection when GPT-4 is unable to generate a response.
- Evaluation using a combination of metrics such as retrieval accuracy, string similarity, and model-generated response correctness
- Want to know more about the custom system prompt used by the bot?: Checkout the full prompt [here](data/prompts/chat_prompt.json)
- Wandbot employs Retrieval Augmented Generation with a [FAISS](https://github.com/facebookresearch/faiss) backend, ensuring efficient and accurate responses to user queries by retrieving relevant documents.
- It features periodic data ingestion and report generation, contributing to the bot's continuous improvement. You can view the latest data ingestion report [here](https://wandb.ai/wandbot/wandbot-dev/reportlist).
- The bot is integrated with Discord and Slack, facilitating seamless integration with these popular collaboration platforms.
- Performance monitoring and continuous improvement are made possible through logging and analysis with Weights & Biases Tables. Visit the workspace for more details [here](https://wandb.ai/wandbot/wandbot_public).
- Wandbot has a fallback mechanism for model selection, which is used when GPT-4 fails to generate a response.
- The bot's performance is evaluated using a mix of metrics, including retrieval accuracy, string similarity, and the correctness of model-generated responses.
- Curious about the custom system prompt used by the bot? You can view the full prompt [here](data/prompts/chat_prompt.json).

## Installation

The project uses `python = ">=3.10.0,<3.11"` and uses [poetry](https://python-poetry.org/) for dependency management. To install the dependencies:
The project is built with Python version `>=3.10.0,<3.11` and utilizes [poetry](https://python-poetry.org/) for managing dependencies. Follow the steps below to install the necessary dependencies:

```bash
git clone [email protected]:wandb/wandbot.git
Expand Down Expand Up @@ -43,7 +45,7 @@ These datasets are also stored as wandb artifacts in the project defined in the

### Running the Q&A Bot

You will need to set the following environment variables:
Before running the Q&A bot, ensure the following environment variables are set:

```bash
OPENAI_API_KEY
Expand All @@ -59,17 +61,17 @@ WANDB_PROJECT="wandbot-dev"
WANDB_ENTITY="wandbot"
```

Then you can run the Q&A bot application, use the following commands:
Once these environment variables are set, you can start the Q&A bot application using the following commands:

```bash
(poetry run uvicorn wandbot.api.app:app --host="0.0.0.0" --port=8000 > api.log 2>&1) & \
(poetry run python -m wandbot.apps.slack > slack_app.log 2>&1) & \
(poetry run python -m wandbot.apps.discord > discord_app.log 2>&1)
```

Please refer to the [run.sh](./run.sh) file in the root of the repository for more details on commands related to installing and running the bot.
For more detailed instructions on installing and running the bot, please refer to the [run.sh](./run.sh) file located in the root of the repository.

This will start the chatbot applications - the api, the slackbot and the discord bot, allowing you to interact with it and ask questions related to the Weights & Biases documentation.
Executing these commands will launch the API, Slackbot, and Discord bot applications, enabling you to interact with the bot and ask questions related to the Weights & Biases documentation.

### Evaluation

Expand All @@ -84,14 +86,14 @@ cd wandbot
poetry run python -m eval
```

## Implementation Overview
## Overview of the Implementation

1. Document Embeddings with FAISS
2. Building the Q&A Pipeline with llama-index
3. Model Selection and Fallback Mechanism
4. Deploying the Q&A Bot on FastAPI, Discord and Slack
5. Logging and Analysis with Weights & Biases Tables
6. Evaluation of the Q&A Bot
1. Creating Document Embeddings with FAISS
2. Constructing the Q&A Pipeline using llama-index
3. Selection of Models and Implementation of Fallback Mechanism
4. Deployment of the Q&A Bot on FastAPI, Discord, and Slack
5. Utilizing Weights & Biases Tables for Logging and Analysis
6. Evaluating the Performance of the Q&A Bot

You can track the bot usage in the following project:
You can monitor the usage of the bot in the following project:
https://wandb.ai/wandbot/wandbot_public
Loading

0 comments on commit c50ec35

Please sign in to comment.