Feat v1.1 Updates to chat client, retrieval, and evauations #54
Conversation
| query: str | ||
| language: str = "en" | ||
| initial_k: int = 50 | ||
| top_k: int = 5 |
There was a problem hiding this comment.
Maybe adding tags here to do filtering on the app slide, after-retrieval but before-response
There was a problem hiding this comment.
Added include_tags and `exclude_tags keys for both inclusions and exclusions
| import regex as re | ||
|
|
||
|
|
||
| class MrkdwnFormatter: |
There was a problem hiding this comment.
Not a typo, Mrkdwn is actually a formatting language used by slack
|
|
||
| class MultiLabel(BaseModel): | ||
| label: Labels = Field(..., description="The label for the query") | ||
| reasoning: str = Field( |
There was a problem hiding this comment.
I think reasoning should come before label to allow the model to think first
| "support", | ||
| Labels.BEST_PRACTICES.value: "The query is related to best practices for using Weights & Biases. Answer the query " | ||
| "and provide guidance where necessary", | ||
| Labels.COURSE_RELATED.value: "The query is related to a Weight & Biases course and/or skill enhancement. Answer " |
There was a problem hiding this comment.
maybe specify that the W&B courses are "Machine Learning" and "AI" courses
| { | ||
| "role": "system", | ||
| "content": ( | ||
| "You are a Weights & Biases support manager. Your goal is to enhance the user query by adding " |
There was a problem hiding this comment.
"Your goal is to enhance...", maybe something a little more specific related to what the end goal is
"Your goal is to refine, clarify and augment the user query before it is passed to another AI support assistant. Please add the following information to the query: ...."
| return enhanced_query | ||
|
|
||
|
|
||
| class QueryHandlerConfig(BaseSettings): |
There was a problem hiding this comment.
should this live in a separate config file?
| return all_nodes | ||
|
|
||
|
|
||
| class RetrieverConfig(BaseSettings): |
There was a problem hiding this comment.
should this be in a separate config?
| Your job is to judge the factful consistency of the generated answer with respect to the document. | ||
| - An answer is considered factually consistent if it contents can be inferred solely from the provided documentation. | ||
| - if an answer contains true information, if the information is not found in the document, then the answer is factually inconsistent. | ||
| - The generated answer must provide only correct information according to the documentation. |
There was a problem hiding this comment.
"correct information" feels ambiguous, maybe clarify to something more like "provide only information found in the documentation"
| safe_parse_eval_response, | ||
| ) | ||
|
|
||
| SYSTEM_TEMPLATE = """You are a Weight & Biases support expert tasked with evaluating the factful consistency of answers to questions asked by users to a technical support chatbot. |
There was a problem hiding this comment.
in general, should the system templates be broken out into a separate file(s)?
| {{ | ||
| "reason": <<Provide a brief explanation for your decision here>>, | ||
| "score": <<Provide a score as per the above guidelines>>, | ||
| "decision": <<Provide your final decision here, either 'relevant', or 'irrelevant'>> |
There was a problem hiding this comment.
same decision comment as above
| except Exception as e: | ||
| print(e) | ||
| print(eval_response) | ||
| score = 0 |
There was a problem hiding this comment.
maybe set to -1 just so its extremely obvious this isn't a valid score returned by the LLM?
| spec = row_dict["spec"] | ||
| content = json.loads(spec) | ||
| markdown_content = self.parse_content(content) | ||
| output["content"] = ( |
There was a problem hiding this comment.
i think it'd be useful to put the title ("display_name") and description in as their own standalone kv pairs, it could be a useful bit of metadata to have
|
lgtm! |
Overview
This PR introduces enhancements and fixes to the chat client, focusing on query handling, You.com retrieval, and various formatting and compatibility improvements.
Key Enhancements
Additional Features