Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evaluation Dataset mentioned in Hugging GPT paper is not available #208

Open
ssdasgupta opened this issue Jun 8, 2023 · 2 comments
Open

Comments

@ssdasgupta
Copy link

As mentioned in the paper - "Furthermore, we also invite some expert annotators to label task planning for some complex requests (46 examples) as a high-quality human annotated dataset. We also plan to further improve the quality and quantity of this dataset to better help us to evaluate the LLM capability in planning, which leaves as future work.", are you planning to release the evaluation dataset? Or if it is there already in the repository, could you send me the folder location?

Thanks.

@StillKeepTry
Copy link
Collaborator

@ssdasgupta We are currently working with our labeling teams to iteratively improve the quality of this dataset and our legal team to ensure compliance of the dataset release. We will release a work about this dataset in the future. Please be patient.

@Belonng
Copy link

Belonng commented Aug 16, 2024

Hello @StillKeepTry ,

I hope you’re doing well. I wanted to kindly follow up on the status of the evaluation dataset mentioned in the previous discussion. I understand that the team has been working on improving the quality and ensuring legal compliance. Could you please provide any updates on when we might expect the release of this dataset?

This dataset would be extremely valuable for my work, and I’m sure many others in the community are also eagerly awaiting it. Your efforts are greatly appreciated.

Thank you for your time!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants