Adding R-Bench MCQ Section #509
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kennymckormick
added a commit
to TousenKaname/VLMEvalKit
that referenced
this pull request
Oct 10, 2024
* Adding R-Bench MCQ Section * [Minor] Fix Lint * [Minor] Update Lint --------- Co-authored-by: unknown <[email protected]> Co-authored-by: kennymckormick <[email protected]>
kennymckormick
added a commit
that referenced
this pull request
Oct 11, 2024
* add GMAI_MMBench Test * add GMAI_MMBench Test * add GMAI_MMBench Test * [Model] add support for Llama-3.2-11B/90B-Vision-Instruct (#490) * add llama3.2-vision-instruct * update max_new_tokens * update llama3.2 vision with self prompt and parameters * add XinYuan-VL-2B (#488) * update README * [Fix] Fix Y/N type error (#506) * [Model] Update Step-1V * [Minor] Add warning for benchmarks test splits without GT answers: InfoVQA_TEST, Q-Bench1_TEST, etc. * [Model] Add Kosmos2 (#494) * update mmbench-video repo_id * [Minor] Fix format issue (#508) * [Dataset] Adding R-Bench MCQ Section (#509) * Adding R-Bench MCQ Section * [Minor] Fix Lint * [Minor] Update Lint --------- Co-authored-by: unknown <[email protected]> Co-authored-by: kennymckormick <[email protected]> * add GMAI_MMBench benchmark * Fix Lint * [Feature]: Add POINTS (#501) * [Feature]: Add points-yi-1.5 * [Feature]: Update docstring * [Fix]: Fix model name * [Fix]: Set use_fast to false * [Feature]: Add version check * [Feature]: Update model id * [Fix]: Update id * [Feature]: Update config * [Fix]: Fix lint * [Feature]: Add support for dataset is None --------- Co-authored-by: bensenliu <[email protected]> * remove duplicate --------- Co-authored-by: Xinyu Fang <[email protected]> Co-authored-by: thomas-yanxin <[email protected]> Co-authored-by: Haian Huang(深度眸) <[email protected]> Co-authored-by: kennymckormick <[email protected]> Co-authored-by: tackhwa <[email protected]> Co-authored-by: Chunyi Li <[email protected]> Co-authored-by: unknown <[email protected]> Co-authored-by: kennymckormick <[email protected]> Co-authored-by: Yuan Liu <[email protected]> Co-authored-by: bensenliu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Chunyi Li, providing R-Bench only for the (MCQ) task.
For the whole project with MCQ/VQA/CAP task please check https://github.com/Q-Future/R-Bench.