Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: IFEval dataset introduces the correct langdetect expectation path #1451

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

catohzs
Copy link

@catohzs catohzs commented Aug 23, 2024

Motivation

When I tested the IFEval dataset locally, I found that this error would occur during the evaluation.

Traceback (most recent call last):
File "C:\Users\12072.projects\PyCharmProjects\evaluation\executor\opencompass\opencompass\tasks\openicl_eval.py", line 364, in
inferencer.run()
File "C:\Users\12072.projects\PyCharmProjects\evaluation\executor\opencompass\opencompass\tasks\openicl_eval.py", line 110, in run
self._score()
File "C:\Users\12072.projects\PyCharmProjects\evaluation\executor\opencompass\opencompass\tasks\openicl_eval.py", line 219, in _score
result = icl_evaluator.score(**preds)
File "C:\Users\12072.projects\PyCharmProjects\evaluation\executor\opencompass\opencompass\datasets\IFEval\ifeval.py", line 49, in score
example = test_instruction_following_strict(input, pred)
File "C:\Users\12072.projects\PyCharmProjects\evaluation\executor\opencompass\opencompass\datasets\IFEval\evaluation_main.py", line 77, in test_instruction_following_strict
if response.strip() and instruction.check_following(response):
File "C:\Users\12072.projects\PyCharmProjects\evaluation\executor\opencompass\opencompass\datasets\IFEval\instructions.py", line 166, in check_following
except langdetect.LangDetectException as e:
AttributeError: module 'langdetect' has no attribute 'LangDetectException'. Did you mean: 'lang_detect_exception'?

This seems to be caused by introducing the wrong path

Modification

I modified this path to be correct to avoid the problem of evaluation failure.

@tonysy
Copy link
Collaborator

tonysy commented Aug 23, 2024

I can successfully import LangDetectException on linux by:

from langdetect import LangDetectException 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants