UnicodeDecodeError while running benchmark

[carina-ding]

Hi, Thank you for developing this amazing benchmark!

I created a virtual environment using anaconda and successfully installed crfm-helm (I followed the instructions from the official document https://crfm-helm.readthedocs.io/en/latest/installation/).

When I tried to run benchmarking with the example given in the document https://crfm-helm.readthedocs.io/en/latest/quick_start/, I followed exactly the instructions given. However, when I run "helm-run --conf-paths run_entries.conf --suite v1 --max-eval-instances 10", I got this error: "UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 5580: character maps to undefined". I attached some screenshots of traceback below.

Could anyone help me with this issue?

Thank you very much!

[yifanmai]

Hi @carina-ding, it looks like you have a malformed model_metadata.yaml file. The YAML parser is complaining that it is invalid YAML. You can check this by running this:

import dacite
import yaml

from helm.benchmark.model_metadata_registry import ModelMetadataList


path = "path/to/model_metadata.yaml"

with open(path, "r") as f:
    raw = yaml.safe_load(f)

model_metadata_list = dacite.from_dict(ModelMetadataList, raw)

[carina-ding]

Hi @yifanmai , thank you very much for your reply!

Sorry for the slight delay, as I was away last week.

I tried running the codes that you provided, it seems that it's producing the same error while running the codes.

I have attached some screenshots below, please let me know if I did anything wrong.

Thank you!

[yifanmai]

It looks like the problem is because you're running on Windows, which is not officially supported yet. We're planning to add support in the next couple of months. In the meantime, you could consider running it on Linux using WSL2.

[carina-ding]

Hi @yifanmai , thank you very much for your reply and suggestions! I will try to run it using WSL2.
Thank you!

[yifanmai]

Closing this issue due to staleness; feel free to reopen if you have further questions.

I opened a new issue to track adding Windows support. Feel free to subscribe to it for updates: #2907