Baseline: EncT5

[bclavie]

No description provided.

[raphaelsty]

I would be happy to contribute on this one which seems great to start with !

Don't hesitate to tell me if I'm going the wrong way:

• We would add an extra argument namely T5 to the function build_model of glue.py script and create a dedicated t5_module.create_t5_classification.

• I have found a reference code available at t5-fine-tuning, would it be consistent with building a Baseline EnCT5?

[NohTow]

I think this issue refer to using EncT5 approach on T5-Pile (except for the large size) as a baseline.

Actually go ahead and eval Pile-T5 with the EncT5 treatment, as that's likely to be the best Encdec around. (except for the large size, where we'd use T5-1.1, since there seems to have been an issue with PileT5 for that size).

I'll ping you on the related discussion.

[bclavie]

Hey @raphaelsty ! Yes, @NohTow is correct, this is about setting up baselines with only the Encoder + the EncT5 recipe from the paper above (GitHub repo) for:

• PileT5-base
• T51.1-large (there's an unknown issue with PileT5-large where it performs surprisingly weakly)
• PileT5-XL
  To serve as a very strong comparison point for our own base/large/XL variants.