Skip to content

add EN prompts for XCOPA (selected languages) - #795

Merged
VictorSanh merged 11 commits into
bigscience-workshop:eval-hackathonfrom
haileyschoelkopf:xcopa
Jul 18, 2022
Merged

add EN prompts for XCOPA (selected languages)#795
VictorSanh merged 11 commits into
bigscience-workshop:eval-hackathonfrom
haileyschoelkopf:xcopa

Conversation

@haileyschoelkopf

Copy link
Copy Markdown

This PR adds EN prompts (from COPA) for the XCOPA languages that are in Bloom's training data, so we can use it for training Bloom-T0++ (name TBD :P)

@Muennighoff @thomasw21 @lintangsutawika

@afaji

afaji commented Jul 15, 2022

Copy link
Copy Markdown

will add id translation

@Muennighoff Muennighoff left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! 👻

  1. There's a lot of double spaces. I didn't mark all, cuz maybe this is on purpose? As humans would still answer correctly.. (@VictorSanh, @stephenbach?)

  2. Do we need to repeat the same template for each lang? If it's easiest, let's do it, else I think we can use something like @lintangsutawika described here & just use the same template for different langs.

Comment thread promptsource/templates/xcopa/id/templates.yaml Outdated
Comment thread promptsource/templates/xcopa/id/templates.yaml Outdated
Comment thread promptsource/templates/xcopa/id/templates.yaml Outdated
Comment thread promptsource/templates/xcopa/id/templates.yaml Outdated
@VictorSanh

Copy link
Copy Markdown
Member
  • There's a lot of double spaces. I didn't mark all, cuz maybe this is on purpose? As humans would still answer correctly.. (@VictorSanh, @stephenbach?)

Yes this kind of noise/variance is totally fine. Some prompts have a single \n, others have triple \n, and we encouraged this kind of variations.
Note that a handful of tokenizers have pre-normalization rules that remove these, but this is independent of the prompts

VictorSanh and others added 4 commits July 18, 2022 10:36
Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com>
Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com>
Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com>
Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com>

@VictorSanh VictorSanh left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks! will merge as soon as the tests pass

@VictorSanh
VictorSanh merged commit ad0e59e into bigscience-workshop:eval-hackathon Jul 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants