How to perform common sense reasoning task with GPT-2?

[h56cho]

Hello,

I am new to NLP so I have lots of questions.
I am interested in carrying out common sense reasoning task with GPT-2, for example, with Winograd Schema Challenge dataset.

Q1. How should I tokenize the Winograd Schema Challenge dataset to process it with GPT-2 (with the double heads model, for instance)? Can someone please give me an example?

Q2. Can GPT2DoubleHeadsModel be used to conduct common sense reasoning task with Winograd Schema Challenge dataset?

Thank you,

[zhangguanheng66]

There is a paper about commonsense reasoning here.

I didn't take a look at Winograd Schema Challenge dataset. Based on GPT-2, you need two sentences for the two heads (language modeling and classification). Depending on the structure of the raw data, you may write some custom pipeline (see the one in TextClassification dataset). The raw data are saved as follow:

0, This is a sport news.
1, the golf star is going to retire.

Where "0" is the label and "This is a sport news." is the text. We call unicode_csv_reader to read the csv file with two heads

[h56cho]

Hello,

Thank you very much for your reply.
So suppose the raw data is structured like below:

q, context, option0, option1, correctLabel
What does 'It' refer to?, Here is a string and a needle. It is very sharp so be careful, needle, string, 0
Choose the next dialog, How old are you?, My name is Rick, I am seventeen, 1

so to process this type of data with GPT-2 double heads model, do I need to rearrange the data like below, and feed the rearranged sequences into GPT2 for training?:

q, context, option0, option1, correctLabel
What does 'It' refer to?, Here is a string and a needle. It is very sharp so be careful, needle
What does 'It' refer to?, Here is a string and a needle. It is very sharp so be careful, string
Choose the next dialog, How old are you?, My name is Rick
Choose the next dialog, How old are you?,  I am seventeen

(notice that I got rid of the proper label at the end)
one of the things that I am unsure about is when creating two sentences from the raw dataset, how I should place the label, as labels are needed for training.

In my example, how should I rearrange the original text to use the double headed GPT2 for common sense tasks?

Thank you,

[zhangguanheng66]

I may not correct. IMO, you need to split the raw text into q, context, option0, option1, correctLabel. You need to write a very custom pipeline for this part (similar to _csv_iterator). After this step, you need to build vocab (like this) and numericalize q, context, option0, option1.

Then, send context to the language model head in GPT-2 and q to the classification head. The output side of the classification head is based on the label. Again, IMO.