You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Would an EvalPort-format export of the MATH dataset be welcome in EvalPort's benchmarks/? #33
Hi — I maintain EvalPort, an open JSON spec for portable LLM eval datasets and results. benchmarks/ already includes GSM8K; MATH is a natural complement (harder, worked-solution format) and isn't in there yet.
I loaded the dataset this repo's README points to (the HF mirror linked as "MATH dataset here") and read the real fields: problem, level (e.g. "Level 5"), type (e.g. "Algebra"), solution (the full worked solution, with the final answer inside a \boxed{...} — which is how this repo's own modeling/math_equivalence.py extracts and compares answers).
That maps onto EvalPort cleanly: problem -> TestCase.input, the \boxed{} extraction of solution -> TestCase.expected_output, level/type -> metadata, graded with a grader mirroring math_equivalence.py's own equivalence check rather than naive string match.
Would you be open to an EvalPort-format export living in EvalPort's benchmarks/, with attribution and license preserved? Happy to do the conversion and open a PR for review first. No worries if this isn't a fit for a repo that's mostly loaders/eval code at this point.
Activity