Issues

• OpenAI-compat: --model openai-chat-completions with full chat URL, not curator or vLLM

  mlfoundations/evalchemy#161 · Filter by author cursor opened on Aug 28, 2026

• [bug] GPQA-Diamond ground truth answers are all D

  mlfoundations/evalchemy#155 · Filter by author xiefiona opened on Feb 24, 2026
  2

• vLLM backend crashes with "please provide at least one prompt" when max_gen_toks exceeds model context window

  mlfoundations/evalchemy#152 · Filter by author Ali-Elganzory opened on Jan 3, 2026

• [bug] LiveCodeBenchv5

  mlfoundations/evalchemy#149 · Filter by author cyr0930 opened on Dec 23, 2025

• [Bug] google/codegemma-7b-it Evaluation Results All Zero

  mlfoundations/evalchemy#147 · Filter by author juyongjiang opened on Dec 16, 2025

• Only return loose result in IFEval?

  mlfoundations/evalchemy#145 · Filter by author luyy12 opened on Oct 28, 2025

• final answer extraction in reasoning model's output

  mlfoundations/evalchemy#144 · Filter by author maziye-azad opened on Oct 2, 2025

• LM Eval Tasks not found

  mlfoundations/evalchemy#143 · Filter by author AahilA opened on Sep 18, 2025

• Difference between self.max_tokens self.max_new_tokens in eval_instruct.py

  mlfoundations/evalchemy#142 · Filter by author richardbaihe opened on Sep 3, 2025

• Add logical reasoning task tests? (e.g., Enigmata data)

  mlfoundations/evalchemy#141 · Filter by author Siki-cloud opened on Aug 29, 2025

• Support for SWEBench

  mlfoundations/evalchemy#140 · Filter by author sravan500 opened on Aug 12, 2025
  1

• Request to add Debug mode

  mlfoundations/evalchemy#139 · Filter by author richardzhuang0412 opened on Jul 23, 2025
  1

• per prompt performance and public data

  mlfoundations/evalchemy#132 · Filter by author chanansh opened on Jun 27, 2025

• LCB benchmarks are duplicated

  mlfoundations/evalchemy#129 · Filter by author slimfrkha opened on Jun 13, 2025

• How many responses are sampled per query?

  mlfoundations/evalchemy#105 · Filter by author noobimp opened on Apr 8, 2025
  1

• How to change model's temperature ?

  mlfoundations/evalchemy#99 · Filter by author hrdkbhatnagar opened on Mar 25, 2025
  2

• A suggestion for evalchemy about LiveCodeBench

  mlfoundations/evalchemy#96 · Filter by author BeastyZ opened on Mar 20, 2025
  2

• Log samples to huggingface

  mlfoundations/evalchemy#89 · Filter by author RyanMarten opened on Mar 5, 2025

• How to log model response?

  mlfoundations/evalchemy#84 · Filter by author TundeAtSN opened on Feb 21, 2025
  4

• Support multi-node evaluation with vLLM

  mlfoundations/evalchemy#46 · Filter by author marianna13 opened on Dec 28, 2024

• lm-evaluation-harness dependency conflicts

  mlfoundations/evalchemy#45 · Filter by author penfever opened on Dec 21, 2024

• Race condition when running alpaca eval?

  mlfoundations/evalchemy#31 · Filter by author GeorgiosSmyrnis opened on Nov 23, 2024

• Integrate Berkeley Function Calling Leaderboard into Evalchemy

  mlfoundations/evalchemy#28 · Filter by author CharlieJCJ opened on Nov 21, 2024
  2