Repository navigation
[INTEGRATION] Add GPTQModel support into transformers + optimum + peft #729
Description
Function Uptreaming
optimum
2064 <-- MERGED
transformers
35012 <-- MERGED
peft
2247 <-- MERGED
Tests
optimum
test_quantization
RUN_SLOW=1 pytest tests/gptq/test_quantization.py
- cpu tests
- cuda tests
transformers
test_gptq
RUN_SLOW=1 pytest tests/quantization/gptq/test_gptq.py
- cpu tests
- cuda tests
peft
PeftGPTQGPUTests
pytest tests/test_gpu_examples.py::PeftGPTQTests and pytest tests/test_common_gpu.py::PeftCommonTests::test_lora_gptq_quantization_from_pretrained_safetensors
- cpu tests
- cuda tests
I suppose we don't need new unit tests for gptq in HF, just need to pass all gptq tests with gptqmodel lib. Please help to confirm it.
cc @Qubitium @SunMarc
Activity
SunMarc commentedon Dec 3, 2024
Sounds good to me @jiqing-feng ! Also, after discussing a bit with other members at HF, I think it is better for now to put any mention of deprecating AutoGPTQ for now. If both libraries are installed, we can use GPTQModel library and put a clear warning sign. Thanks for working on this !
Qubitium commentedon Dec 3, 2024
@jiqing-feng Internal GPTQModel code refractor to support hf/optimum passing internal tests.
Transformer/Optimum PRs needs the following merges:
https://github.com/jiqing-feng/transformers/pull/2/files
https://github.com/jiqing-feng/optimum/pull/2/files
@SunMarc We will do more testing tomorrow and let you know when everything is kosher so you can do review on proper code that is passing all tests.
Update: above 2xPR both Num.2 for transformer/optimum have been merged.
Qubitium commentedon Dec 4, 2024
Status update. We have started testing the above Transformer/Optimum/Peft tests under Nvidia/cuda GPU using gptqmodel[main] + the following PRs:
https://github.com/jiqing-feng/transformers/pull/3/files
https://github.com/jiqing-feng/optimum/pull/3/files
Qubitium commentedon Dec 4, 2024
@jiqing-feng @SunMarc One issue we found is all the gptq tests in optimum are super flaky not only between transformer/torch version but also between gptqmodel vs auto-gptq. In gptqmodel we moved away from string compare and directly do eval harness benchmarks and check for regression floor value on a fixed benchmark. This is not going to be addressed in this PR but needs to addressed in future PRs.
https://github.com/jiqing-feng/transformers/pull/3/files#diff-5f4148e9e983fe6fb9bd7a8eb1e3e8fe65971cabb1cf2d4ce9b18885c07b7d44
Qubitium commentedon Dec 4, 2024
Update: All 3 pending (not-yet ready) PRs we will are testing for @jiqing-feng PRs
Transformers: https://github.com/jiqing-feng/transformers/pull/3/files
Optimum: https://github.com/jiqing-feng/optimum/pull/3/files
Peft: https://github.com/jiqing-feng/peft/pull/1/files
Qubitium commentedon Dec 4, 2024
Final update for today. We are trying to find min pain-point for fixing peft/gptqmodel non-ipex paths. There are solutions but none are great since only three kernels (torch/cuda/triton) actually supports accelerated + trainable forward.
But selecting right kernel in transformer load, then optimum quantize, followed finally by peft/training mode are very disjointed steps in transformer: there appears to be no auto-hook to register so gptqmodel internals can switch non trainable kernel into trainable kernel.
Can a nn.module actually register a hook to know its going into training mode beyond a boolean self.trainable state that is only checked in forward()?
We will test the best available method tomrrow. Ran out of time today.
[-]Integrate gptqmodel into HF[/-][+][INTEGRATION] Add GPTQModel support into transformers + optimum + peft[/+]Qubitium commentedon Dec 5, 2024
Best-fit solution has been found. We are refactoring gptqmodel code at the moment so peft compat can be added cleanly.
Qubitium commentedon Dec 5, 2024
Almost there. All code 99.8% ready for all 3 prs. Internal testing starting. @jiqing-feng Once all our tests pass, I will let you know to merge to your 3 PRS and before we can start final round of testing.
Tracking:
Transformers: https://github.com/jiqing-feng/transformers/pull/3/files
Optimum: https://github.com/jiqing-feng/optimum/pull/3/files
Peft: https://github.com/jiqing-feng/peft/pull/1/files
Qubitium commentedon Dec 5, 2024
@jiqing-feng All 3 prs passing all cpu/gpu tests internally here. There are some string output mismatches but that's random/variability on different cpu. Cpu tests may fail, tensors on cuda, when cpu tests are run in env where both cpu + cuda device are exposed despite
device_mapis set to cpu only. I think this is hf bug but we need to check.Once these 3 prs are merged to the larger PRS, I will write a length explanation on some of the obvious and not so obvious small/large changes we require/pushed.
jiqing-feng commentedon Dec 5, 2024
All merged, will check it again on CPU.
Qubitium commentedon Dec 8, 2024
All tests passing. Both gptqmodel internal tests and transformer/optimum/peft tests passing for cpu/xpu/cuda.
Qubitium commentedon Dec 11, 2024
GPTQModel v1.4.0 has been released. If there are any change required for the above upstream PRs, we will make changes and cut a new release.
Qubitium commentedon Jan 31, 2025
@jiqing-feng Closing this as completed.
add original convert file and support for the newest llama.cpp (Model…