Repository navigation
Conversation
Qubitium
changed the title
|
Thank you for your contribution 🤗! CI Security Gate — automatic approval blockedThis PR was not automatically approved for CI because the security gate failed. Possible reasons:
See the workflow run for the exact violations. A maintainer can review and manually approve CI if a finding is a false positive. |
|
cc @SunMarc |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: ggml |
CI recapDashboard: View test results in Grafana
|
| def _version_meets_minimum(is_available: bool, package_version: str, min_version: str) -> bool: | ||
| if not is_available: | ||
| return False | ||
|
|
||
| try: | ||
| return version.parse(package_version) >= version.parse(min_version) | ||
| except version.InvalidVersion: | ||
| logger.debug(f"Skipping optional package with invalid version string: {package_version!r}") | ||
| return False | ||
|
|
||
|
|
There was a problem hiding this comment.
why touch this ? what issues are you getting with the previous behavior ?
There was a problem hiding this comment.
@SunMarc @Rocketknight1 This WIP PR is no longer needed. It was previously written assuming we (gpt-qmodel) would use the hf gguf bindings to load prism model for infernce so I try to push as much up-stream as possible. But since then, I have decided to re-implement the prism gguf specific kernel for cuda (1/2 bit) so there is no need to patch or depend on HF. I apologize as I have totally forgotten about this PR. Will close it for now.
| def _dequantize_gguf_tensor(data: np.ndarray, tensor_type, dequantize_fn) -> np.ndarray: | ||
| try: | ||
| return dequantize_fn(data, tensor_type) | ||
| except NotImplementedError: | ||
| if _is_prism_q1_0_g128(tensor_type): | ||
| return _dequantize_prism_q1_0_g128(data) | ||
| raise | ||
|
|
||
|
|
There was a problem hiding this comment.
not ideal to have this. can't we have it in dequantize ?
What does this PR do?
Add PrismML 1bit (gguf based, group-size 128) model inference suppport.
Code Agent Policy
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Someone, just not yet. ..WIP