Repository navigation
add HyperClovaX Vision - #44314
Merged
Merged
add HyperClovaX Vision#44314
Conversation
Contributor
Author
|
When I ran the test in my environment, hcx test ran normally without any failures.
I think the work will proceed in this manner? |
Contributor
Author
|
Hmm... In my environment, |
…iteration and using cache
…orConditionalGeneration
…d processor classes
…template application
…ing in HyperClovaXProcessor
Collaborator
|
Yep don't worry we'll fix this one on main! |
zucchini-nlp
reviewed
Mar 2, 2026
zucchini-nlp
left a comment
zucchini-nlp
left a comment
Member
There was a problem hiding this comment.
Hey @jp1924, thanks a lot for the PR!
I think we need to move all the code to modular file first, since there is a lot of copied stuff from other models. I left you comments below about where exactly can copy from. Ping me for another review when the modular is ready
…deo processor classes.
… to HCXVisionConfig and remove unnecessary code.
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, hyperclovax_vision_v2, qwen2_5_vl |
Collaborator
|
run-slow: hyperclovax_vision_v2 |
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, hyperclovax_vision_v2, qwen2_5_vl |
Contributor
Nvidia CIThis comment contains models: ["models/hyperclovax_vision_v2"] |
Contributor
AMD CIThis comment contains models: ["models/hyperclovax_vision_v2"] |
vasqu
enabled auto-merge
September 11, 2026 16:48
Collaborator
|
Merging now 🤗 Thanks a lot @jp1924 and @bigshanedogg ❤️ |
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, hyperclovax_vision_v2, qwen2_5_vl |
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, hyperclovax_vision_v2, qwen2_5_vl |
Contributor
CI Results (AMD)Commit Info
Model CI Report❌ 1 new failed tests from this PR 😭
|
vasqu
disabled auto-merge
September 11, 2026 16:57
vasqu
enabled auto-merge
September 11, 2026 16:57
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, hyperclovax_vision_v2, qwen2_5_vl |
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 11, 2026
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, hyperclovax_vision_v2, qwen2_5_vl |
Contributor
Author
|
Sorry for the late reply. |
Contributor
CI recapDashboard: View test results in Grafana |
Contributor
Author
Finally... it's finally merged! I'll open the video_duration follow-up issue today. @zucchini-nlp @vasqu @bigshanedogg @ArthurZucker |
sbucaille
pushed a commit
to sbucaille/transformers
that referenced
this pull request
Sep 16, 2026
* add: hcx vision model
* add: modeling test
* fix: update vision model selection logic for HCXVisionModel
* fix: prepare_inputs_for_generation to handle first iteration logic
* fix: IndentationError:
* fix: lint
* fix: update conditional checks for model type comparisons
* fix: set pixel_values and pixel_values_videos to None when not first iteration and using cache
* fix: remove unnecessary coding declaration from HyperClovaX configuration file
* fix: streamline imports in test_modeling_hyperclovax_vision.py
* fix: add hyperclovax model type to SPECIAL_MODEL_TYPE_TO_MODULE_NAME mapping
* fix: add docstrings for image and video grid dimensions in HCXVisionForConditionalGeneration
* fix: update docstrings for parameters in HyperClovaX configuration and processor classes
* test: add unit tests for HyperClovaXProcessor functionality and chat template application
* fix: update image processing logic and enhance multimodal token handling in HyperClovaXProcessor
* fix: update parameter types and docstrings in HyperClovaX configuration classes
* 'models/auto' change
* remove image processor
* Refactor HyperClovaX configuration classes and update type hints for clarity
* Change the class name to HCXVisionV2Processor and update the image/video processor classes.
* Refactor HCXVisionModel and related classes: rename HyperClovaXConfig to HCXVisionConfig and remove unnecessary code.
* Add checkpoint conversion mapping for hcxvision model
* Fix formatting issue in processor mapping section
* HyperClovaXForCausalLM -> HyperClovaXTextModel
* add hyperclovax_vision conversion mapping
* rm breakpoint()
* Remove unused imports and refactor
* add modular file
* HCXVisionConfig: remove model_type from kwargs in constructor
* HCXVisionModelTester: refactor imports and update config usage to HCXVisionConfig
* Add video support and update image handling in HCXVision integration tests
* fix: video in-place error
* refactor: simplify auto_map handling and update model_type for HCXVisionConfig
* fix: update sub_configs to use AutoConfig for text_config in HCXVisionConfig
* fix: update HCXVisionModel to use Qwen2_5_VLVisionConfig for vision configuration
* fix: add inv_freq initialization for Qwen2_5_VisionRotaryEmbedding in HCXVisionPreTrainedModel
* fix: remove mm_projector_type parameter from HCXVisionModelTester
* fix: add mm_projector_type parameter to HCXVisionModelTester and remove skipped tests
* fix: remove mm_projector_type parameter and skip unsupported tests in HCXVisionModelTester
* fix: remove unsupported test cases and add reverse loading mapping test in HCXVisionForConditionalGenerationTest
* fix: remove unused parameters from HyperClovaXConfig and HCXVisionConfig
* fix: format code for better readability in HCXVisionV2Processor
* fix: simplify weight renaming patterns in checkpoint conversion mapping for hyperclovax_vision
* fix: update copyright year and correct processor import in HyperClovaXProcessorTest
* fix: remove 'hcxvision' from VLMS model list
* feat: add 'hyperclovax_vision' to VLMS model list
* fix: remove 'hcxvision' conversion mapping from checkpoint conversion
* fix: update HyperClovaX vision model names in configuration and processing files
* fix: update weight renaming for hyperclovax_vision mapping
* fix: update copyright year and refactor HyperClovaX configuration classes
* refactor: clean up imports and remove unused classes in modular_hyperclovax_vision.py
* Refactor HyperClovaX model components and update class names
- Renamed classes for consistency: HyperCLOVAX* to HyperClovaX*.
- Consolidated RMSNorm and MLP implementations into HyperClovaXRMSNorm and HyperClovaXMLP.
- Updated attention mechanism to use externally computed position embeddings.
- Removed deprecated code and comments related to internal RoPE embeddings.
- Enhanced model initialization and weight handling.
- Improved documentation and type hints across the model classes.
* feat: model configuration mappings for Qwen2_5_VL and HyperClovaX
* feat: update model type to 'qwen2_5_vl_image' and add VisionTransformerPretrainedModel to exports
* fix: ename vision_config to vision_model in HCXVisionModel
* fix: rename vision_config to vision_model in HCXVisionModel
* add weight renaming mappings for HyperClovaX model
* fix: update HyperClovaX and HCXVision model type mappings
* fix: update output handling in HCXVisionModel and modify processor class attributes
* fix: update HCXVisionModel and processor class attributes for improved handling of hidden sizes and output processing
* feat: enhance HCXVisionModel with new methods for handling video and image features
* feat: refactor HCXVisionModel to extend VideoLlama3Model and streamline input handling
* feat: update HCXVisionModel and configuration for improved handling of vision and text features
* feat: remove redundant pixel_values type conversion in HCXVisionModel
* apply fix-repo
* feat: refactor HCXVisionModel to streamline forward method and enhance image/video feature handling
* feat: update hyperclovax model configurations and rename mappings for improved clarity
* feat: update weight renaming mappings for VLM and hyperclovax models
* feat: update HCXVisionModel to support optional video_merge_sizes parameter and add get_video_features method
* Edit review 3999491762
* refactor: simplify weight renaming patterns and enhance vision config handling
* refactor: hyperclovax_vision > hyperclovax_vision_v2
* fix: update config type in HCXVisionV2ForSequenceClassification to HCXVisionV2Config
* config & processor to modular
* fix test-error
* update processor test
* remove unused import
* fix: syntax error
* feat: add forward method for HCXVisionV2ForSequenceClassification with sequence classification support
* fix date
* remove hypercloax & fix test error
* fix `image_newline` failures
* fix `test_transformers_specific_model_import` failures
* rm hyperclovax docs
* refactor: simplify import statements in configuration_auto.py
* Update HyperCLOVAX Vision V2 documentation and code structure
- Updated release date in documentation for HyperCLOVAX Vision V2.
- Refined weight renaming in conversion mapping for HyperCLOVAX Vision V2.
- Added "vlm" to models with incorrect hub tokenizer class.
- Simplified forward method in HCXVisionV2VisionRotaryEmbedding.
- Enhanced HCXVisionV2Processor to support video metadata and improved input handling.
- Removed outdated test file for HCXVisionV2Processor.
* hyperclovax: update conversion mapping and simplify reverse loading test
* revert hyperclovax_vlm
* add comment to clarify model type usage in HCXVisionV2Config
* HCXVisionV2Model refactor inheritance to include HCXVisionV2PreTrainedModel
* Update src/transformers/models/hyperclovax_vision_v2/modular_hyperclovax_vision_v2.py
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* Update docs/source/en/model_doc/hyperclovax_vision_v2.md
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* Refactor HyperCLOVAX Vision V2 model and processor classes
- Updated class names and configurations to follow the new naming convention (e.g., HCXVisionV2 to HyperCLOVAXVisionV2).
- Enhanced documentation strings for clarity and consistency.
- Adjusted default token IDs for image and video processing.
- Removed unnecessary attributes and methods related to token type IDs and temporal RoPE.
- Improved the processor's handling of multimodal inputs, including image and video tokens.
- Updated test cases to reflect the new class names and configurations.
* refactor test and modular
* docs: update HyperCLOVAX Vision V2 model documentation and examples
* fix: remove hyperclovax tokenizer
* feat: Add HyperCLOVAX Vision V2 image and video processors
- Implemented HyperCLOVAXVisionV2ImageProcessor and HyperCLOVAXVisionV2ImageProcessorPil for image processing with smart resizing and normalization.
- Added HyperCLOVAXVisionV2VideoProcessor for video processing, including frame sampling and patchification.
- Updated auto image and video processing mappings to include HyperCLOVAX processors.
- Introduced modular structure for HyperCLOVAX Vision V2, ensuring compatibility with existing frameworks.
- Enhanced documentation and type hints for better clarity and usability.
* Add HyperCLOVAX vision processors to auto mappings and remove deprecated entries
* apply style
* Update HyperCLOVAX model documentation and processing logic
- Updated release date in English and Korean documentation.
- Refactored image and video processing methods for improved functionality.
- Added RGB conversion option in video processing.
* Add HyperCLOVAX Vision V2 documentation updates and new test cases
- Updated input format in documentation for images and videos to use "type": "image" and "type": "video".
- Removed deprecated entries and clarified usage notes regarding media inputs.
- Added new test files for image and video processing to ensure functionality of HyperCLOVAX Vision V2.
- Implemented tests for various input formats and processing methods, including handling of nested inputs and custom pixel settings.
* Remove out-of-scope kosmos-2/kosmos-2.5 processor auto-mapping entries
* Update src/transformers/models/hyperclovax_vision_v2/modular_hyperclovax_vision_v2.py
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* Update docs/source/en/model_doc/hyperclovax_vision_v2.md
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* Update src/transformers/models/hyperclovax_vision_v2/modular_hyperclovax_vision_v2.py
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* Update src/transformers/models/hyperclovax_vision_v2/modular_hyperclovax_vision_v2.py
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* Update src/transformers/models/hyperclovax_vision_v2/modular_hyperclovax_vision_v2.py
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* Update src/transformers/models/hyperclovax_vision_v2/modular_hyperclovax_vision_v2.py
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* Image/video processors:
- Delete HyperCLOVAXVisionV2{ImageProcessor,ImageProcessorPil,VideoProcessor} and their tests; they differed from Qwen2-VL only by the default pixel budget, which the hub already carries (preprocessor_config.json / video_preprocessor_config.json declare image/video_processor_type = Qwen2VL... with sizes in-config).
- Map hyperclovax_vision_v2 -> Qwen2VL{Image,Video}Processor in the auto overrides (same as exaone4_5); regenerate auto_mappings.
- test_processing: add _setup_video_processor with the checkpoint budget (min_pixels=3136, max_pixels=12845056) so shared mixin video tests match.
Modeling / modular:
- Remove HyperCLOVAXVisionV2ForSequenceClassification (no classifier ckpt).
- Drop the redundant _prepare_position_ids_for_generation override so the base GenerationMixin 1D path is used (the raise-AttributeError delete trick doesn't apply here since Exaone already strips it).
- Add the missing accepts_precomputed_kwargs import; use config.text_config.vocab_size in the loss (self.vocab_size was removed).
- Drop the dead second_per_grid_ts filter in the processor (Exaone's model_input_names never contains it).
Docs / tests:
- Use AutoProcessor in the en/ko examples.
- Pin revision="refs/pr/14" on the integration-test loading for CI validation against the hub PR (to be dropped once it lands on main).
- Convert plain asserts to unittest asserts; drop the now-passing test_load_with_mismatched_shapes skip.
- Revert the whitespace-only churn in qwen2_5_vl.md; keep only the Qwen2_5_VisionTransformerPretrainedModel autodoc (needed for auto).
* Fix check_repo: drop dangling - preprocess under HyperCLOVAXVisionV2Config autodoc
* Processor:
- Delete the HyperCLOVAXVisionV2Processor subclass; it was `Exaone4_5_Processor` with no override after the earlier second_per_grid_ts removal, so it added no behavior.
- Map hyperclovax_vision_v2 -> Exaone4_5_Processor in the processing auto overrides (MISSING_PROCESSOR_MAPPING_NAMES); regenerate auto_mappings (the derived processor entry is gone).
- Remove the generated processing_hyperclovax_vision_v2.py, drop it from the model __init__, and delete test_processing_hyperclovax_vision_v2.py (no bespoke processor left to test).
Modeling / modular:
- forward: read `outputs.last_hidden_state` by attribute instead of `outputs[0]`, and annotate the scaled-logits line (`# Key difference: additional logits scaling applied`). Note: kept the singular attribute `last_hidden_state`; the review suggestion's `last_hidden_states` does not exist on
BaseModelOutputWithPast.
Docs:
- Use AutoProcessor in the modular docstring example; drop the HyperCLOVAXVisionV2Processor autodoc block from the en/ko model cards.
Tests:
- Route the image/video URLs through url_to_local_path (both already in URLS_FOR_TESTING_DATA) so CI caches the media.
- Load media via the chat template instead of manual cv2 decode: switch the integration image/video prompts to the standard `{"type": "image"}` / `{"type": "video"}` contract and apply_chat_template(tokenize=True, return_dict=True); the refs/pr/14 template routes video by content type. Drop the
cv2/tempfile/requests/PIL/require_cv2/is_cv2_available/is_vision_available imports.
- Remove the stale `test_torch_export` skip override (it shadowed nothing; base uses the @slow test_torch_export_{static,dynamic,...}).
- Regenerate EXPECTED_TEXTS for the text/image/video integration tests against refs/pr/14 — the updated hub template emits real `<think>` reasoning (not the old empty-think), and the mime block now renders default filenames (a.jpg/a.mp4).
* Fix get_image_features output shape and projector weight mapping
get_image_features / get_video_features now return `pooler_output` as a
tuple of per-image 2D tensors (split by `grid_thw`), matching the contract
enforced by huggingface#46405 and already implemented by the parents. `forward`
re-concatenates with `torch.cat(..., dim=0)` before `masked_scatter`, so the
embedding fed to the LM is bit-identical to before (`cat(split(x)) == x`);
generation output is unchanged. Fixes test_get_{image,video}_features_output_*.
Collapse the two projector weight renamings into the single key the released
checkpoint uses (`model.vision_projector` -> `model.projector`). Two source
patterns mapping to one target made the reverse mapping ambiguous and failed
test_reverse_loading_mapping.
Drop the refs/pr/14 revision pin from the integration test and move the video
expected text to a per-hardware Expectations map (default + cuda 8.6), since
the only delta is one GPU-dependent token (this/the).
* fix docs date
* Clean up HyperCLOVAX Vision V2 modular
- Remove dead get_rope_index / get_vision_position_ids / compute_3d_position_ids stubs (no parent defines them, nothing calls them)
- Drop the redundant prepare_inputs_for_generation override (Exaone's handles it; generation output is unchanged)
- Fix the docstring example (auto processor/model, drop dtype)
* revert revision refs/pr/14
* For each class/method/attribute/config-field/test/doc-line you added, answer: *does an existing model, mixin, auto-map, or `auto_docstring` already provide this, and is it backed by a released checkpoint?* If any answer is "no checkpoint" or "already provided", delete it. That single pass removes ~80% of the comments this guideline is built from.
feat(hyperclovax_vision_v2): fill video_duration from decoded metadata
Re-declare HyperCLOVAXVisionV2Processor (previously auto-mapped to Exaone4_5_Processor with no override). The refs/pr/14 chat template reads video_duration from the message content dict and falls back to a <|video_duration|> placeholder when absent; this processor fills the placeholder from the metadata the video processor already decodes, handling mixed input where some videos supply the value and others don't.
- apply_chat_template: scan conversations before Jinja rendering, record which video content blocks are missing video_duration.
- _process_videos: compute duration per video from decoded metadata, filtered to the videos that actually need the placeholder.
- get_text_with_replacements: substitute <|video_duration|> with the computed value before delegating to the parent implementation.
Drop the now-redundant manual hyperclovax_vision_v2 -> Exaone4_5_Processor entry from MISSING_PROCESSOR_MAPPING_NAMES; the real class is now auto-discovered into auto_mappings.py. Add the missing HyperCLOVAXVisionV2Processor autodoc entry (en/ko).
Add processor tests for all-missing, explicit, and mixed video_duration cases, verified against the hub PR checkpoint (refs/pr/14) directly through HyperCLOVAXVisionV2Processor since AutoProcessor still resolves processor_class to Exaone4_5_Processor there until the hub PR lands.
* Add slow
* fix repo consistency
* apply repository consistency
* apply code quality
* docs(hyperclovax_vision_v2): fix contributed-on date to match latest commit
* fix(hyperclovax_vision_v2): resync after main merge
- Regenerate modeling_hyperclovax_vision_v2.py from the modular source; the main merge landed the vision rotary-embedding refactor upstream and the generated file was stale, missing HyperCLOVAXVisionV2VisionRotaryEmbedding's _init_weights override.
- Fix the contributed-on date in the model doc to match add_dates.py's expectation after the merge.
- Add the B200 (compute capability (10, 0)) expected-output entry to the video-generate integration test.
* - docs: use AutoModelForImageTextToText in en/ko examples & refresh contributed-on date
- processor: document video_duration stopgap, drop or [] guard
- tests: refactor processor slow tests, drop refs/pr/14 pins & update stale video_duration golden values
- fix: forward **kwargs to loss_function (shift_labels was ignored)
* use new cleanup
* thats on me
* load with BitsAndBytesConfig(load_in_4bit=True) instead (~39 GiB peak)
* apply check code
* test(hyperclovax_vision_v2): gate integration test on GPU memory
* add tests
* running variation with slight copy from og weights
* style
* are you good CI?
* test(hyperclovax_vision_v2): add rocm expectation for test_text_generate
---------
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
Co-authored-by: vasqu <antonprogamer@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Hello, Transformers team!
I submitted a PR to add naver-hyperclovax/HyperCLOVAX-SEED-Think-32B (hereafter HCX), developed by the Korean IT company Naver while executing the government's national AI model project.
The HCX code was written based on Transformer 4.52.4, leading to the following issues:
Moving to Transformer 5.0.0 significantly improved the readability and development convenience of the modeling code. We aim to leverage this to add the HCX model to transformers.
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@zucchini-nlp @yonigozlan @molbap