Repository navigation
Conversation
6 tasks
… Tensor (huggingface#48359) A number of `forward` methods declare a `tuple[...]` return annotation but return a bare `torch.Tensor`. Same class of issue as huggingface#45208, fixed for `Qwen3MoeSparseMoeBlock` in huggingface#45352. Each of the 29 classes touched here was instantiated and called with a minimal config, and the returned object's type observed to be `torch.Tensor`. Annotation only: no runtime behaviour is affected. Eleven annotations were edited in `modular_*.py` sources and the corresponding `modeling_*.py` files regenerated with `utils/modular_model_converter.py`; `utils/check_modular_conversion.py` and `utils/check_copies.py` both pass. Co-authored-by: Gustavo <gustavo.vp350@gmail.com>
…ggingface#48391) fix Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
* docs: update GLM 5.3 Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com> * docs: update toc Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com> * fix Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com> --------- Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
…output_mismatch` (list output differs (4)) (huggingface#48133) Co-authored-by: serge[bot] <serge[bot]@users.noreply.github.com> Co-authored-by: Tarek Ziade <tarek@ziade.org>
…ith `other` (other (6)) (huggingface#48425) * [serge] Fix 6 integration tests for model `seamless_m4t_v2` failing with `other` (other (6)) * input_features is a misleading name here! --------- Co-authored-by: serge[bot] <serge[bot]@users.noreply.github.com> Co-authored-by: eustlb <94853470+eustlb@users.noreply.github.com>
* del * fix repo * new models - fix modular
* re-order stufff a bit for easier navigation * CI fix
* feat: add neomme * tests: temporarily remove integration tests (need to make the weights public first) * chore: add aurelien as co-author Co-authored-by: uminaty <lacaurelien@yahoo.fr> * docs: fix neomme copyright year * docs: simplify neomme model loading * docs: use shared patch size description * docs: use shared image token description * test: use configured neomme initializer * refactor: serialize neomme layer types * refactor: use strict neomme config validation * refactor: modularize neomme image processor * refactor: inline neomme resize scale * refactor: standardize neomme image normalization * fix: match neomme image resize rounding * docs: clarify neomme resize defaults * test: reject invalid neomme template tasks * refactor: use processor input helpers * fix: keep neomme image grid internal * refactor: source neomme markers from tokenizer * test: use tokenizer markers in template * refactor: split neomme patch projection * refactor: reuse parameter-free rms norm * fix: remove obsolete attention mask workaround * refactor: use standard value embeddings * refactor: inline neomme rotary dimensions * refactor: store neomme residual scale * refactor: register neomme input embeddings * refactor: remove wrapper embedding accessors * style: move neomme shape comments * refactor: remove redundant config assignments * refactor: reuse base retrieval output fields * refactor: reuse nemotron mlp * fix: require exactly one model input * refactor: name retrieval outputs before return * refactor: narrow neomme weight initialization * refactor: remove redundant retrieval helpers * refactor: centralize neomme prompt formatting * refactor: name neomme retrieval heads * style: clarify neomme shape comments * style: inline neomme shape comments * refactor: use standard neomme rope layout * fix: harden neomme input handling - Preserve chat task state while forwarding processing options - Validate image layouts, model dimensions, and rotary settings - Clarify public documentation and expand regression coverage * style: tighten neomme comments * fix: validate neomme position ID shapes * refactor: simplify neomme processor helpers * fix: satisfy neomme repository checks * feat: support pixel area image size bounds - Add min_pixels and max_pixels to the shared SizeDict schema - Move NeoMME pixel budgets into the standard size configuration - Preserve max-side rounding while enforcing pixel floors and caps - Validate size keys and positive bound values - Cover serialization, resizing, rounding, and patch-count behavior * refactor: make neomme value embeddings mandatory * style: group neomme code by phase * style: clarify neomme tensor comments * style: tighten neomme test comments * refactor: validate neomme dimensions from field definitions * refactor: validate neomme rotary dimensions after standardization * docs: remove duplicate neomme image size docs * refactor: reuse standard neomme image normalization constants * style: compact neomme image processor defaults * refactor: validate neomme image kwargs before preprocessing * fix: compute neomme rotary embeddings outside autocast * docs: clarify neomme pixel area size bounds * refactor: rename neomme residual multiplier * refactor: remove legacy neomme rope scaling support * refactor: validate neomme image dimensions once * refactor: reuse laguna rotary initialization in neomme * refactor: configure neomme windows per layer * docs: simplify neomme config arguments * refactor: split neomme key and value projections * feat: add neomme image feature helpers * perf: batch same-shape neomme images * style: format neomme follow-up changes * perf: batch neomme patch conversion * refactor: keep neomme checkpoint conversion in exporter * refactor: reuse batched image patch conversion * fix: validate pixel area image size keys * refactor: remove unused neomme config defaults * refactor: reuse laguna rope frequencies * refactor: centralize neomme input validation * refactor: source neomme image token from tokenizer * refactor: use standard neomme image validators * refactor: simplify neomme patch count helper * refactor: iterate neomme layer configs directly * style: sync generated neomme files * refactor: load neomme windows from config * refactor: reuse neomme image token placeholder * refactor: move neomme validation into jinja * refactor: use standard neomme image hook * refactor: use standard neomme vlm flow * docs: use neomme chat template processing * docs: document neomme retrieval task * refactor: remove neomme retrieval scoring helper * test: remove private neomme checkpoint coverage * docs: clarify neomme retrieval scoring * refactor: remove neomme embedding tie flag * feat: support direct neomme retrieval inputs * fix: support generic neomme processor inputs * docs: refine neomme usage examples * refactor: reuse base neomme text processing * refactor: simplify neomme assistant mask policy * test: cover neomme processor without chat template * refactor: organize neomme processor methods * refactor: simplify neomme input finalization * docs: clarify neomme processor helpers * refactor: simplify NeoMME processor flow - Route retrieval formatting through apply_chat_template and delegate tokenization, padding, and keyword handling to ProcessorMixin - Keep only NeoMME-specific image placeholder expansion, two-axis positions, and validation for task markers and truncation - Remove direct task routing, manual sequence padding, and the duplicate query expansion processor setting - Update documentation and tests for the template-first API, malformed layouts, truncation, and ragged image batches - Add the PR review notes, chat template draft, and handoff context used to resolve feedback on huggingface#47992 * docs: simplify neomme retrieval example * style: align neomme processor variable names * fix: enforce NeoMME input and image limits * fix: allow task-free NeoMME chat templates * docs: simplify NeoMME encode helper * fix: remove hardcoded NeoMME processor defaults * style: clarify NeoMME implementation comments * docs: pad NeoMME retrieval batches * feat: support untied NeoMME word embeddings * docs: update neomme model doc * clean up a bit * docs: improve and fix snippets model doc * docs: clarify neomme processor docstring * fix: define neomme per-layer defaults * refactor: modularize neomme retrieval heads * refactor: modularize neomme attention output * refactor: require input ids for neomme * refactor: make tied neomme decoder explicit * chore: remove stale neomme fixme * fix: sync generated neomme model * fix: expose neomme word embeddings * fix: validate neomme layer types * fix: validate neomme image options * refactor: remove neomme image option validation * refactor: expose neomme unembedding projection * chore: sync neomme repository files * refactor: simplify neomme unembedding projection * refactor: build neomme positions with torch * refactor: require neomme input ids * refactor: rely on neomme forward signature * docs: update neomme contribution date --------- Co-authored-by: uminaty <lacaurelien@yahoo.fr> Co-authored-by: raushan <raushan@huggingface.co>
…VLM/Idefics2/Idefics3 (huggingface#47714) * Fix generate() crash when reusing image_hidden_states with pixel_values still set (SmolVLM/Idefics2/Idefics3) huggingface#47622 deleted these models' prepare_inputs_for_generation overrides, which used to null pixel_values whenever image_hidden_states was present. The new shared logic has no equivalent, so generate() now raises "You cannot specify both pixel_values and image_hidden_states at the same time" whenever a caller reuses a previous call's image_hidden_states while pixel_values is still in kwargs (the documented reuse pattern). Moves the precedence logic into each model's forward(), per the deleted override's own comment anticipating this move, instead of raising. Adds a regression test to each affected model. * Keep the pixel_values/image_hidden_states error, document reuse instead @zucchini-nlp: the ValueError is valid, not a regression -- there's no reason to pass both if precomputed image embeddings are available, and generate()'s own loop never produces this combination on its own (MULTIMODAL_INPUTS_TO_DROP_OUTSIDE_PREFILL already drops pixel_values after the first step regardless of image_hidden_states). It only fires when a caller explicitly hands both in on the first call. Revert forward() in SmolVLM/Idefics2/Idefics3 back to raising, with a slightly more actionable message. Document the actual reuse pattern (image_hidden_states alone, pixel_values cleared) in the image_hidden_states docstring of each model. Replace the three regression tests with one asserting the error still fires on both being set, and one asserting the correct reuse pattern actually works end to end. * Remove the new tests per review Superseded by proper shared-TestMixin coverage coming with next month's vision-LLM refactor for passing precomputed states. --------- Co-authored-by: verma8076 <vermoid@gmail.com>
…ders (huggingface#48407) * fix: Drop the position-indexed token type lookup in RoPE encoders * fix: Changes
…oder models (huggingface#48364) * fix: Enforce the auto-compile cache check for encoder-decoder models * change: Update generation/utils.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * fix: Changes --------- Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
…`other` (other (2)) (huggingface#48440) Co-authored-by: serge[bot] <serge[bot]@users.noreply.github.com>
…ggingface#47913) * This PR improves DFlash generation support for MuseGlimmer under tensor parallelism and reduces memory pressure during DFlash assisted decoding. Main changes: - Add selective hidden-state capture via `output_hidden_states_layers`, so DFlash only stores the main-model hidden states needed by the assistant instead of retaining all layers. - Update `DFlashTokenCandidateGenerator` to request only the configured target layers and prepare the assistant context from those sparse hidden states. - Support TP-sharded DFlash assistant context when the assistant encoder projection uses `rowwise` tensor parallelism, avoiding materializing the full concatenated context on every rank. - Fix DFlash with MuseGlimmer TP by letting its normed embedding module expose an `apply_norm=False` path, so DFlash can obtain raw noise-token embeddings without bypassing TP-aware embedding dispatch. - Add default tensor-parallel plans for MuseGlimmer text, vision, multimodal projection, LM head, and MuseGlimmer assistant modules. These changes allow DFlash to work correctly with MuseGlimmer/MuseGlimmer assistant in TP setups while reducing unnecessary hidden-state memory usage. Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> * fmt Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> * update Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> * update Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> * update Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> * update Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> * update Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> * return outside try --------- Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com>
…d output (value drift) (huggingface#48406) * [MiniCPMV4_6] Update test_small_model_vision_generation_batch expected output (value drift) Output drifted after huggingface#47625 refactored the Gated Delta Rule computation in `modeling_qwen3_5.py` (MiniCPMV4_6's text backbone). Verified by bisect: parent commit passes 3/3, the PR commit fails consistently. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Added expectations for b200 --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: remi-or <remi.pierre_o@orange.fr> Co-authored-by: Rémi Ouazan <83456801+remi-or@users.noreply.github.com>
* Add offload to gradient checkpointing Gradient checkpointing keeps one activation per checkpointed layer on the device, which is `layers x sequence x hidden` bytes and dominates at long sequence lengths. `offload` holds those in pinned host memory instead, through torch's `save_on_cpu`, so it follows whichever checkpointing path the model already uses. * Follow the repo conventions for the new option Document `offload` in `TrainingArguments.gradient_checkpointing_kwargs`, next to `every_n_layers`, and use the `backend_*` test helpers instead of `torch.accelerator` directly.
…huggingface#48228) * Init the process group with a load-scaled timeout for sharded loading Sharded loading of a TB-scale checkpoint takes tens of minutes and ranks finish far apart (measured: >10 minutes spread on GLM-4.6, 714 GB over 16 ranks), so the fastest ranks' first collective dies on the default 10-minute NCCL watchdog. Use a 2-hour timeout when transformers initializes the group itself. * Move the timeout comment onto the argument it explains
* align dtype of queries, keys in Grounding DINO To support huggingface#32672 * update dtype of grid_y, grid_x * Create dtype-dependent tensors at the model dtype in Grounding DINO valid_ratios and the reference point grid were built as float32 regardless of the model dtype, so grid_sample and the reference point head received float32 against half weights. fp16 and bf16 both crashed. get_valid_ratio gains the dtype parameter that deformable_detr, lw_detr, mask2former, oneformer and rf_detr already have. Fixes huggingface#32672 Co-authored-by: Sam <samhk1807@gmail.com> --------- Co-authored-by: Sam <samhk1807@gmail.com>
…uggingface#47793) Fix decode() batch path hardcoding clean_up_tokenization_spaces=False instead of None The batch path of decode() was hardcoding clean_up_tokenization_spaces=False when the kwarg was not explicitly provided, overriding self.clean_up_tokenization_spaces. The single-sequence path correctly leaves it unspecified (None), allowing _decode to fall back to the tokenizer's configured value. Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com>
* CanaryConverter draft * init: Add files (canary-1b-v2) * fix: Resolve Bandit warning * fix: Make CI green 1 * refactor: Address review round 1 - 1 * refactor: Address review round 1 - 2 * refactor: Address review round 2 * DIrectly create config file since doesn't need modular. * Testing nits/refactor. * Refactor common processing utillities. * refactor: Address review round 3 * Nits * refactor: Address review round 4 * refactor: Address review round 5 * Modular nits * fix: CI issue * Address comments on docs. * Apply suggestion from @vasqu Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * address config comment * fix: Make changes * address modular comments * fix: Make changes 2 * Move dtype casting. * Address comments * Add back decorators. * replace: NVIDIA Canary-1B-V2 repo --------- Co-authored-by: Eustache Le Bihan <eulebihan@gmail.com> Co-authored-by: Eric B <ebezzam@gmail.com> Co-authored-by: Eric Bezzam <4757445+ebezzam@users.noreply.github.com> Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* fix Co-authored-by: alwaysprince05 <218506877+alwaysprince05@users.noreply.github.com> * add test * remove redundant --------- Co-authored-by: alwaysprince05 <218506877+alwaysprince05@users.noreply.github.com>
* fix qwen4exp-fp8 ple embedding * fix weight_scale resave * add modules_to_convert * unnecessary dtype * lets use super * automatically anchor weight targets
…or_config` (other (2)) (huggingface#48414) * [serge] Fix 2 integration tests for model `cwm` failing with `import_or_config` (other (2)) * Use CwmAttention.layer_type instead of adding attention_type to the layer --------- Co-authored-by: serge[bot] <serge[bot]@users.noreply.github.com> Co-authored-by: Tarek Ziade <tarek@ziade.org>
* Add BatchRebalanceSampler for cost-balanced distributed training
New sampler that balances sequence length cost across DP ranks while
minimizing padding waste. Uses Poorman's Batch Rebalance algorithm with
configurable cost model.
Changes:
- trainer_pt_utils.py: Add BatchRebalanceSampler class
- training_args.py: Add 'batch_rebalance' to train_sampling_strategy
choices, batch_rebalance_alpha and batch_rebalance_max_tokens params
- trainer.py: Add batch_rebalance branch in _get_train_sampler
Usage:
TrainingArguments(train_sampling_strategy='batch_rebalance',
batch_rebalance_alpha=0.001)
* Add unit tests for BatchRebalanceSampler
Tests cover:
- Basic correctness (coverage, micro-batch count)
- No overlap between ranks within same global batch
- Effective batch size invariant
- max_tokens constraint enforcement
- Determinism (same seed = same result)
- Custom cost_fn override
* Fix: remove invalid super().__init__ call in BatchRebalanceSampler
PyTorch's Sampler base class doesn't accept data_source kwarg.
Drop the super().__init__() call entirely (consistent with other
HF samplers like LengthGroupedSampler).
* Remove unused micro_batch_size param, fix silent data loss in max_tokens rebalance
- Remove micro_batch_size from BatchRebalanceSampler: it was stored but never
read anywhere in the class, and its name was misleading since it did not
reflect the actual minimum per-group size used by the algorithm.
- Fix a bug in _poormans_rebalance where counts[gi] -= 1 was applied
unconditionally before searching for a donor group, causing samples to be
silently dropped when max_tokens could not be satisfied and no group had
spare capacity. Now the decrement only happens once a donor is found, and a
warning is logged (with a fallback to exceed max_tokens rather than drop
samples) when no donor is available.
- Fix docstring typo (Poorman's -> Poor Man's).
- Update/add tests: drop micro_batch_size from existing tests, add a
regression test for the max_tokens data-loss fix, and rewrite the cost_fn
test to directly assert on _cost() output instead of an unstable partition
overlap comparison.
* Fix batch_rebalance sampling strategy: missing import and DataLoader wiring
BatchRebalanceSampler yields a full batch of sample indices per iteration
(BatchSampler semantics), but _get_dataloader() was passing it via
DataLoader's sampler= kwarg together with batch_size=, which expects the
sampler to yield one sample index at a time. This caused
'TypeError: list indices must be integers or slices, not list' as soon as
train_sampling_strategy="batch_rebalance" was used with the Trainer.
Also fixes a NameError (BatchEncoding was referenced in _get_train_sampler
but never imported), which was the first failure hit before reaching the
DataLoader wiring bug.
Together these two bugs meant the batch_rebalance strategy could never
actually run end-to-end through Trainer.train(); verified with a CPU
multi-process (torchrun --nproc_per_node=4) benchmark on Qwen2.5-0.5B with a
synthetic variable-length dataset after the fix.
* Remove unused variable flagged by ruff F841
* Fix import sort order for BatchRebalanceSampler (ruff I001)
* Apply ruff format to changed files
* Replace dead count-adjustment code with explicit ValueError
When n < K * min_per_group, there are not enough samples to fill all groups.
Previously this was silently patched by trimming counts (and a dead 'total < n'
branch that could never trigger). Now it raises a clear error telling the user
to reduce effective_batch_size or increase dataset size.
* Fix can_transfer accuracy and iterate all lo_candidates in rebalance
- Replace can_increase(cts, gi) with can_transfer(cts, from_idx, to_idx)
which simulates the actual -1/+1 transfer to compute the correct
start index and max_len for the token limit check. The previous
implementation used stale positions that didn't account for shifts
caused by the donor group shrinking.
- When the lowest-cost group cannot accept a transfer due to max_tokens,
iterate through remaining candidates by ascending cost instead of
falling through to the nail-house branch or breaking entirely.
* Scale BatchRebalanceSampler iteration budget with effective batch size
Each iteration of the rebalance loop moves exactly one sample between groups,
so worst-case convergence is O(n). The fixed `K * 30` cap could therefore
truncate before convergence for large `effective_batch_size` with a small
`dp_size * grad_accum` (K), leaving the partition at a non-optimal cost spread.
Use `max(K * 30, effective_batch_size)` as the budget: the `effective_batch_size`
term covers the move-dominated regime (large batch, small K), while `K * 30`
remains a floor for the detection-dominated regime (tiny batch, n ~= K) where
convergence is reached via `spread == 0` / `seen` rather than the move budget.
`best_counts` is still tracked, so hitting the cap only degrades the result
gracefully (no samples are dropped).
Adds a regression test (large n=246, small K=2, heavily skewed lengths) that
asserts full sample coverage across all ranks and convergence to the
brute-force-optimal contiguous K=2 cost spread.
* Fix BatchRebalanceSampler correctness issues in Trainer integration
Three review fixes for PR huggingface#47340: defeat double-sharding by neutralizing accelerate's BatchSamplerShard wrapper after prepare; fix effective_batch_size undercount (train_batch_size // world_size undercounts since train_batch_size excludes world_size in DDP); honor drop_last=False by yielding the padded trailing partial batch and passing dataloader_drop_last from the Trainer. Adds regression tests for all three.
* Fix UnboundLocalError on `sampler` for iterable datasets in Trainer._get_dataloader
`sampler` was only assigned inside the non-IterableDataset branch, but
the post-`accelerator.prepare` check referenced it unconditionally.
Initialize `sampler` to None so IterableDataset runs no longer raise.
* Address sywangyi review: effective_batch_size DP fix, bounded tail padding, DDP integration test
- trainer.py: effective_batch_size = train_batch_size * grad_accum * world_size
(was per_device_train_batch_size * ...), so the DP (single-process, multi-GPU) case
no longer drops the n_gpu factor. Covers both DDP and DP.
- trainer_pt_utils.py: when drop_last=False and the remainder < dp_size*grad_accum,
pad the tail up to K (dp_size*grad_accum) instead of the full effective_batch_size,
avoiding heavy over-weighting of replayed samples in the final optimizer step.
- test_trainer_data.py: extend effective_batch_size regression test to cover both DDP
and DP; assert tail padding yields eff_bs + K (not eff_bs + eff_bs).
- Add real DDP integration test (batch_rebalance_ddp.py + test_batch_rebalance_no_double_sharding)
that exercises Trainer.get_train_dataloader -> accelerator.prepare -> patch path end-to-end
under accelerate launch, verifying the structure-dependent wrapper detection and exact
global sample coverage (no double-sharding drops).
* Print summary line in batch_rebalance_ddp worker script
execute_subprocess_async (the distributed test harness) treats a subprocess that
produces no stdout/stderr as a failure. When launched with a full --config_file the
script was silent on success, so the test raised 'produced no output'. Emit a one-line
summary on the main process.
* Trim redundant rationale from effective_batch_size comment
* Apply ruff format to test_trainer_distributed_ddp
* Pad tail by wrapping order when dataset is smaller than K
order[:pad] returns at most len(order) elements, so for len(dataset) < K the tail
underfills and _poormans_rebalance(..., min_per_group=1) raises ValueError. Repeat the
order (math.ceil) to yield exactly pad padding indices. Adds an n=1, K=8 regression test.
* Address comments
Signed-off-by: Guokai Ma <guokai.ma@gmail.com>
* Remove batch_rebalance_alpha and batch_rebalance_max_tokens args
Per review: no ablation was run on these knobs and there is no user demand,
so they do not belong in the public API. The quadratic cost weight is fixed
at 0.001 (previous default) as an internal class constant; intercept and
cost_fn were unused and are deleted along with the max_tokens machinery
(feasibility repair loop, can_transfer, _padded_tokens, _within_token_limit).
Output is bit-identical to the previous defaults, verified against the
pre-change implementation over 30 randomized inputs.
Signed-off-by: Guokai Ma <guokai.ma@gmail.com>
* Simplify BatchRebalanceSampler for readability
Per review: document and restructure the sampler internals.
- _assign: explain the snake deal (groups ranked by cost, slot k takes ranks
[k*dp_size, (k+1)*dp_size) so every slot has the same cost profile).
- __iter__: deduplicate the full-batch loop and drop_last=False tail block into
a single loop over global batches plus a _yield_global_batch helper.
- _poormans_rebalance -> _balance_groups: drop the three redundant kwargs
(derivable from self), document the hill-climbing search, and extract the
freeze fallback into _slot_groups (pure: which groups share the capped
group's slot) and _relieve_and_freeze (top up the slot from outside groups,
then freeze it).
Output verified bit-identical to the previous implementation over 286
randomized inputs, including adversarial freeze-prone configurations.
Signed-off-by: Guokai Ma <guokai.ma@gmail.com>
* Consolidate batch_rebalance tests and extend the DDP e2e to two epochs
Per review: keep it to a few unit tests and focus on the DDP integration test.
- Merge basic/no_overlap/eff_batch_size/deterministic into one basic test
asserting coverage, no cross-rank overlap, exact per-global-batch counts
and seed determinism (original hand-picked lengths kept).
- Merge the four drop_last tests into one test covering: tail dropped with
drop_last=True, tail yielded as-is otherwise, padding up to K (not
effective_batch_size) when the remainder is smaller, and order wrap-around
when the whole dataset is smaller than K.
- Drop test_batch_rebalance_defeats_double_sharding: it simulated
accelerate's BatchSamplerShard by hand; the real path is covered by the
DDP e2e test.
- Extend the DDP e2e worker to actually train for two epochs and assert the
per-epoch invariants end to end: each epoch covers all samples across
ranks, set_epoch reaches the wrapped sampler (inner_epoch advances) and
later epochs reshuffle. Validated on 2x RTX 5090.
Every assertion from the removed tests maps into the consolidated ones.
Signed-off-by: Guokai Ma <guokai.ma@gmail.com>
* Reuse train.py for the batch_rebalance DDP test
Drop the bespoke batch_rebalance_ddp.py worker script and its white-box
assertions in favour of the shared train.py, so the sampler is exercised
against a real model end to end. train.py gains a --padding option because
the default max_length padding makes every sample the same length, which
leaves the rebalancing with nothing to do.
* [docs] Add batch_rebalance sampling strategy to the training guide
* [docs] Move batch_rebalance sampling guide to trainer_recipes
Move the batch_rebalance sampling strategy docs from training.md to a
dedicated section in trainer_recipes.md, next to Dataloader performance,
following review feedback. The new section explains the two imbalances
it addresses (padding within a micro-batch and idle time across
devices), how the sampler works (sort then shard), the average
per_device_train_batch_size semantics, and how to skip the startup
length scan with precomputed lengths.
---------
Signed-off-by: Guokai Ma <guokai.ma@gmail.com>
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* Remove sentencepiece warning for tiktiokens * Fix non contiguous * Fix potential None * Added test * Inline a single use fn * Better size estimation
Fix possessive typo in Whisper long-form warning. Co-authored-by: Cursor <cursoragent@cursor.com>
* fix * fix-repo and image-pil-kwargs fix
huggingface#48864) * fix(vibevoice-asr): use integer ceiling division for audio token count padding_mask sums are int64; dividing by hop length in float32 loses precision past 2**24 samples, so torch.ceil undercounts by one vs the processor's float64 np.ceil for long audio. Integer (n + d - 1) // d is exact for all counts. Fixes huggingface#48835 * Update src/transformers/models/vibevoice_asr/modeling_vibevoice_asr.py * Update src/transformers/models/vibevoice_asr/modular_vibevoice_asr.py --------- Co-authored-by: ege-arhan <ege@example.com> Co-authored-by: Eric Bezzam <4757445+ebezzam@users.noreply.github.com>
* Improve lazy import error messages * Address lazy import review feedback * Refine lazy import error diagnostics * Test lazy import debug diagnostics * Fix lazy import diagnostic formatting * Use CaptureLogger in lazy import test * Keep lazy import test logger local * Format lazy import test * Apply Ruff formatting to lazy import test * Fix debug log capture in lazy import test * Fix lazy import log capture test * Format lazy import regression test --------- Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* fix tests * fix ! * updated ! * up * audioo * remove unneeded fix
…ace#48904) - MaxTimeCriteria: the docstring prose and Args entry said 'initial_time'; the parameter is 'initial_timestamp'. - RTDetrLoss / DFineLoss: the Args blocks documented seven constructor parameters that do not exist (they are attributes built internally from config, and the matcher type was also stale); replaced with a single config entry. Follows up on huggingface#48575/huggingface#48576 in the same docstring-vs-signature lane.
…gingface#48651) * DOC: Improve documentation for remap_legacy_layer_types function * fix: remove trailing whitespace from blank lines in docstring ruff's W293 flagged three blank lines inside the remap_legacy_layer_types docstring that carried indentation-width trailing whitespace instead of being empty. Made them genuinely empty. * Apply review suggestions to remap_legacy_layer_types docstring --------- Co-authored-by: Mathew Kadambatt <mathew.kadambatt@walmart.com>
* Align special tokens on the text config On a composite model the special token ids live on the text sub-config, and the top-level config does not forward attribute lookups to it, so align_special_tokens read None, reported a mismatch on every run and wrote the ids onto the top-level config. Reading and writing through get_text_config() fixes it; the method returns the config itself for a text-only model, so nothing changes there. * Drop the comment --------- Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
…gingface#48905) - README_ro/tr/fa linked ./awesome-transformers.md relative to i18n/, where no such file exists; the fr/de/ko/zh-hans/ja translations use the absolute URL, and these three now match. - The xlnet reorder cache docstring cross-referenced PreTrainedModel.beam_search/beam_sample, which were removed; only _beam_search exists.
…huggingface#48941) - Rename AXK1IntegrationTest → AXK2IntegrationTest - Update CUDA (8, 6) expected generation output to match actual model output Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
…commit_hash around (huggingface#47611) * Resolve Hub revision to a commit hash once per load Replace the private `_commit_hash` plumbing by resolving the requested revision with `HfApi.resolve_revision` at each public loading entry point, then passing the returned `ResolvedRevision` around as the regular `revision` argument. Signed-off-by: Wauplin <lucainp@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Test revision resolution against released Hub API * remove proxies + tests * revert reverted tests * revert reverted tests * revert tests * docs * Preserve cached missing-file errors for pinned revisions * Remove added cache regression tests * Report a missing file the same way whether it comes from the cache or the Hub `cached_files` has a last escape before raising for missing entries: a lone `config.json` that does not exist is not an error, it returns `None` and the caller loads without it. The short circuit that serves files from the cache when `revision` pins a commit had its own `raise` instead, without that escape. It used to be reached only for the few calls that threaded `_commit_hash` around, so it rarely mattered; now that every load resolves its revision, it is the path taken by any repository without a `config.json` whose absence is already cached, and those loads started failing with `OSError: Could not locate config.json inside <repo>`. Both paths now end in the same step, so missing entries are reported identically - same escape, same error message - whether the answer came from the Hub or from the cache. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Simplify cached missing config handling * Share cached file finalization in a nested helper --------- Signed-off-by: Wauplin <lucainp@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Cyril Vallez <cyril.vallez@huggingface.co> Co-authored-by: Agent Manager <agents@agent-manager.local>
…#48739) * Keep image processor backends in sync on keys and dtypes The torchvision backends of Idefics2, Idefics3 and SmolVLM allocated the batch pixel_attention_mask buffer without a dtype, so the int64 masks built by pad() were stored as float32 while the PIL backend returned int64. The Fuyu PIL backend computed the original image sizes but never returned them, so its output was missing the image_sizes key the torchvision backend has. test_backends_equivalence only compared pixel_values, and the Idefics2, Idefics3 and SmolVLM overrides compared the masks as pixel_attention_mask.float(), which hid the dtype difference. Compare every returned key instead: same keys, same dtype and shape, exact equality for integral outputs and the existing tolerances for float ones. * Compare non-tensor outputs and drop the mask-only helper _assert_encodings_equivalence now walks lists and tuples and compares everything else exactly, so rows, cols and other list outputs are checked instead of skipped. The model overrides keep only their inputs and use it for every key. * Drop the SmolVLM and Idefics3 backend equivalence overrides With torchvision >= 0.27 the torchvision backend supports LANCZOS on CPU, so the base tests pass for both models and the BICUBIC workaround is not needed anymore. --------- Co-authored-by: Raushan Turganbay <raushan@huggingface.co>
…face#47730) * Fix infeasible cost matrix errors in hugarian matcher losses * add warning and test * replace warning by docs * remove test
* fix(minicpmv4_6): correct get_sliced_grid semantics and tie-break for training parity * feat: add M-RoPE canvas * feat: add minicpmv4_7 with canvas mRoPE; revert minicpmv4_6 * refactor(minicpmv4_7): thin-inherit from minicpmv4_6 * refactor(minicpmv4_7): inherit processor from minicpmv4_6 * Fix canvas M-RoPE: resolve structural token ids from config, reject malformed visual spans, add regression tests * Order target_sizes_mrope by text placeholder order, not modality processing order * Drop stale canvas helper comment that cites a removed function * refactor(minicpmv4_7): drop redundant tp/ep plan rewriting from config __post_init__ * feat(minicpmv4_7): add packed-sequence training support with cu_seqlens * Reuse the shared get_max_seqlen utility in _prepare_packed_attention_kwargs instead of recomputing the packed max length inline * Remove packed-sequence handling from modeling since FA varlen kwargs are expected to come from a user data collator * Drop the unused CONFIG_MAPPING import and reformat so the code-quality gate stops skipping every CI test job * Move all processor classes into the modular file so no cross-model imports remain outside it * Stop numbering video placeholders with local image ids since a video is one temporal sequence and was trained without them * Remove low-value comments in the config class and the dangling video local-id note * Add the MiniCPM-V 4.7 model card and register the model in the doc toc and repo checks * Collapse per-axis canvas position writes into leading-axis slices and drop the input sanity checks the processor already guarantees * Merge vision windows per image so slicing batches with mixed patch grids no longer crash The vectorised merge lifted one (h, w) from target_sizes[0] and reshaped the whole batch against it. MiniCPM-V slicing emits a global thumbnail grid next to the slice grids, and the two differ whenever the image is not square-ish, e.g. 896x448 at max_slice_nums=9 gives [[24, 44], [32, 32], [32, 32]] and the view() raises. Merge each image against its own grid instead, matching the pre-huggingface#46738 behaviour. * Cover the vision window merger with mixed per-image patch grids Pins the token count a slicing batch produces when the thumbnail grid disagrees with the slice grids, and checks that an image merges the same way regardless of what else is packed next to it. * Drop the standalone mrope test module * fix(minicpmv4_7): regenerate modeling from modular * Cover canvas M-RoPE position ids with get_rope_index unit tests * review comments - rope tests are green * Add auto_docstring to the 4.7 model classes and drop the unreachable token-id branch gates (#3) * feat: reimpl mrope (huggingface#4) * Pin the canvas M-RoPE layouts with golden position ids Snapshot of the exact (3, batch, seq) coordinates for every layout the processor can emit, taken before refactoring the canvas internals. Verified identical to the pre-review implementation, so the numbers describe intended behaviour rather than the current code. * Return the cached-delta positions while decoding and drop a debug print * Derive the LLM grid from config.downsample_mode instead of counting tokens * Build the canvas from `mm_token_type_ids` and fold the video path into the image one * Let the LLM grid follow the per-call downsample_mode * Group adjacent frames whatever their modality * Drop the unused special_token_ids argument and fix the return-shape docstring * Drop the unreachable grid fallbacks and inline the one-line grid helper * test with bugfix (huggingface#5) * Keep the per-sample mrope grids out of tensor stacking and run the merger test on the accelerator * Pad the per-sample mrope grids into one tensor and refresh the video slice-mode token count * Regenerate the processing file so the new property sits where the converter puts it * Point the integration tests at MiniCPM-V-4.7 and refresh the expected generations * first clean-up on processors * modualr, will upstream in a sec on main * fix mrope tests and use shared fn for meshgrid * run fix repo * the missing test * be gone, now we have a test to check it matches * same here, remove - let config set attr to False * trf * delete tests as well * Anton won't like it here... * unnecesary revert * Update docs/source/en/model_doc/minicpmv4_7.md Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * Update src/transformers/models/minicpmv4_7/modular_minicpmv4_7.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * Update src/transformers/models/minicpmv4_7/modular_minicpmv4_7.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * address comments (huggingface#7) * Guard window merger divisibility, rename patch_residual, and fix processor kwargs unpacking * Address review comments * address review comments: validate mm_token_type_ids, forward **kwargs to loss_function, align naming with Qwen VL, and regenerate modular files with pinned ruff (huggingface#8) * move _frame_end_idx and _group_visual_frames into MiniCPMV4_7Model (huggingface#9) * address review (huggingface#10) * unskip test_generate_from_inputs_embeds and test_left_padding_compatibility, which pass as-is * export MiniCPMV4_7VisionModel and register it in the auto model mapping, following Qwen3_5 * return metadata tuples instead of dicts from _group_visual_frames * last commit * CI checks * again, repo check * export * comment --------- Co-authored-by: airlsyn <airlsyn.ai@gmail.com> Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
… path (huggingface#48897) * Optimize torchvision image normalization Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com> * Test Pixtral normalization input preservation * Fix LFM2-VL grayscale conversion test * Update src/transformers/image_processing_backends.py Co-authored-by: Matt <Rocketknight1@users.noreply.github.com> --------- Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com> Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes # (issue)
Code Agent Policy
The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. These often are low-quality, or fix extremely minor issues that occur rarely or never in practice.
As a result, we're instituting a rule that first-time contributors should not use code agents to submit PRs or issues.
We'd also ask autonomous "OpenClaw"-like agents not to open any PRs or issues.
Issues/PRs from first-time contributors that violate this rule will probably just be closed without review, and we
might block you, especially if you open more than one or appear to be deliberately ignoring this. We especially do not
want new contributors to jump in on random issues to contribute an agent-written fix. This creates lots of noise
for reviewers and other users and will almost certainly get you blocked.
For more information, please read
CONTRIBUTING.md.Before submitting
Pull Request checks?
to it if that's the case.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.