Phase 1a.1: Merge 3-level hierarchy into dataset-planning baseline - #16
Merged
research-developer merged 8 commits intoOct 24, 2025
Merged
research-developer merged 8 commits into
research-developer merged 8 commits into
Conversation
Provides unified testing interface across all three domain worktrees: **Commands**: - `make test-all`: Run tests across Causal, KG, Planning domains - `make test-[domain]`: Run individual domain tests - `make clean-all`: Clean generated files in all branches - `make push-all`: Push all branches to remote - `make status-all`: Show git status for all branches - `make setup-env`: Verify conda environment and worktrees **Worktree Paths** (configured as variables): - CAUSAL_DIR := ../nsm-causal - KG_DIR := ../nsm-kg - PLANNING_DIR := ../nsm-planning **Integration**: - Works with parallel exploration branches (dataset-*) - Standardized pytest configuration (-v --tb=short) - Supports NSM-27, NSM-28, NSM-29 (branch-specific testing) Enables efficient cross-domain comparison for NSM-10 dataset exploration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Changed __getitem__() to return 1D tensors with shape (1,) instead of scalars with shape torch.Size([]): - torch.tensor(1 if ... else 0) → torch.tensor([1 if ... else 0]) This fixes failing test: tests/data/test_planning_dataset.py::TestPlanningTripleDataset::test_dataset_indexing All datasets now return consistent label shapes across domains (Causal, KG, Planning). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Identified root cause of training collapse across all domains: **Problem Analysis**: - Planning: 43.5% accuracy (class collapse - always predicts class 1) - Causal: 52.9% accuracy (barely above random) - KG: 46.0% accuracy (below random) - Cycle loss: 0.78-0.98 (target <0.2) **Root Causes**: ✅ Dataset balance: All datasets properly balanced (50/50 or close) ✅ PyG extensions: SAGPooling works despite warnings (pure PyTorch fallback) ❌ Cycle loss dominance: Weight 0.1 × loss 0.98 = 0.098 competing with task gradient ❌ No class weighting: Binary classification without anti-collapse mechanism ❌ Learning rate too high: 1e-3 causing unstable training **Implementation**: - Add `class_weights` parameter to NSMTrainer.__init__() - Pass weights to F.cross_entropy() in compute_task_loss() - Supports both classification and link_prediction tasks **Next Steps** (NSM-31): Phase 1: Reduce cycle_loss_weight (0.1 → 0.01), LR (1e-3 → 5e-4), add class weights Phase 2: Progressive cycle loss warmup, cosine LR scheduler Phase 3: Adaptive cycle weight tuning See NSM-31-TRAINING-FIXES.md for complete implementation plan. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Implements comprehensive validation to catch NSM-31 issues early:
**Automated Checks**:
1. Dataset balance (prevent class collapse)
2. Cycle loss weight (≤0.05, prevent gradient dominance)
3. Learning rate (≤5e-4, prevent instability)
4. PyG extensions (verify SAGPooling works)
5. Model architecture (validate required components)
6. Class weights (recommend for imbalanced datasets)
**Usage**:
```python
from nsm.evaluation import run_preflight_checks
results = run_preflight_checks(
dataset=train_dataset,
model=model,
cycle_loss_weight=0.01,
learning_rate=5e-4,
strict=True
)
```
**Features**:
- Clear error messages citing NSM-31 analysis
- Warnings for suboptimal (but not critical) settings
- Self-test mode for validation
- Integrated into nsm.evaluation module
**Files**:
- nsm/evaluation/preflight_checks.py: Core validation logic (450+ lines)
- nsm/evaluation/__init__.py: Module exports
- NSM-31-TRAINING-FIXES.md: Updated with preflight documentation
Prevents repeat of NSM-31 failures:
- Planning: 43.5% accuracy (class collapse)
- Causal: 52.9% accuracy (barely above random)
- KG: 46.0% accuracy (below random)
- All: Cycle loss 0.78-0.98 (target <0.2)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
13 unit tests validating NSM-31 issue detection:
**Test Coverage** (11/13 passing initially):
- Dataset balance checks (3 tests)
- Cycle loss weight validation (3 tests)
- Learning rate validation (3 tests)
- PyG extension verification (1 test)
- Integration tests (3 tests)
**Validates Detection Of**:
- Class imbalance (prevent collapse)
- High cycle loss weight (>0.05)
- High learning rate (>5e-4)
- Broken PyG pooling operations
**Test Examples**:
```python
# Good parameters pass
run_preflight_checks(
dataset=balanced_dataset,
cycle_loss_weight=0.01,
learning_rate=5e-4
) # ✅ Passes
# Bad parameters warn/fail
run_preflight_checks(
cycle_loss_weight=0.1, # ❌ Too high
learning_rate=1e-3 # ❌ Too high
) # Warns or raises error
```
Fixed warning tracking to properly capture PreflightCheckWarnings
during validation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The dataset constructor uses num_problems, not num_plans. This was causing 0 plans to be generated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Add num_levels=3 to NSMModel to test alternating bias hypothesis: - L1 (concrete): Primitive actions, state observations - L2 (mid): Subgoals, method decompositions - L3 (abstract): High-level goals, strategic objectives Expected: Breaking 2-level WHY>WHAT>WHY>WHAT symmetry reduces class collapse by providing richer gradient pathways. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
|
You have run out of free Bugbot PR reviews for this billing cycle. This will reset on November 24. To receive reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial. |
This was referenced Oct 24, 2025
research-developer
added a commit
that referenced
this pull request
Oct 24, 2025
…_processes parameter) Dataset-planning (from PR #16) removed the check_processes parameter from run_preflight_checks(). Accepting this change for consistency. 🤖 Generated with [Claude Code](https://claude.com/claude-code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merges the 3-level hierarchy implementation from
dataset-planning-3levelinto thedataset-planningbaseline branch. This is Phase 1a.1 of the comprehensive branch merge strategy.Changes
experiments/train_planning.pyupdated for 3-level compatibilityValidation
pytest tests/data/test_planning_dataset.py)nsm/data/planning_dataset.pyTesting
pytest tests/data/test_planning_dataset.py -v # 25 passed, 3 warnings in 5.19sReferences
.claude/specs/merge-main-and-3level-branches/01-product-requirements.md.claude/specs/merge-main-and-3level-branches/02-system-architecture.mdpre-merge-dataset-planning-20251024Next Steps
After approval:
dataset-planningdataset-causal-3level→dataset-causalmain🤖 Generated with Claude Code
Co-Authored-By: Claude noreply@anthropic.com