Repository navigation
ChatGPT Importer Fails with RecursionError on Deep Conversation Threads #234
Description
ChatGPT Importer Fails with RecursionError on Deep Conversation Threads
Description
The ChatGPT importer fails with a RecursionError: maximum recursion depth exceeded when importing conversations with deep message threads. This occurs because some ChatGPT conversations can have very deep nesting (1000+ levels), exceeding Python's default recursion limit.
Steps to Reproduce
- Export ChatGPT conversations that contain long conversation threads
- Run
basic-memory import chatgpt - Import fails with RecursionError
Error Output
PS C:\Users\[Username]\OneDrive\Desktop> basic-memory import chatgpt
Importing chats from conversations.json...writing to C:\Users\[Username]\basic-memory\conversations
Failed to import ChatGPT conversations
Traceback (most recent call last):
File "C:\Users\[Username]\AppData\Roaming\uv\tools\basic-memory\Lib\site-packages\basic_memory\importers\chatgpt_importer.py", line 217, in _traverse_messages
child_msgs = self._traverse_messages(mapping, child_id, seen)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Previous line repeated 974 more times]
RecursionError: maximum recursion depth exceeded
Analysis
After analyzing my conversations.json file (1244 conversations), I found:
- Maximum conversation depth: 2,565 levels
- Average depth: 83.1 levels
- 10+ conversations exceed depth of 900
The issue is in the _traverse_messages method which uses recursion to traverse the message tree.
Proposed Solutions
Option 1: Increase Recursion Limit (Quick Fix)
Add this to the importer:
import sys
sys.setrecursionlimit(5000) # or higherOption 2: Iterative Approach (Better Solution)
Replace the recursive _traverse_messages method with an iterative version:
def _traverse_messages(self, mapping: Dict[str, Any], root_id: Optional[str], seen: Set[str]) -> List[Dict[str, Any]]:
"""Traverse message tree iteratively to handle deep conversations."""
messages = []
stack = [root_id] if root_id else []
while stack:
node_id = stack.pop()
if not node_id or node_id in seen:
continue
node = mapping.get(node_id)
if not node:
continue
if node.get("message"):
seen.add(node_id)
messages.append(node["message"])
# Add children in reverse order to maintain conversation flow
children = node.get("children", [])
stack.extend(reversed(children))
return messagesWorkaround
Users can work around this by running:
python -c "import sys; sys.setrecursionlimit(10000); from basic_memory.cli.main import app; sys.argv=['basic-memory', 'import', 'chatgpt']; app()"Additional Considerations
The Claude conversations importer might have the same issue if Claude exports have similarly deep conversation threads. Both importers should be checked and updated to handle deep nesting gracefully.
Environment
- OS: Windows 11
- Python version: (from UV tools environment)
- basic-memory version: (latest)
Impact
This bug prevents users from importing their ChatGPT conversation history if they have long conversation threads, which is common for power users who use ChatGPT for extended problem-solving sessions.
Activity
phernandez commentedon Jul 18, 2025
@claude can you look at this issue and make PR?
claude commentedon Jul 18, 2025
Claude finished @phernandez's task —— View job •
claude/issue-234-20250718-2109• Create PR ➔Todo List:
_traverse_messagesmethodAnalysis: Found the issue in
/src/basic_memory/importers/chatgpt_importer.pyline 217. The_traverse_messagesmethod uses recursion to process child messages, which fails on deep conversation threads (2,565+ levels). Converted to iterative approach using a stack.Implementation: Replaced recursive
_traverse_messageswith iterative version that uses a stack to process nodes. This eliminates the recursion depth limitation while maintaining the same message processing order.Fix Details:
fix: replace recursive _traverse_messages with iterative approach to …
phernandez commentedon Jul 18, 2025
@claude can you check the claude importer for the same issue and fix it also?
claude commentedon Jul 18, 2025
Claude finished @phernandez's task —— View job
Analysis Complete
Todo List:
Results
Good news: The Claude importers are already safe from the recursion issue!
After examining both Claude importers:
claude_conversations_importer.py- Uses simple flat iteration over messages:claude_projects_importer.py- Uses simple loops over projects and documents:Why the difference?
The ChatGPT importer needed the
_traverse_messagesmethod with recursion to handle the tree structure, but Claude's simpler data format already avoids this issue entirely.No code changes needed - Claude importers are already robust against deep conversation threads.
•
claude/issue-234-20250718-2116