Repository navigation
[ISSUE] status endpoint crash #1032
Description
If I make a larger query like this:
graph indexer allocations get --deployment QmYNPk3Mabrx4VmxPKmBevdi11cfVUfTXVtgygG4M4m9tQwhere the SG allocation is very old or the query itself is very abstract and the API needs to perform a larger query, I notice that the status API endpoint is crashing with no logs related.
This command above produces this error:
✖ [GraphQL] Failed to query Epoch Block Oracle Subgraphthen I query the status network
graph indexer status network --network arbitrum-oneand I get this output with an error indicating that (possibly) the JVM crashed (again no logs and no pods restarted)
graph@indexer-cli:~$ graph indexer status network --network arbitrum-one
Registration
┌───────────────────────────────────────┬────────────────────────────────────────────┬─────────────────┬────────────┬─────────────────────────────────────┐
│ url │ address │ protocolNetwork │ registered │ location │
├───────────────────────────────────────┼────────────────────────────────────────────┼─────────────────┼────────────┼─────────────────────────────────────┤
│ https://indexer.upgrade.thegraph.com/ │ 0xbdfb5Ee5a2aBF4fc7bB1bD1221067aEf7f9dE491 │ eip155:42161 │ true │ 89.99997854232788,36.57848596572876 │
└───────────────────────────────────────┴────────────────────────────────────────────┴─────────────────┴────────────┴─────────────────────────────────────┘
Endpoints
┌─────────┬─────────────────┬─────────────────────────────────────────────┬────────┐
│ name │ protocolNetwork │ url │ status │
├─────────┼─────────────────┼─────────────────────────────────────────────┼────────┤
│ service │ arbitrum-one │ https://indexer.upgrade.thegraph.com/ │ down │
├─────────┼─────────────────┼─────────────────────────────────────────────┼────────┤
│ status │ arbitrum-one │ https://indexer.upgrade.thegraph.com/status │ up │
└─────────┴─────────────────┴─────────────────────────────────────────────┴────────┘
The following endpoint tests failed:
https://indexer.upgrade.thegraph.com/
Test: http get https://indexer.upgrade.thegraph.com/
Error: fetch failed
Possible actions:
- Make sure https://indexer.upgrade.thegraph.com/ can be resolved and reached from this machine
- Make sure the port of https://indexer.upgrade.thegraph.com/ is set up correctly
- Make sure the test command returns an HTTP status code < 400
Indexer Deployments
<--- Last few GCs --->
[26273:0x64f1df0] 83686 ms: Mark-Compact (reduce) 4090.9 (4116.2) -> 4090.9 (4109.2) MB, 35.44 / 0.00 ms (+ 8.5 ms in 1 steps since start of marking, biggest step 8.5 ms, walltime since start of marking 48 ms) (average mu = 0.501, current mu = 0.311) [26273:0x64f1df0] 83747 ms: Mark-Compact (reduce) 4093.2 (4111.5) -> 4093.2 (4111.5) MB, 55.82 / 0.00 ms (average mu = 0.340, current mu = 0.078) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----
1: 0xca5580 node::Abort() [node]
2: 0xb781f9 [node]
3: 0xeca4d0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
4: 0xeca7b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
5: 0x10dc505 [node]
6: 0x10f4388 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
7: 0x10ca4a1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
8: 0x10cb635 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
9: 0x10a8c86 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x1503a16 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x7daacfed9ef6
Aborted
Activity
[-][ISSUE] status endpoing crash[/-][+][ISSUE] status endpoint crash[/+]