Error: No connection established

Problem description

Since google-gax forced the grpc-js version to 0.6.4 (googleapis/gax-nodejs@628db9e), I am having the following issue :

Error: No connection established
    at Http2CallStream.call.on (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/call.js:68:41)
    at Http2CallStream.emit (events.js:203:15)
    at Http2CallStream.EventEmitter.emit (domain.js:448:20)
    at process.nextTick (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22)
    at process._tickCallback (internal/process/next_tick.js:61:11)

Reproduction steps

The bug only happens after one hour. We are using grpc-js inside @google-cloud/bigtable and our bigtable instance is autoscaling.

Thank you for the report! Just to confirm, does that mean that the script should have an hour of idle time with no network requests to experience that, or just an hour of runtime?

Similar issue on @google-cloud/pubsub@1.1.0
After about 10 minutes of idle time with no network requests, any call to "publish" callbacks after 10 minutes (maxRetryDelayMillis) to say "Retry total timeout exceeded before any response was received"
To be able to find this error "No connection established", I had to add logs in the google-gax module

@harscoet

We are seeing the exact same issues as @harscoet with @google-cloud/pubsub@1.1.0

It happens after an hour of runtime (I had this issue on our main API). Then it starts to throw those errors on separate processes progressively.

Same issue when using @google-cloud/datastore which depends on google-gax

We are looking into this and will either fix it ASAP, or release an update to google-gax that pins @grpc/grpc-js to an older version.

Just to make sure I understand, are you making some requests that succeed, then going idle for a while, then making more requests and seeing these errors?

Also, do you folks run your code on a physical machine, or in any kind of cloud e.g. cloud functions?

@murgatroid99

@murgatroid99 we are seeing that our services are running fine for 10ish minutes and then we start dropping connections. At this time we see deadline exceeded errors. Once this happens we are unable to have successful requests with both google-cloud/pubsub and google-cloud/datastore

@alexander-fenster we are seeing this using with services running within GKE containers

We downgraded @grpc/grpc-js to 0.5.4 and also other related google libraries to a version that don't depend on @grpc/grpc-js@0.6.x. The problem seems gone. It may related to #1062 & #1061

If my fix is right, this problem was actually in #1015 but it was overshadowed by other problems.

I have published version 0.6.5 of @grpc/grpc-js. You should pick it up if you update your dependencies. Please try that and see if it solves this problem.

@apatora

@apatora @harscoet @reichjustin @weipublic Could you folks please check if the fix in v0.6.5 works for you all, and let us know? Thank you!

I'm experiencing sustained CPU spikes to 100% after an hour since a pod is started. I've created a debug pod which doesn't handle any traffic and it still manifests the same way. After profiling the CPU I've found that calls to startCallStream consistently fail which causes the whole tryPick function to be called again basically blocking the node process:

Is it intended for this code: https://github.com/grpc/grpc-node/pull/1251/files#diff-a96b991cf71efa63f54e0313d45cf39cR234-R257 to create a loop like the one I described?

I'm currently using these versions of google libraries:

    "@google-cloud/connect-datastore": "^4.0.0",
    "@google-cloud/datastore": "^5.0.6",
    "@google-cloud/firestore": "^3.5.1",
    "@google-cloud/pubsub": "^1.5.0",

The libraries are using "@grpc/grpc-js@0.6.16", "@grpc/grpc-js@^0.6.12"
I will update this post with logs from GRPC_TRACK in an hour or so.

@alexandrugheorghe

@alexandrugheorghe That looks like the same bug as reported in googleapis/nodejs-pubsub#890, and that user says that the problem was fixed in @grpc/grpc-js version 0.6.18.

@murgatroid99

@murgatroid99 I'll update now and check it again in an hour.

@murgatroid99

@murgatroid99 Sorry for the late reply. Pods have been stable, version 0.6.18 has fixed my issues.
Thanks again for the quick reply and the fix.

I'm hoping this issue hasn't resurfaced, but I did encounter the same error in a project that had been idle for some time. I recycled the pod so I should have an idea of the timeout being the cause in the future, but it impacted all our instances in a test environment.

The error:

err:
    message: '14 UNAVAILABLE: No connection established'
    name: Error
    stack: >
        Error: 14 UNAVAILABLE: No connection established
            at Object.callErrorFromStatus (/app/packages/animus/node_modules/@grpc/grpc-js/src/call.ts:81:24)
            at Object.onReceiveStatus (/app/packages/animus/node_modules/@grpc/grpc-js/src/client.ts:334:36)
            at Object.onReceiveStatus (/app/packages/animus/node_modules/@grpc/grpc-js/src/client-interceptors.ts:434:34)
            at Object.onReceiveStatus (/app/packages/animus/node_modules/@grpc/grpc-js/src/client-interceptors.ts:397:48)
            at Http2CallStream.outputStatus (/app/packages/animus/node_modules/@grpc/grpc-js/src/call-stream.ts:230:22)
            at Http2CallStream.maybeOutputStatus (/app/packages/animus/node_modules/@grpc/grpc-js/src/call-stream.ts:280:14)
            at Http2CallStream.endCall (/app/packages/animus/node_modules/@grpc/grpc-js/src/call-stream.ts:263:12)
            at Http2CallStream.cancelWithStatus (/app/packages/animus/node_modules/@grpc/grpc-js/src/call-stream.ts:597:10)
            at ChannelImplementation.tryPick (/app/packages/animus/node_modules/@grpc/grpc-js/src/channel.ts:387:22)
            at ChannelImplementation._startCallStream (/app/packages/animus/node_modules/@grpc/grpc-js/src/channel.ts:433:10)
            at Http2CallStream.start (/app/packages/animus/node_modules/@grpc/grpc-js/src/call-stream.ts:573:18)
            at BaseUnaryInterceptingCall.start (/app/packages/animus/node_modules/@grpc/grpc-js/src/client-interceptors.ts:374:15)
            at BaseUnaryInterceptingCall.start (/app/packages/animus/node_modules/@grpc/grpc-js/src/client-interceptors.ts:437:11)
            at ServiceClientImpl.makeUnaryRequest (/app/packages/animus/node_modules/@grpc/grpc-js/src/client.ts:315:10)
            at ServiceClientImpl.<anonymous> (/app/packages/animus/node_modules/@grpc/grpc-js/src/make-client.ts:174:15)
            at args (/app/packages/animus/node_modules/dialogflow/src/v2beta1/sessions_client.js:175:35)
            at /app/packages/animus/node_modules/google-gax/src/normalCalls/timeout.ts:54:13
            at OngoingCallPromise.call (/app/packages/animus/node_modules/google-gax/src/call.ts:82:23)
            at NormalApiCaller.call (/app/packages/animus/node_modules/google-gax/src/normalCalls/normalApiCaller.ts:46:15)
            at funcPromise.then.then (/app/packages/animus/node_modules/google-gax/src/createApiCall.ts:103:26)
            at Object.dynatraceOnServiceExecutionIndicator [as doInvoke] (/opt/dynatrace/oneagent/agent/bin/1.195.54.20200529-113801/any/nodejs/nodejsagent.js:1803:20)
            at Object.a.safeInvoke (/opt/dynatrace/oneagent/agent/bin/1.195.54.20200529-113801/any/nodejs/nodejsagent.js:1854:29)
            at /opt/dynatrace/oneagent/agent/bin/1.195.54.20200529-113801/any/nodejs/nodejsagent.js:7079:25
            at process._tickCallback (internal/process/next_tick.js:68:7)"
    code: 14

We install dependencies from the package-lock.json, which has @grpc/grpc-js locked in at 1.0.4.

{
  // deps...
  "@grpc/grpc-js": {
    "version": "1.0.4",
    "resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-1.0.4.tgz",
    "integrity": "sha512-Qawt6HUrEmljQMPWnLnIXpcjelmtIAydi3M9awiG02WWJ1CmIvFEx4IOC1EsWUWUlabOGksRbpfvoIeZKFTNXw==",
    "requires": {
      "google-auth-library": "^6.0.0",
      "semver": "^6.2.0"
    },
    "dependencies": { /** deps... */ }
  }
}

Will have another issue opened if this has resurfaced. I'll be back with more if it's an issue we come across again.

Hello everyone, I've been trying to create a secret in the GCP from running the GCP APIs from AWS lambda. I tried setting the environment variable at the run time like process.env.GOOGLE_APPLICATION_CREDENTIALS = '/tmp/serviceAccountCreds.json'; . The GCP API for secret manager was running fine till DEC 29th on my environment. But from then on the API is throwing below error

Error: 14 UNAVAILABLE: No connection established
    at Object.callErrorFromStatus (/home/akhil/onboard/node_modules/@grpc/grpc-js/src/call.ts:81:24)
    at Object.onReceiveStatus /home/akhil/onboard/node_modules/@grpc/grpc-js/src/client.ts:334:36)
    at Object.onReceiveStatus /home/akhil/onboard/node_modules/@grpc/grpc-js/src/client-interceptors.ts:434:34)
    at Object.onReceiveStatus (/home/akhil/onboard/node_modules/@grpc/grpc-js/src/client-interceptors.ts:397:48)
    at /home/akhil/onboard/node_modules/@grpc/grpc-js/src/call-stream.ts:237:24
    at processTicksAndRejections (internal/process/task_queues.js:79:11) {
  code: 14,
  details: 'No connection established',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }

@grpc/grpc-js version installed is 1.1.8

{
"_from": "@grpc/grpc-js@~1.1.1",
"_id": "@grpc/grpc-js@1.1.8",
"_inBundle": false,
"_integrity": "sha1-KEXw/D0b+7FQ7Xp4p2vfQbEm02c=",
"_location": "/@grpc/grpc-js",
"_phantomChildren": {
  "@types/long": "4.0.1",
  "lodash.camelcase": "4.3.0",
  "long": "4.0.0",
  "protobufjs": "6.10.2",
  "yargs": "15.4.1"
},

The secrets manager's version is 3.2.3

 "dependencies": {
    "@google-cloud/secret-manager": "^3.2.3",
    "aws-sdk": "^2.445.0",
    "aws4": "^1.8.0",

I am facing the same issue again:
package.json:

"@google-cloud/pubsub": "^1.1.6",
    "@grpc/grpc-js": "^1.1.7",

package-lock.json

"@google-cloud/pubsub": {
      "version": "1.7.3",
      "resolved": "https://registry.npmjs.org/@google-cloud/pubsub/-/pubsub-1.7.3.tgz",
...
"@grpc/grpc-js": {
          "version": "1.0.5",
          "resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-1.0.5.tgz",
          "integrity": "sha512-Hm+xOiqAhcpT9RYM8lc15dbQD7aQurM7ZU8ulmulepiPlN7iwBXXwP3vSBUimoFoApRqz7pSIisXU8pZaCB4og==",
          "requires": {
            "semver": "^6.2.0"
          }
        }

@sidcool1234 i'm too.
I'm use dialogflow, it doesn't usually happen
"@google-cloud/storage": "^2.5.0",
"@google-cloud/vision": "^1.8.0",
"actions-on-google": "^1.8.0",
"body-parser": "^1.18.3",
"config": "^3.0.1",
"dialogflow": "^1.2.0",
"dialogflow-fulfillment": "^0.6.1",

@ducvu91

@sidcool1234 , @ducvu91 did you guys find any work-around or still persisting with this issue ? Please let me know if you guys found any work-around.

@AkhilA4

@AkhilA4 not yet, Sometimes it happens and it's random, there's no specific case so I can't find where the cause is.

@ducvu91

OK @ducvu91. Since last week, I'm encountering this issue every time I run the createSecret GCP API. Will try to create a new issue to get some attention for the issue. Not sure what needs to be done from dev end if this gets encountered in production environment.

I was getting this issue because I setup the Firestore emulator and wasn't turning it on. If you recently enabled the emulators, you have to start them with firebase emulators:start . Then you'll start getting the information from your local database.

Seeing the same issue with "@google-cloud/scheduler": "^1.6.0":

Error: 14 UNAVAILABLE: No connection established
    at Object.callErrorFromStatus (/app/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Http2CallStream.<anonymous> (/app/node_modules/@grpc/grpc-js/build/src/client.js:96:33)
    at Http2CallStream.emit (events.js:327:22)
    at Http2CallStream.EventEmitter.emit (domain.js:467:12)
    at /app/node_modules/@grpc/grpc-js/build/src/call-stream.js:97:22
    at processTicksAndRejections (internal/process/task_queues.js:75:11)
    at runNextTicks (internal/process/task_queues.js:62:3)
    at listOnTimeout (internal/timers.js:523:9)
    at processTimers (internal/timers.js:497:7)

UPDATE:

Not a frequent issue, it randomly happened today.
It was running on Node.js 12.
Resolved @grpc/grpc-js version was 0.6.15

After reading above I guess getting @grpc/grpc-js >= 0.6.18 could solve the issue 🤔

Hey
Trying to run simple pub/sub code in my code
Every time brings the following error:

Error: 1 CANCELED: Call cancelled
at callErrorFromStatus (C:\storage-actions\node_modules@grpc\grpc-js\build\src\call.js:31:19)
at Object.onReceiveStatus (C:\storage-actions\node_modules@grpc\grpc-js\build\src\client.js:192:76)
at Object.onReceiveStatus (C:\storage-actions\node_modules@grpc\grpc-js\build\src\client-interceptors.js:360:141)
at Object.onReceiveStatus (C:\storage-actions\node_modules@grpc\grpc-js\build\src\client-interceptors.js:323:181)
at C:\storage-actions\node_modules@grpc\grpc-js\build\src\resolving-call.js:94:78
at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
{
code: 1,
details: 'Call cancelled',
metadata: Metadata { internalRepr: Map(0) {}, options: {} },
note: 'Exception occurred in retry method that was not classified as transient'
}

Is the problem related to a certain version and/or incompatible versions?

Navigation Menu

Repository navigation

Error: No connection established #1064

Problem description

Reproduction steps

67 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Error: No connection established #1064

Description

Problem description

Reproduction steps

Activity

alexander-fenster commented on Oct 4, 2019

harscoet commented on Oct 4, 2019

reichjustin commented on Oct 4, 2019

apatora commented on Oct 4, 2019

weipublic commented on Oct 4, 2019

alexander-fenster commented on Oct 4, 2019

murgatroid99 commented on Oct 4, 2019

alexander-fenster commented on Oct 4, 2019

reichjustin commented on Oct 4, 2019

weipublic commented on Oct 4, 2019

murgatroid99 commented on Oct 4, 2019

murgatroid99 commented on Oct 4, 2019

alexander-fenster commented on Oct 5, 2019

67 remaining items

alexandrugheorghe commented on Feb 25, 2020

murgatroid99 commented on Feb 25, 2020

alexandrugheorghe commented on Feb 25, 2020

alexandrugheorghe commented on Feb 26, 2020

tcarrio commented on Jun 5, 2020

AkhilA4 commented on Jan 3, 2021

sidd-kulk commented on Jan 4, 2021

ducvu91 commented on Jan 4, 2021

AkhilA4 commented on Jan 7, 2021

ducvu91 commented on Jan 7, 2021

AkhilA4 commented on Jan 7, 2021

Ignacio1996 commented on Mar 18, 2021

maikelrobier commented on Mar 30, 2021

brachaIzakov commented on Jun 12, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions