11/14/2019

Google Cloud Run - how to fix "ERROR: (gcloud.beta.run.deploy) Resource readiness deadline exceeded."

As a full-stack developer you might want to deploy your containers on Google Cloud Run instead of managing a full-blown Kubernetes cluster with knative on top of it. However please note that Google Cloud Run is currently still in beta so your experience may vary.

Yesterday I saw increased build failure related to a soon-to-be-merge branch from a co-worker, the CI gcloud run deploy command yields:

ERROR: (gcloud.beta.run.deploy) Resource readiness deadline exceeded.

Unlike in a traditional Kubernetes cluster, Readiness probes are not currently configurable in Cloud Run full managed mode and looking inside Cloud Run logs everything seemed to be fine, the app was listening to 8080 after some time.

After a deep dive into the code changes I found that some new code was delaying for up to 12 seconds the app socket listening step. Worst, those 12 seconds delay was a measure I took on my MacBook and this new code was highly Disk I/O bound so it explained why Cloud Run readiness probe timed out.

The fix was then to remove this delay and load what was needed from the filesystem on-demand (instead of loading everything — with a lot of never-used stuff — somewhere in the app life-cycle). I hope this post will help someone because I've found nothing on the web on that matter :) !

« »
 
 
Made with on a hot august night from an airplane the 19th of March 2017.