Google Cloud Run - how to fix "ERROR: (gcloud.beta.run.deploy) Resource readiness deadline exceeded."

As a full-stack developer you might want to deploy your containers on Google Cloud Run instead of managing a full-blown Kubernetes cluster with knative on top of it. However please note that Google Cloud Run is currently still in beta so your experience may vary.

Yesterday I saw increased build failure related to a soon-to-be-merge branch from a co-worker, the CI gcloud run deploy command yields:

ERROR: (gcloud.beta.run.deploy) Resource readiness deadline exceeded.

Unlike in a traditional Kubernetes cluster, Readiness probes are not currently configurable in Cloud Run full managed mode and looking inside Cloud Run logs everything seemed to be fine, the app was listening to 8080 after some time.

After a deep dive into the code changes I found that some new code was delaying for up to 12 seconds the app socket listening step. Worst, those 12 seconds delay was a measure I took on my MacBook and this new code was highly Disk I/O bound so it explained why Cloud Run readiness probe timed out.

The fix was then to remove this delay and load what was needed from the filesystem on-demand (instead of loading everything — with a lot of never-used stuff — somewhere in the app life-cycle). I hope this post will help someone because I've found nothing on the web on that matter :) !


Fix the SchemaError(io.k8s.api.apps.v1beta2.StatefulSetStatus): invalid object doesn't have additional properties error

Issue: Kubernetes API server version and kubectl CLI version mismatch.

  • Kubernetes API server version (e.g. v1.14.7) 
  • kubectl CLI version (e.g. {Major:"1", Minor:"10"})

Solution: upgrade kubectl 👍


Rollback Keycloak admin theme change without downtime

Let start this post by saying that you just f*cked up changed Keycloak master realm admin theme to "base" and now you cry out loud deeply regret it 😅.

Here is how to bring everything's back without neither restarting Keycloak and restoring a backup.

First thing first connect to your database (hopefully PostgreSQL if you are decent) and run this SQL query:

UPDATE public.realm SET admin_theme = null WHERE id ='master'

Are we done yet? Nope, as you may know Keycloak has its own internal cache, you will have to flush it with the script below (note that jq is required and also don't forget to change username, password and endpoint):


# first retrieve a new token
export ACCESS_TOKEN=$(curl -X POST "$endpoint/auth/realms/master/protocol/openid-connect/token" \
 -H "Content-Type: application/x-www-form-urlencoded" \
 -d "username=$username" \
 -d "password=$password" \
 -d 'grant_type=password' \
 -d 'client_id=admin-cli' | jq -r '.access_token')

# then clear keycloak internal realm cache
curl -v -X POST "$endpoint/auth/admin/realms/master/clear-realm-cache" \
-H "Accept: application/json" \
-H "Authorization: Bearer $ACCESS_TOKEN"

Refresh your Keycloak console interface. Yep. You're welcome 👍

Made with on a hot august night from an airplane the 19th of March 2017.