Today I discovered a (insane 🤯) bug in Google Cloud Run.
I was trying to understand WHY 😵💫 Image-Charts works on Google Kubernetes Engine (GKE) but the same container does not work on Cloud-Run.
To be precise, why the signed URL (HMAC) below works on GKE and not on Cloud-Run
https://image-charts.com/chart?chbr=8&chd=t%3A10%2C15%2C25%2C30%2C40%2C80&chf=b0%2Clg%2C90%2C05B142%2C1%2C0CE858%2C0.2&chl=%7C%7C%7C%7C%2033%25%20%21%7Cx2%20&chma=0%2C0%2C10%2C10&chs=700x450&cht=bvs&chtt=Revenue%20per%20month&chxl=0%3A%7CJan%7CFev%7CMar%7CAvr%7CMay&chxs=1N%2AcUSD0sz%2A%2C000000%2C14&chxt=x%2Cy&icac=fgribreau&ichm=d92a75886c0657013da32eda4d82b8db3af39d355b3419caebd5952bc827990d
Note that HMAC signature (ichm query parameter) is computed like this:
ichm=sign(url.search, shared_secret)
My main intuition was that cloud-run had some internal reverse proxies that changes part of the URL on the fly.
If that's trye, let's find what encoded characters Cloud-Run would automatically decode in the URL query part before reaching your app:
new Array(155).join('-').split('-').map((x, i) => '%'+i.toString(16).toUpperCase()).join('-_-')
// output:
// %0-_-%1-_-%2-_-%3-_-%4-_-%5-_-%6-_-%7-_-%8-_-%9-_-%A-_-%B-_-%C-_-%D-_-%E-_-%F-_-%10-_-%11-_-%12-_-%13-_-%14-_-%15-_-%16-_-%17-_-%18-_-%19-_-%1A-_-%1B-_-%1C-_-%1D-_-%1E-_-%1F-_-%20-_-%21-_-%22-_-%23-_-%24-_-%25-_-%26-_-%27-_-%28-_-%29-_-%2A-_-%2B-_-%2C-_-%2D-_-%2E-_-%2F-_-%30-_-%31-_-%32-_-%33-_-%34-_-%35-_-%36-_-%37-_-%38-_-%39-_-%3A-_-%3B-_-%3C-_-%3D-_-%3E-_-%3F-_-%40-_-%41-_-%42-_-%43-_-%44-_-%45-_-%46-_-%47-_-%48-_-%49-_-%4A-_-%4B-_-%4C-_-%4D-_-%4E-_-%4F-_-%50-_-%51-_-%52-_-%53-_-%54-_-%55-_-%56-_-%57-_-%58-_-%59-_-%5A-_-%5B-_-%5C-_-%5D-_-%5E-_-%5F-_-%60-_-%61-_-%62-_-%63-_-%64-_-%65-_-%66-_-%67-_-%68-_-%69-_-%6A-_-%6B-_-%6C-_-%6D-_-%6E-_-%6F-_-%70-_-%71-_-%72-_-%73-_-%74-_-%75-_-%76-_-%77-_-%78-_-%79-_-%7A-_-%7B-_-%7C-_-%7D-_-%7E-_-%7F-_-%80-_-%81-_-%82-_-%83-_-%84-_-%85-_-%86-_-%87-_-%88-_-%89-_-%8A-_-%8B-_-%8C-_-%8D-_-%8E-_-%8F-_-%90-_-%91-_-%92-_-%93-_-%94-_-%95-_-%96-_-%97-_-%98-_-%99-_-%9A
To run this experiment I've set a special route on Image-Charts that outputs what Image-Charts http server got as part of url.search.
On Google Kubernetes Engine (GKE), Image Charts API got an unmodified query string:
%0-_-%1-_-%2-_-%3-_-%4-_-%5-_-%6-_-%7-_-%8-_-%9-_-%A-_-%B-_-%C-_-%D-_-%E-_-%F-_-%10-_-%11-_-%12-_-
%13-_-%14-_-%15-_-%16-_-%17-_-%18-_-%19-_-%1A-_-%1B-_-%1C-_-%1D-_-%1E-_-%1F-_-%20-_-%21-_-%22-_-
%23-_-%24-_-%25-_-%26-_-%27-_-%28-_-%29-_-%2A-_-%2B-_-%2C-_-%2D-_-%2E-_-%2F-_-%30-_-%31-_-%32-_-
%33-_-%34-_-%35-_-%36-_-%37-_-%38-_-%39-_-%3A-_-%3B-_-%3C-_-%3D-_-%3E-_-%3F-_-%40-_-%41-_-%42-_-
%43-_-%44-_-%45-_-%46-_-%47-_-%48-_-%49-_-%4A-_-%4B-_-%4C-_-%4D-_-%4E-_-%4F-_-%50-_-%51-_-%52-_-
%53-_-%54-_-%55-_-%56-_-%57-_-%58-_-%59-_-%5A-_-%5B-_-%5C-_-%5D-_-%5E-_-%5F-_-%60-_-%61-_-%62-_-
%63-_-%64-_-%65-_-%66-_-%67-_-%68-_-%69-_-%6A-_-%6B-_-%6C-_-%6D-_-%6E-_-%6F-_-%70-_-%71-_-%72-_-
%73-_-%74-_-%75-_-%76-_-%77-_-%78-_-%79-_-%7A-_-%7B-_-%7C-_-%7D-_-%7E-_-%7F-_-%80-_-%81-_-%82-_-
%83-_-%84-_-%85-_-%86-_-%87-_-%88-_-%89-_-%8A-_-%8B-_-%8C-_-%8D-_-%8E-_-%8F-_-%90-_-%91-_-%92-_-
%93-_-%94-_-%95-_-%96-_-%97-_-%98-_-%99-_-%9A
On Google Cloud Run however, things are different:
%0-_-%1-_-%2-_-%3-_-%4-_-%5-_-%6-_-%7-_-%8-_-%9-_-%A-_-%B-_-%C-_-%D-_-%E-_-%F-_-%10-_-%11-_-%12-_-
%13-_-%14-_-%15-_-%16-_-%17-_-%18-_-%19-_-%1A-_-%1B-_-%1C-_-%1D-_-%1E-_-%1F-_-%20-_-!-_-%22-_-
%23-_-%24-_-%25-_-%26-_-%27-_-(-_-)-_-*-_-%2B-_-%2C-_---_-.-_-%2F-_-0-_-1-_-2-_-
3-_-4-_-5-_-6-_-7-_-8-_-9-_-%3A-_-%3B-_-%3C-_-%3D-_-%3E-_-%3F-_-%40-_-A-_-B-_-
C-_-D-_-E-_-F-_-G-_-H-_-I-_-J-_-K-_-L-_-M-_-N-_-O-_-P-_-Q-_-R-_-
S-_-T-_-U-_-V-_-W-_-X-_-Y-_-Z-_-%5B-_-%5C-_-%5D-_-%5E-_-_-_-%60-_-a-_-b-_-
c-_-d-_-e-_-f-_-g-_-h-_-i-_-j-_-k-_-l-_-m-_-n-_-o-_-p-_-q-_-r-_-
s-_-t-_-u-_-v-_-w-_-x-_-y-_-z-_-%7B-_-%7C-_-%7D-_-~-_-%7F-_-%80-_-%81-_-%82-_-
%83-_-%84-_-%85-_-%86-_-%87-_-%88-_-%89-_-%8A-_-%8B-_-%8C-_-%8D-_-%8E-_-%8F-_-%90-_-%91-_-%92-_-
%93-_-%94-_-%95-_-%96-_-%97-_-%98-_-%99-_-%9A
Yep. Characters like ")", "(", "a", "~" were converted on the fly by some Cloud Run internal proxy.
Now lets filter this for better readability:
const search_raw = new Array(155).join('-').split('-').map((x, i) => '%'+i.toString(16).toUpperCase());
const search_processed_by_cloud_run = `%0-_-%1-_-%2-_-%3-_-%4-_-%5-_-%6-_-%7-_-%8-_-%9-_-%A-_-%B-_-%C-_-%D-_-%E-_-%F-_-%10-_-%11-_-%12-_-%13-_-%14-_-%15-_-%16-_-%17-_-%18-_-%19-_-%1A-_-%1B-_-%1C-_-%1D-_-%1E-_-%1F-_-%20-_-!-_-%22-_-%23-_-%24-_-%25-_-%26-_-%27-_-(-_-)-_-*-_-%2B-_-%2C-_---_-.-_-%2F-_-0-_-1-_-2-_-3-_-4-_-5-_-6-_-7-_-8-_-9-_-%3A-_-%3B-_-%3C-_-%3D-_-%3E-_-%3F-_-%40-_-A-_-B-_-C-_-D-_-E-_-F-_-G-_-H-_-I-_-J-_-K-_-L-_-M-_-N-_-O-_-P-_-Q-_-R-_-S-_-T-_-U-_-V-_-W-_-X-_-Y-_-Z-_-%5B-_-%5C-_-%5D-_-%5E-_-_-_-%60-_-a-_-b-_-c-_-d-_-e-_-f-_-g-_-h-_-i-_-j-_-k-_-l-_-m-_-n-_-o-_-p-_-q-_-r-_-s-_-t-_-u-_-v-_-w-_-x-_-y-_-z-_-%7B-_-%7C-_-%7D-_-~-_-%7F-_-%80-_-%81-_-%82-_-%83-_-%84-_-%85-_-%86-_-%87-_-%88-_-%89-_-%8A-_-%8B-_-%8C-_-%8D-_-%8E-_-%8F-_-%90-_-%91-_-%92-_-%93-_-%94-_-%95-_-%96-_-%97-_-%98-_-%99-_-%9A`.split('-_-');
search_raw.reduce((m, encoded, i) => encoded === search_processed_by_cloud_run[i] ? m : m.concat([encoded, search_processed_by_cloud_run[i]].join(' => ')), []).join('\n')
// output
%21 => !
%28 => (
%29 => )
%2A => *
%2D => -
%2E => .
%30 => 0
%31 => 1
%32 => 2
%33 => 3
%34 => 4
%35 => 5
%36 => 6
%37 => 7
%38 => 8
%39 => 9
%41 => A
%42 => B
%43 => C
%44 => D
%45 => E
%46 => F
%47 => G
%48 => H
%49 => I
%4A => J
%4B => K
%4C => L
%4D => M
%4E => N
%4F => O
%50 => P
%51 => Q
%52 => R
%53 => S
%54 => T
%55 => U
%56 => V
%57 => W
%58 => X
%59 => Y
%5A => Z
%5F => _
%61 => a
%62 => b
%63 => c
%64 => d
%65 => e
%66 => f
%67 => g
%68 => h
%69 => i
%6A => j
%6B => k
%6C => l
%6D => m
%6E => n
%6F => o
%70 => p
%71 => q
%72 => r
%73 => s
%74 => t
%75 => u
%76 => v
%77 => w
%78 => x
%79 => y
%7A => z
%7E => ~
I reported this bug to Steren (Product Manager at Google Cloud Run) and hopefully it will be fixed soon :)