asymptotes (1) barbican (1) blogging about blogging (3) bracken house (1) burnout (3) camping (3) chess (1) choir (5) climbing (1) films (4) floorp (1) games (2) greenbelt (1) holiday (1) hospital (1) inlaws (1) kettricken (2) keyboards (1) lasers (1) leicester (1) london (1) meetings (1) melting (1) metal gear (1) minecraft (2) modular (5) music (4) musical (1) need for speed (1) operation mincemeat (1) origami (2) ourple (1) peak district (1) photos (1) platforms (1) secrets (1) sick (1) skin (1) supercars (1) synth repair (1) tech lead (2) theatre (1) tool kit (1) weeknotes (22) work (5)
oh wow it’s actually joever, didn’t see that coming.
we’ve been trying to diagnose a Weird Problem at work, the kind you’d see in breathless Medium posts or SCP containment reports. me, Ivo and Rowan had each spent a few days trying to understand what was going on before getting burnt out, and the CDN support engineers were clueless.
on tuesday i got everyone together around the breakout area table in the office, trying out stupid things like editing the CDN config live to add more logging and point it to a temporary app i wrote and deployed in five minutes to log full request details we weren’t getting from the CDN logs themselves. by the end of the day things were pointing to the WAF as the culprit. the next day we narrowed it down with the help of a WAF support engineer to the WAF’s CDN loop detection (crucially not the CDN’s CDN loop detection, which we’d already configured our way around).
Read more…