thoughts on the latest cloudflare outage

2025-11-21T20:06:00-08:00

against my best instincts, i will put forward my opinion on the recent cloudflare outage. you should probably read their postmortem</a> for context.

intro and background</h2>

tl;dr is that a quarter of the internet went down because a database change caused oversized config files to be propagated to a critical component of their core proxy. this component used a fixed amount of preallocated memory as an optimization, and nothing validated the size of the config it was parsing. when the oversized config was loaded, the application crashed, which translated into an http 500 server error, which blocked all traffic attempting to traverse that component. in other words:

cloudflare:</span> i would like to parse dynamic config generated by an external service
</span>        </span>me:</span> fair enough
</span>cloudflare:</span> i would like to restrict the size of this config, for performance
</span>        </span>me:</span> fair enough
</span>        </span>me:</span> what will you do if the config is too big?
</span>cloudflare:</span> die instantly
</span>        </span>me:</span> ah.
</span></code></pre>

spooky scary sloppy rust code</h2>
the snippet of code provided in the postmortem is as follows:</p>
pub fn </span>fetch_features</span>(
</span>    &</span>mut </span>self</span>,
</span>    </span>input</span>: &dyn BotsInput,
</span>    </span>features</span>: &</span>mut</span> Features,
</span>) -> Result<(), (ErrorFlags, </span>i32</span>)> {
</span>    features.checksum &= </span>0xFFFF_FFFF_0000_0000</span>;
</span>    features.checksum |= </span>u64</span>::from(</span>self</span>.config.checksum);
</span>    </span>let </span>(feature_values, _) = features
</span>        .</span>append_with_names</span>(&</span>self</span>.config.feature_names)
</span>        .</span>unwrap</span>();
</span></code></pre>
the problematic part is that last line: .unwrap()</code>. in rust, errors are handled using sum types, namely, Result<T, E></code>. this Result</code> object can then be dealt with in various ways: either explicitly with match</code>, if let</code>, or let ... else</code>; automatically with ?</code> or .unwrap_or_else()</code>; dangerously with .unwrap()</code>; or unsafe</code>ly with .unwrap_unchecked()</code>. here's how each would have looked; don't mind the owl references, we'll get to it:</p>
//we don't know the actual return type of .append_with_names(), 
</span>//so this is a guesstimate:
</span>let</span> res: Result<(Vec<FeatureValues>, Unknown), AppendError> 
</span>    = features.</span>append_with_names</span>(&</span>self</span>.config.feature_names);
</span>
</span>//handle the error directly
</span>match</span> res {
</span>    Ok((feature_values, _)) => </span>/*do whatever you needed to do here*/</span>,
</span>    Err(error) => {
</span>        log_error!(error, "</span>failed to append features when loading config</span>");
</span>        </span>//draw the rest of the owl
</span>    },
</span>}
</span>
</span>//find a safe default
</span>let </span>(feature_values, _) = res.</span>unwrap_or_else</span>(|</span>error</span>| {
</span>    log_error!(error, "</span>failed to append features when loading config</span>");
</span>    (vec![], Default::default()) </span>//draw the rest of the owl
</span>});
</span>
</span>//bubble it up - let the caller draw the rest of the owl
</span>let </span>Ok((feature_values, _)) = res </span>else </span>{
</span>    log_error!("</span>failed to append features when loading config</span>");
</span>    </span>return </span>Err((ErrorFlags::AppendError, </span>120</span>)) 
</span>    </span>//who knows what this error return should be
</span>};
</span>
</span>//another way to bubble it up, more conveniently but without a local log 
</span>//(assuming the error type is convertible)
</span>let </span>(feature_values, _) = res?;
</span>
</span>//the bare minimum - cloudflare's actual solution.
</span>//crash the program if you can't append
</span>let </span>(feature_values, _) = res.</span>unwrap</span>();
</span>
</span>//more or less equivalent to the above, with slightly better debuggability
</span>let </span>(feature_values, _) = res.</span>expect</span>(
</span>    "</span>there should be a max of 200 feature values</span>"
</span>);
</span>
</span>//abdicate all responsibility. perform undefined behaviour if you can't append. 
</span>//here there be memory corruption. summon the nasal demons.
</span>//thankfully cloudflare had the sense not to do this
</span>let </span>(feature_values, _) = </span>unsafe </span>{
</span>    res.</span>unwrap_unchecked</span>()
</span>};
</span></code></pre>
what's wrong with unwrapping</h2>
unwraps are to be used in one of two1</a></sup> scenarios:</p>

you do not care about your code very much. if it crashes, who cares</li>
the error is believed to be completely impossible. there is an invariant being upheld that rust's type system is insufficiently smart to prove. if the value is an error, something has gone catastrophically wrong and no assumptions can be made about the state of the program. the only safe response is to panic, unwind the stack and kill the entire thread. we can only pray that is enough.</li>
</ol>
this unwrap falls in neither of these. cloudflare's core proxy is critical infrastructure and thus must make every effort not to crash, eliminating the first point. the second is where cloudflare-positive discourse seems to have placed them - the engineer and code reviewer must have believed it impossible to fail here. i disagree - in a situation where you are parsing input provided from an external system, generated by a database, across a network boundary, no assumptions can be made. operate as if it is, if not actively malicious, at least highly suspicious.</p>
cloudflare's core proxy code should probably have zero potential panics in it. this makes it more difficult to write rust, i understand that. if they must write panicking code, at the very least</em> it should have safety-style comments above each potential panic indicating why the engineer believes a panic is impossible.</p>
additionally, this component of the core proxy was a filter designed to keep llm scrapers and other (semi-)malicious bots from accessing a client site. in the event of a catastrophic failure like this, the component should obviously</em> fail open - bot management is not a reliable security boundary, and increasing load / visibility is far superior to taking the website off the internet altogether. present one of those horrible little javascript challenges on each page load if you must (i'm thinking of ticket-sellers looking to avoid being bought out by bots), but don't break the web. different components have different threat models; i'm sure e.g. their zero trust product would prefer failing closed.</p>
crux</h2>
the core of the issue is an unstated, unvalidated assumption - the config will never exceed 200 features. rust did its best to warn the coder about this issue; the coder responded by explicitly ignoring the concern, and this outage is the result. a sufficiently conscientious programmer and reviewer would have looked at this Result</code>, thought "how might i handle this properly", and pursued a proper solution.</p>
now we finally return to "draw the rest of the owl"; it is not obvious what this solution should have been. i have several ideas: there could have been a validator in the config generator or distributor, plus a //SAFETY</code> comment immediately before this unwrap; that would have prevented the outage. i'm hesitant about this, though - if the validator fails, we're back where we started. the solution could have also been a configuration manager that stores past versions, so the proxy could revert to a previous known-good config in the event of a misconfiguration like this.</p>
how i would have fixed this</h2>
configuration is about defining a point in multidimensional space</strong>. this point is fed to your program to define its initial state and rules about how it handles input and output. if configuration fails, a critically important program should continue to work at all costs, in this case by statically choosing an acceptable default point in the state-space, inhabiting it, and logging a maximum-severity error message. ideally, such a message would trigger an alert and page an on-call engineer to fix the problem immediately, but until then, nothing is fundamentally broken. for this program, the sensible static default is "allow every request through".</p>
postscriptum - response to "this is rust's fault"</h2>
if this had been c, cloudflare would have failed to check the error code and proceeded to perform undefined behaviour. whee, memory safety cves! if this had been java, javascript, python, go, haskell, c#, kotlin, lisp, etc., this program would have been ripped out and rewritten in rust; if cloudflare is so memory-constrained as to only allow 200 entries in an array, a garbage-collected language would have been right out. if this had been c++, it would have thrown an unchecked exception, same as the panic, and the outage would have been exactly the same. maybe zig solves this; i haven't written any zig.</p>
rust is not a perfect language. the naming of .unwrap()</code> is suboptimal; error handling in tutorials and educational work is often brushed under the rug with it. maybe in rust 2.0, which will never happen, we can rename unwrap. until then, it is the professional responsibility of engineers, at least those who create programs which must have extremely high reliability, to understand the language in which they work and not litter their code with potential crashes.</p>




there is a third^{2</a></sup>, false option: if you handle panics with catch_unwind</code></a> or a panic hook, and you will convert the panic into an error 500 or something. cloudflare probably had this, it's common in web frameworks. this is not something you want to rely on. panic catching is dangerous; if you panic within a panic (either in a Drop</code> implementation or the panic handler) it hard-aborts your program immediately. also, responding with a 500 is not actually the correct response in this case! as i said above, this filter should have failed open, not responded to every request with a 500. listen to the rust documentation: "it is not recommended to use this [...] for a general try/catch mechanism" ↩</a></p>
</li>}

another, even less defensible idea: you are allowed to panic if there is no sensible way of responding to such an error. i disagree with this - in a webserver, you want to respond to the client no matter what; in a user-driven app with a gui, tui, or cli, splatting a panic stacktrace is not a friendly user experience; in a library you should really</em> never panic but always return some indication of failure, etc. ↩</a></p>
</li>
</ol>
</section>

on correctness

2025-11-17T13:00:00-08:00

correctness must be (normative) baseline in software. luckily, correctness is a mathematical property of an algorithm, and thus we can use computers to help humans build correct software. millions of lines of code have been written for the express purpose of helping humans prove subsets of correctness. i'm thinking of (ranked by how much assurance they provide, in decreasing order): formal verification, type systems, linters, property testing, unit testing, fuzzers, autoformatters, syntax highlighters.

formal verification is the highest standard of correct software. i do not have direct experience with it. to my understanding, it converts many programming errors into much fewer specification errors, which tend to be easier to solve. </li>

type systems, when implemented kindly, are the best tool for correctness. they enable local reasoning, tooling support, invariants. when implemented with the sharp edge against the programmer, the programmer learns to hate them. the difference between a kind type system and a sharp type system is how the programmer feels about the type system. if the programmer writhes in its grasp, flailing against the oppressive all-seeing eye of "computer says no", the type system is not helping achieve correct software. if the programmer uses the type system as a tool, sketching their software out in broad strokes, knowing with certainty the potential states and being assured of the correctness of composition, it's good.

i have a distaste for gradual typing systems, as the flow goes something like this: working, highly dynamic program -> overlaid type system incorrectly proclaims incorrectness -> programmer develops hatred for type systems</li> </ol> </li>

linters are a good way to soften the blow of a compiler error. moving from "computer says no" to "computer says you're wrong, but here's an escape hatch labelled 'do not use'". the downside of this approach is obvious, however: programmers are creatures of infinite creativity and chaotic alignments. when deadlines are imposed, the programmer is burnt out, or years of useless lints have jaded their saliency checker, lints are ignored and software is built incorrectly. equally, escape hatches allow lint writers to be more careless about false positives, pushing the vicious cycle one step further </li>

property testing is interesting to me. i have used it but lightly, and never in anger. from my perspective, property testing gives you 80% of the assurance of formal verification with 20% of the effort. of course, an 80% correct program is rarely good enough. the core conceit is to formalize properties about your functions, formalize ways of generating input to said functions, and machine-checking the whole shebang. </li>

i have described unit testing suites as brittle exoskeletons attached to the nerves of a program. with the slightest twitch, a change to a program can shatter its exoskeleton to dust while self-inflicting great pain. the result of this, and thus the purpose of unit testing systems, is to disincentivise change of any kind. have fun migrating off .net 6.

however, i have had the immense displeasure of refactoring a complicated program with no test suite whatsoever. combined with the total lack of documentation, hyrum</a> ensured that any minor modification would break something with no warning. therefore, i consider unit testing to be something of a necessary evil, and keep hope that a better world is possible.</li>
property testing suffers from the same problem in theory, yet in practice the properties are sufficiently abstract as to allow for enough program movement so as to not atrophy.</li> </ol> </li>

fuzzers are a reduced form of property testing applied to an entire program, usually with some level of codepath-aware analysis to minimize breadth-first-search-itis. the exoskeleton is kept at a safe distance from the slithering muscles beneath, which allows for major refactors that don't change the basic api. however, they don't check correctness. fuzzers can root out memory safety bugs, edge case crashes, and input mishandling, for which they are invaluable, but a fuzzer on its own isn't good enough. </li>

autoformatters, like type systems, are best when they are human-focused and not impositional. an autoformatter offloads mental effort from the programmer by keeping a program in a consistent style - the writer doesn't have to fuss about indentation and the reader can load the program into their mental approximation without being distracted by a format different from the one they use. it serves the same purpose as typesetting rules at a scientific journal. the formatter has to be universally applied. it must be consistent and disrespect programmer intentions to be valuable, but its style must be smart enough to not cause anger.

autoformatters can also catch errors themselves, usually by papering over language-level problems. goto fail</a>, while indicative of a half-dozen problems with the c language (unreachable code is legal, if statements don't require braces, goto in general, rampant mutability, error codes</a>), would have nevertheless been caught by a human reviewer if the code was autoformatted.</li> </ol> </li>

syntax highlighting instantly catches syntax-level errors without requiring a full compiler pass or linter run. it's the fastest feedback we have available - misspelled a keyword? it's the wrong colour, your brain knows something's wrong before you can consciously respond to it. i consider consciousness a slow and error-prone1</a> method of cognition, and short-circuiting it lets you write better code faster. </li> </ol>

this requires some explanation. when you're doing your best work, you "lose yourself" in it. no one considers a self-conscious or self-aware coding style any good. our consciousness lies, makes up narratives, and generally runs interference. of course, preconscious cognition is much less sophisticated, and so conscious composition is indispensable. for a longer-form meditation on consciousness as a liability, try reading blindsight</a> ↩</a> </li> </ol> </section>

nove.dev

advent of code 2025

thoughts on the latest cloudflare outage

on correctness

software engineers don't exist

catalogue of personal computation

blogpost about static site generation

nove.dev

advent of code 2025

thoughts on the latest cloudflare outage

spooky scary sloppy rust code</h2> the snippet of code provided in the postmortem is as follows:</p>

on correctness

software engineers don't exist

catalogue of personal computation

blogpost about static site generation