Complains about TLS inspection, yet fronts their website on the biggest and most widely deployed TLS introspection middle box in the world ...
Why do we all disdain local TLS inspection software yet half the Internet terminates their TLS connection at Cloudflare who are most likely giving direct access to US Intelligence?
It's so much worse as it's infringing on the privacy and security of billions of innocent people whilst inspection software only hurts some annoying enterprise folks.
I wish we all hopped off the Cloudflare bandwagon.
Three of the banks I use have their websites/apps go through CloudFlare. So does the electronic records and messaging system used by my doctor. A lawyer friend uses a secure documents transfer service that is protect by guess who.
Who needs to let CF directly onto their network when they already sit between client and provider for critically-private, privileged communications and records access?
> Putting Cloudflare anti-DDoS in front of your website is not the same as breaking all encryption on your internal networks.
You misunderstood, they're complaining about it as a user. If your website uses Cloudflare then our conversation gets terminated by Cloudflare, so they get to see our unencrypted traffic and share it with whomever they want, compromising my privacy.
One thing that has not quite been mentioned in the blog, is how much of the MITM spyware comes from very big and well known „security“ companies.
You know, the ones that really know about security. X-PAN-AUTHCHECK type of security.
The amount of CVEs some of the big firewall companies collect make it seem like it is a competition for the poorest security hygiene.
The real problem we have is compliance theatre where someone in management forces these solutions onto their IT department just so they can check a box on their sheets and shift all responsibilities away.
The fact that most tools have completely different ways to allow them to add certificates is the biggest pain. Git, Python and Rust also have large issues. Git doesn't default to "http.schannel". Python (or rather requests, or maybe urllib3) only looks at its own certificate store, and I have no idea how Rust does this (well, I use uv, and it has its own problems - I know about the --use-native-tls flag, but it should be a default at the least).
Yeah, and Java has its nice cacerts file so that should have been easy, but then we were using Bazel which does the "hermetic builds" thing so that had to be told about it seperately, and on and on with all the other special-snowflake tools.
It added huge amounts of friction which was one reason I decided to move on from that gig.
It's such a nightmare at my current job as well. Everything always just breaks and needs investigating how to fix.
Even putting aside the MITM and how horrendous that is, the amount of time lost from people dealing with the fallout got to have cost so much time (and money). I can't fathom why anyone competent would want to implement this, let alone not see how much friction and safety issues it causes everywhere.
I have this similar gripe when it comes to http proxy configuration. It's invisible to you until you are in an execution environment where you are mandated to use the providers proxy configuration.
Some software reads "expected" env variables for it, some has its own config or cli flags, most just doesn't even bother/care about supporting it.
Chiefly because "supporting it" requires a full JavaScript interpreter, and subscribing to changes in "system settings" during the lifetime of your program. Easier just to support http_proxy/https_proxy/no_proxy (and what standard for no_proxy? Does it support CIDR ranges?) or even less flexibility than that.
And many things break in different, exciting ways. For example, we discovered that whilst the JVM can be configured to use system certificate store, that does not apply to websocket connections. So the product seems to be able to connect to the server, but all socket connections bork out with TLS errors.
I remember at my first job, the internet stopped working at my workstation. I got on the phone with IT, and the guy said "looks like you don't have our new certificates." I asked why I would need my employer's certificates. He said "because we MITM every connection." I asked if that was even legal, and he said yes it's legal.
At another job I was handling a support ticket where a customer was asking, in so many words, "can I get HTTP headers of requests flowing through my Envoy TLS reverse proxy?" I said that they could terminate TLS at the proxy and redo things that way, but then that wouldn't be a TLS proxy it'd be a MITM or a gateway. They could log the downstream/upstream and duration of connections, but that wouldn't help.
I'm hoping this doesn't apply to thigs like Fiddler, because without the ability to see what's actually coming over the wire with a https connection, things can be a nightmare to debug sometimes
I agree with the sentiment, but I think it's a pretty naive view of the issue. Companies will want all info they can in case some of their workers does something illegal-inappropiate to deflect the blame. That's a much more palpable risk than "local CA certificates being compromised or something like that.
And some of the arguments are just very easily dismissed. You don't want your employer to see you medical records? Why were you browsing them during work hours and using your employers' device in the first place?
Does GDPR (or similar) establish privacy rights to an employee’s use of a company-owned machine against snooping by their employer? Honest question, I hadn’t heard of that angle. Can employers not install EDR on company-owned machines for EU employees?
They can, but the list of "if..." and "it depends..." is much longer and complicated, especially when getting to the part how the obtained information may be used
> Consider this - what is the likelihood of every certificate authority on the Internet having their private keys compromised simultaneously?
Considering that CloudFlare has managed to MitM a huge part of the internet, I'd say that probability is not just non-zero, but greater than by a worrying margin.
> Consider this - what is the likelihood of every certificate authority on the Internet having their private keys compromised simultaneously? I’d wager that’s almost at the whatever is the statistics equivalent of the Planck length level of probability.
It doesn't matter if every certificate authority is compromised or just one. One is all that is needed to sign certificates for all websites.
The whole point of the logs is that they're tamper-evident. If you think the certificate you've seen wasn't logged you can show proof. If you think the logs tell you something different from everybody else you can prove that too.
It is striking that we don't see that. We reliably see people saying "obviously" the Mossad or the NSA are snooping but they haven't shown any evidence that there's tampering
I work for a school. My traffic is not MITM'd, but the kids' traffic is, because we don't want them using their school issued laptops to play games or go shopping, and you can't adequately block stuff if it's all encrypted.
Lame on user machines, but sometimes needed in a server environment. Easier to detect if someone is hauling off with your database as that will be the one you can’t see what’s going on. Of course, solve one problem and introduce three more.
Our cyber team have installed zscaler on most people's laptop, and somewhere in the fabric of the office internet connection.[1]
For those that don't know, its a MITM proxy with certificates so that it can inspect and unroll TLS traffic.
ostensibly its there to stop data exfiltration, as we've had a number of incidents where people have stolen data and sent it to competitors. (our c-suite don't have as much cyber shit installed, despite them being the ones that are both targets more, and broken the rules more....)
Now, I don't like zscaler, and I can sorta see the point of it. But.
Our cyber team is not a centre of technical excellence. They somehow managed to configure zscaler to send out the certs for a random property company, when people were trying to sign into our VPN.
this broke loads of shit and made my team (infra) look bad. The worrying part is they still haven't accepted that serving a random property company's website cert instead of our own/AWS's cert is monster fuckup, and that we need to understand _why_ that happened before trying anything again.
[1] this makes automatic pen testing interesting because everything we scan has vulnerabilities for NFS/CIFS, FTP and TCP dns.
Security team in most of the corporates is just a bunch of checklists markers, so for zscaler, crowdstrike or whatever they’re doing for compliance and/or certification and you can’t say no to it because it’s the company policy and who know better than “security” team?
It's definitely annoying if you work in enterprise, but on the flip side: the fact that these enterprise requirements exist is the main reason that TLS certificate configurability is possible at all, without which it would be dramatically harder (or impossible) to reverse engineer or do security & privacy research on mobile apps, IoT, etc etc etc.
Enterprise control over company devices and user control over personal devices are not so different.
A few apps do use certificate pinning nowadays, which creates similar problems, but saying "you can never add your own MitM TLS cert" is not far from certificate pinning everything everywhere all the time. Good luck creating a new home assistant integration for your smart airfryer when you can't read any of the traffic from its app.
Imo: let's make it easier! Standardize TLS configuration for all tools, make easy cert configuration of devices a legal requirement (any smart device sold with hardcoded CA certificates is a device with a fixed end date, where the CA certs expire and it becomes a brick), guarantee user control over their own TLS trust, and provide good tools to check exactly who you're trusting (and expose that clearly to users). Not really practical of course (and opens all sorts of risky games with nation state interception as well) but there are upsides here as well.
I largely agree with the author. When our SOC wanted to implement TLS inspection I blocked it. Mostly because we not nearly at the security level for this, but also because it just fucks with so many things.
That said, we are not a business dealing with highly sensitive data or legal responsibilities surrounding data loss prevention.
If you are a business like that, say a bank or a hospital, you want to be able to block patient / customer data leaving your systems. You can do this by setting up a regex for a known format like patient numbers or bank account numbers.
This requires TLS inspection obviously.
Though this makes it harder to steal this data, not impossible.
It does however allow the C-suite to say they did everything they could to prevent it.
Netskope and the other DLP tools at my last gig would completely lock up my network connection for around 30 seconds every hour or two while maxing out 100% of a core. Fun times. The issue was still there a year after I first encountered it so I have grave doubts about the competence of those vendors.
On the other hand I am sympathetic to the needs of big regulated orgs to show they're doing something to avoid data loss. It's a painful situation.
I agree with the sentiment, but this part is complete bullshit:
> what is the likelihood of every certificate authority on the Internet having their private keys compromised simultaneously
Who cares? It's not like all CAs would have to be breached, just one. CA certs are not scoped, so the moment one CA gets breached, we're all fucked. CT helps, but AFAIK it's still not enforced everywhere yet
There's no actual market pressure to be secure, so nobody cares about threat modeling, cost/benefit of security solutions, etc. The only pressure in case of breach is political blame that you need to deflect. The point of a cybersecurity solution is to be there, remind you it is there, and allow you to deflect blame in case of disaster. Whether it actually increases security is merely a bonus side-effect.
Honestly, the author is spot on about the normalisation problem. I've watched this play out at multiple organisations. You implement TLS inspection, spend ages getting certs deployed, and within six months `curl -k` is in half your runbooks because "it's just the corporate proxy again".
He's also absolutely right about the architectural problems too, single points of failure, performance bottlenecks, and the complexity in cloud-native environments.
That said, it can be a genuinely valuable layer in your security arsenal when done properly. I've seen it catch real threats, such as malware C2 comms, credential phishing, data exfiltration attempts. These aren't theoretical; they happen daily. Combined with decent threat intelligence feeds and behavioural analytics, it does provide visibility that's hard to replicate elsewhere.
But, and this is a massive but, you can't half-arse it. If you're going to do TLS inspection, you need to actually commit:
Treat that internal CA like it's the crown jewels. HSMs, strict access controls, proper rotation schedules, full-chain and sensible life-span. The point about concentrated risk is bang on, you've turned thousands of distributed CA keys into one single target. So act like it. Run it like a proper CA with proper key signing ceremonies and all the safeguards etc.
Actually invest in proper cert distribution. Configuration management (Ansible/Salt/whatever), golden container base images with the CA bundle baked in, MDM for endpoints, cloud-init for VMs. If you can't reliably push a cert bundle to your entire estate, you've got bigger problems than TLS inspection.
Train people properly on what errors are expected vs "drop everything and call security". Document the exceptions. Make reporting easy. Actually investigate when someone raises a TLS error they don't recognise. For dev's, it needs to just work without them even thinking about it. Then they don't need to work around it, ever. If they need to, the system is busted.
Scope it ruthlessly. Not everything needs to go through the proxy. Developer workstations with proper EDR? Maybe exclude them. Production services with cert pinning? Route direct. Every blanket "intercept everything" policy I've seen has been a disaster. Particularly for end-users doing personal banking, medical stuff, therapy sessions, do you really want IT/Sec seeing that?
Use it alongside modern defences. ie EDR, Zero Trust, behavioural analytics, CASB. It should be one layer in defence-in-depth, not your entire security strategy.
Build observability, you need metrics on what's being inspected, what's bypassing, failure rates, performance impact. If you can't measure it, you can't manage it.
But Yeah, the core criticism stands though, even done well, it's a massive operational burden and it actively undermines trust in TLS. The failure modes are particularly insidious because you're training people to ignore the very warnings that are meant to protect them.
The real question isn't "TLS inspection: yes or no?" It's: "Do we have the organisational maturity, resources, and commitment to do this properly?" If you're not in a regulated industry or don't have dedicated security teams and mature infrastructure practices, just don't bother. But if you must do it, and plenty of organisations genuinely must, then do it properly or don't do it at all.
zScaler is a load of shit, especially with some of its absolutely dumb policies like “malicious TLDs”.
Because the Framework laptop site at frame.work is malicious, of course.
God, I love CURLing crap from my workstation and not getting the files I needed but instead a bunch of mangled HTML telling me zScaler was going to scan what I was going to download.
Bonus points that it puts me in the wrong country because I’m closer to Montreal than any American locations so half the time I’m stuck in French Canadian on the web from my New York office.
Triple bonus points that I’m required to test speed at client sites and zScaler completely mangles our presentable results.
Quadruple bonus points that I put in "because I feel like it" into every elevation request I make on my corporate machine and our "cyber team" has literally never looked at elevation reports to ask what the hell I'm doing...
Got acquired by a Fortune 500 and recieved new laptop. First hour I'm seeing TLS errors everywhere except the browser.
They'd half-baked their internal CA rollout, so wasn't trusted properly.
By day two I started validating their setup. The CA literally had a typo in the company name, not a great sign.
A quick check with badssl.com showed that any self-signed(!) cert was being transparently MITM'ed and re-signed by their trusted corporate cert. Took them 40 days to fix it.
Another fun side-effect of this is that devs will just turned off TLS verification, so their codebase is full of `curl -k`, `verify_mode = VERIFY_NONE`, `ServerCertificateValidationCallback = () => true`, ... Exactly the thing you want to see at a big fintech company /s
Hey, allowing your employees to have secure connection to websites shows up in red in some Excel spreadsheet. We can't have Excel spreadsheets showing red in fintech. /s
Why do we all disdain local TLS inspection software yet half the Internet terminates their TLS connection at Cloudflare who are most likely giving direct access to US Intelligence?
It's so much worse as it's infringing on the privacy and security of billions of innocent people whilst inspection software only hurts some annoying enterprise folks.
I wish we all hopped off the Cloudflare bandwagon.
reply