As someone who doesn't know much about it, would a malicious homeserver have any way to break the end to end encryption, save for putting massive amounts of men in the middle between each conversation which would be busted the second anyone decides to compare public keys?
Clients always need a way to re-negotiate keys in case one user decides to switch to new device and not import keys.
The problem is, say a MITM attack really happens. What is the user going to do? Will they assume it's a bug, will they press "accept" blindly without checking the fingerprints? Will they contact The Intercept to reveal MITM attacks are taking place? Will the press take their claims seriously? Will anyone on the internet drowning in misinformation, disinformation really care?
Then there's the key exfiltration problem. Snowden was very vocal about it at the beginning of the leaks. The NSA goes around E2EE by hacking endpoints and stealing keys. That's not a reason not to use E2EE of course, there are simply more threats with less capabilities but still, threats that are insane amounts of less risk-averse, who won't give two shits about e.g. abusing plaintext data exfiltrated from the server. My point is if you have to deal with risks such as MITM you're probably dealing with adversaries willing to infect your endpoint as well, so you might not detect MITM attacks. For that you need better endpoint security architecture with some guarantees against key/pt exfiltration.