The most interesting part is that there is no direct line between someone's accidental death and a chatbot giving life-threatening advice.
Imagine one of the models that has "accidental-deaths-via-bad-advice" just slightly turned up, with the model-provider's intent being to kill 5% more people per year.
If you're paranoid (or a hawk), imagine a Chinese LLM that only offers fatal advice when queried English, or an American LLM that only offers fatal advice when queried in Chinese. Or American and Russian models which only offer up fatal advice when queried in German, Finish, or Danish.
Imagine one of the models that has "accidental-deaths-via-bad-advice" just slightly turned up, with the model-provider's intent being to kill 5% more people per year.