XSLT 3.0 can be directed to output HTML5 [0]. However, browsers only implement XSLT 1.0, and as far as I am aware there is no open-source XSLT 3.0 implementation.
Still, it's possible with XSLT 1.0 to produce documents in the common subset of XML and HTML5 ("XHTML5"). It can't produce the usual <!DOCTYPE html> at the top of the document, but it can produce the alternative <!DOCTYPE html SYSTEM "about:legacy-compat">.
On the input side, every XSLT version only accepts valid XML, as far as I am aware.
`xsltproc --html` is an example of HTML input (probably HTML4 parsing rules though?) if you really need it. This is an XSLT 1.0 processor, wrapping libxslt which most browsers use.
As for output, the difference is largely irrelevant for browser purposes since they just want a tree.
I'm not sure how many extensions the browsers allow, but a major part of the reason XSLT 2/3 failed to take off is because libxslt already provides most of the useful features from newer versions as extensions (many via EXSLT-namespaced modules, at least partially supported in browsers - see MDN); what it doesn't do is implement the unnecessary complexity that the Java world loves.
Still, it's possible with XSLT 1.0 to produce documents in the common subset of XML and HTML5 ("XHTML5"). It can't produce the usual <!DOCTYPE html> at the top of the document, but it can produce the alternative <!DOCTYPE html SYSTEM "about:legacy-compat">.
On the input side, every XSLT version only accepts valid XML, as far as I am aware.
[0] https://www.w3.org/TR/xslt-xquery-serialization-30/#html-out...