Skype I know is infamously hard, but I'm wondering about the more general case -- are the problems there similar to the problems with Skype, or are they due to something else?
As far as I know, they're a combination of it being hard work and there just not being enough of a qualified developer community around Pidgin etc. to put in the effort required. I think people generally underestimate the sheer amount of work involved in reverse engineering an IM protocol, even one that's not particularly well obfuscated.
But what about other projects? Back in the day there were multiple multi-protocol clients. Pidgin (or rather Gaim back then), Trillian, and others; now there's... maybe Franz, as cosmie mentions? Simply saying "Pidgin kind of died off" doesn't seem like enough of an explanation here; why aren't there more such projects popping up?
And you've hit the nail on the head: there's just a lot less _demand_ for multi-protocol clients these days. Official clients are much better than they were in the 1990s. Like piracy: undesired use of a protocol is a service problem, one that modern IM clients are actually addressing.
That hardly seems like much of an explanation. Look at the old protocols Pidgin and other clients of the time (e.g. Trillian) spoke -- AOL, MSN, Yahoo... those didn't run over HTTP. More importantly, the companies did respond by changing the protocols; the client-writers kept up all the same. So that's not something that's different between then and now.