I don't think that xkcd applies. I mean if I am on a clothing seller's website, and they show me a pair of pants and I add 4 colors to the cart of the same size of the same design/model number/style pants, I feel like they should all fit the same.

They usually do, never found that issue on medium to high quality clothing (!= luxury brands).

I have. So I don't buy that it's not a problem just because you haven't had it..

Sometimes they also use different fabric for different colors. Maybe a color is internally consistent, but if you buy one of each they're very different.