I've been working remote for 3 years now in software and I agree completely.

The biggest downsides are: loneliness, feeling like a shut in, and working all the time because you're only two steps from your "office". These are NP-hard problems to solve, especially for an introverted 4-eyes like me.