Granite, for instance, has a 1200c melting point which sounds good. Its large grainy structure might cause porosity issues. And it's main constituents are silicon and aluminum, both of which do transmute under neutron bombardment. (An expert could say if they transmute to anything you really don't want around, and how long it'll be there.)

I think ceramics are an answer to the porosity and similar issues. Consistent structure, no inclusions or other weaknesses. But they would face the same transmutation and induced radioactivity problems as stone of the same material.