So far the walls have stood the test of time... but I haven't handled them very roughly at all, and I haven't traveled anywhere with them.
I should be taking them to a game this week, so I can let you know how they travel towards the end of the week.
The 'stones' are only held together by white glue (PVA glue in my part of the world) and then painted. The good thing about the kitty litter is it only cost me $2 for a 2kg bag, which should last me a life time.
I am also using the smaller pieces of litter to make rubble, so it gets a lot of use.