Extruded polystyrene foam (the dense kind used for insulation) will take carving fairly well, to make an exterior stone structure. Some folks prefer to cut individual "stones" and build up the outer wall one stone at a time, but that's too fiddly, IMO.
If a broch weren't so round and irregular, I would even suggest using one of the textured rolling pins from Green Stuff World to do the stone texture. I've had good results on smaller pieces of high-density styrofoam. But you might get better and quicker results with an exacto blade.
Assuming you don't have access to a 3D printer. From what I've seen, with a nice 3D printer the world's your oyster.
Edit: I've been thinking more about this, and might just make my own, now that you've got me hooked.
Step 1: take a big sheet of insulation foam, cut rough squares to desired size.
Step 2: use a cheap foam cutter (like
this one) and cut circles by rotating the insulation squares on a nail. Stack 3-4 pieces to desired height.
3) the foam is about the right thickness to do one floor of the broch, assuming you want to do a playble interior, in which case you also need to cut out the centers. A few furniture pegs glued to one level will allow you to have removable floors that line up right and don't go slipping away at the wrong moment. If you just want a solid model, skip this step.
4) sculpt stonework onto the exterior of the broch with an exacto knife. Maybe do a couple practice pieces first, to train yourself to do it. My first efforts were clumsy compared to what I was doing an hour later.
5) add any details like doors and windows and roofs from materials like bank-calendar cardstock and balsa strips.
6) Paint with a thick primer that won't react to the styrofoam, like most spraypaints will, then paint as desired.