Yeah that is how I approached my dungeon and cavern builds too. It's just that with limited space and even less time to actually get stuff done, what I would like to do and actually can do are a world apart.
One thing I am certain of is I want to get a Necromunda table done, and add to my dungeon. Besides that, it has to be either a pure Frostgrave table or something more generic, but then it's a toss up between village and city... So many choices... So little time and space.