That is pretty much how I have been running Frostgrave games - I am acting as the DM/Critter Controller, and the otehr players are running bands of looters, er, adventurers. I generally have at least two 'known' bands of critters on the board wandering around when the players start, and each time the players enter a building or block there is a chance for an encounter, which can result in more critters wandering around if the player flees instead of clearing the spot. And then of course, while a player is occupied with one band of critters, other wandering bands might join in on the fun...