I think the game works great with smaller warbands. We played Heritor, Warden, 1 specialist, 2 standard crew. In generally, I'd probably allow 2 specialists, as they are bit more fun. The only real issues is in how you handle treasure. We basically counted a treasure as 'secure' as soon as it was moved to a designated area - that way the figure didn't have to exit the table.