I try with just 1 character each - perhaps have 2 off-table characters, guarding the horses or something, that a player can take over if their character dies early on?
The game moves pretty quickly so keeping each player turn short, will keep the whole thing moving along.