Yes, pirates / sailors with some head swaps is the first thing that springs to mind. Some of your bandits might be deserters, so maybe a ragged looking soldier or two thrown in would work also.
You might find some useful figures in Blue Moon's "Swashbucklers, Rogues, and Desperadoes" line [url][http://www.bluemoonmanufacturing.com/index.php?cat_id=11/url] although most would appear to be too early for Napoleonics.