A larger warband is definitely better than a smaller, more elite warband. Given the wide range of statistical results a D20 roll will produce (as opposed to, say, 2D10 or 3D6, etc.), an extra point or two of Fight doesn't produce combat results which are all that reliable. A Knight will certainly have a slight advantage over a Thug, but the Thug still has a reasonable chance of defeating the Knight. Even the rulebook cautions that a Knight isn't necessarily worth five Thugs, which is true.
Also, since the goal is to get treasure and carry it off of the table, you will be severely handicapped if you come to the table with fewer models.