I've never played smaller, but I've often - indeed, usually - played larger. Smaller games might be a little dull in comparison with the standard 24 points, simply because the fewer units you have, the fewer potential interactions you have. I'd be tempted to steal Strength Points from Dragon Rampant and play the odd heroic lone knight as Elite Infantry or whatever so that you have more moving parts on the table. That said, if you have - say - 16 points consisting of foot, archers sergeants, fierce foot and two units of bidowers, you'll get a decent game - whereas two units of men-at-arms and a unit of archers might be less fun.
Larger works just fine. We've played with anything from 30 to around 100 points without any problem.