It's a toughie. Does putting someone in armour make them better off? Well, training them for years to move and fight in heavy armour and then them having that armour will make them better off. Dumping a weight onto someone who lacks the training or experience of using that type of armour certainly won't. A mail shirt isn't going to fatigue someone of strong or medium build, but someone of light build might struggle.
It's one of those cans of worms that need to have a cut-off point of complexity and only you can say where you want the tinkering of rules to reflect realism to stop.
Personally, I've always felt that armour and shields are kind of underpowered in games. I like the idea that they simply reflect extra 'wounds', which would help to represent the idea that after a certain point, the shield is going to get hacked to pieces, the armour is going to get dented and broken, the fighter is going to get fatigued and is going to be less able to defend themselves (even a fully armoured knight will die if a lowly peasant sticks a dagger through the eye slot, but that's not likely to happen while he's still actively fighting). Do you then allow a measure of 'recovery', where someone away from the fighting can have their shield replaced, their armour repaired/replaced, where they can get their breath back? Or is this too cumbersome for a skirmish game?
I would also reduce movement for heavier armour and heavily penalise the 'wrong type' of armour being worn by the 'wrong class', or just not allow it at all, eg. a scout trying to wear a suit of plate.
But it's all a fudge and any extra rules will slow down the game a little bit. It's just the player's call as to where they draw the line.