with prison and barracks You are desribing a small town already or a lively garrisoned outpost.
Farmers would not live in the village/town, but outside, yes
as would the hacienda of a nobleman be
if You seriously want a "lived in" settlement and not a gaming "soundstage" some basic research into the social structure of the period and setting would be necessary.
Alternatively, watching "the magnificent seven" might also be a good approach