It's common for wizards in the warhammer fluff world to be shunned by others. Practitioners of magic are just one step away from the influence of chaos, so they are regarded with suspicion and distrust. Having said that, I'm pretty sure that many of them have hired bodyguards to protect them from the mistrusting natives. You can always twist the fluff to suit your models!