No, it's not impossible.
It's a one in three chance if you just have the single token, as this unit did.
In three turns (because we forgot about removing at the end of the turn) Stephen tried three times and failed each time. Statistically unfortunate and frustrating for him (though great for me

)
It is an oddity of the rules that the more stress tokens you have the more likely you are to pass at least one of them (because you roll more dice). In the end Stephen was relieved when I shot at the unit because it added a stress token, he was able to pass and then could remove the red hand (rather than the token added by the shooting).
Remembering to remove the red token was the main problem I think - not the ability or otherwise to pass the rallying test.