Not buying that explanation for why your cultist men get clothes and your cultist women don't. If you're going to do cheesecake at least own up to it.
How about simple economics? Supply and demand.
For a woman in that setting to get the attention of a male, she needs to be ever so slightly more attractive and available than her competition. So off to the gym and and off with the clothes...
As for the few men; they could really let themselves go. After all, the demand is so high, that even as a wheezing, overweight couch potato, after a few months, they'd not only have bags under their eyes...