I would be inclined to just buy replacement bases, they’re very inexpensive from most sources. My suspicion is that any liquid with a strong enough solvent (or even water), is going to wreck the MDF base at the same time as it lifts the texture paint.
Just an after-thought - I have popped minis off of MDF bases (I just use white tacky glue to hold them on), then used a knife to peel edge of the acrylic based texture material put and pulled it off. The bases cleaned up well enough to be reused.