Does latex dry faster than PVA glue? If so I'd gladly switch to that instead.
Definately. Latex is great when applied in a thickish layer. Put the cast under a 60watt lamp to speed up the setting process. If you do 2 or 3 layers it will be easily robust enough to be applied as texture. A word of advice though, dont put it on too thick or pour it as this will increase the drying time by many hours!
PVA is supposed to be used as a glue, so has really annoying bonding properties, plus it sets wierdly sometimes, and will shrink unevenly and generally big a great pain in the back side to model with.
LARP weapons are painted with latex, and they can take quite a beating, so I think it will work for what you want. My friend made an excellent cover for a book of the hobbit he gave to his brother. He made and painted up a reflief of golum's face out of latex. It worked really well, and he just painted it with acryllics.
You should definately experiment with mixing paint into the liquid latex. Ive never tried it but it sounds like a good plan...