If you need to knock down the existing rough surface to smooth it before the application of filler, my best suggestion would be to get some aggressive sanding paper (60-80 grit) and wrap it around a wooden block to grind off the rough spots. depending on how rough and how durable the surface, wear a glove - I once tore up my hand trying this trick with A) bare hands, B) no sanding block, C) over-enthusiastic scrubbing.