I use felt tip pens, something like Sharpies (i.e.permanent) though I have, on occasion, had to use washable ones. They were given a light spray of varnish & have lasted through quite a bit of wargaming.
There is a wide variety of colours, as well as point sizes.