Some of the early engines had problems where the fuel pump piston would wear through the follower and impact directly on the cam-lobe eventually causing them to need replacing.
I think VAG changed the design of the lobe when the BWA engine came out (it may even have been earlier) and thus alleviated the problem somewhat.
I changed the follower on mine 4 times over 67k miles. One time I left it, due to work and forgetting, for 23k miles and it wasn't worn excessively.
However for the sake of a £25 part and 30mins work I used to change them as a matter of course. As the saying goes prevention is better than cure and I'd rather pay £25 every 10-15k miles than pay around £700 for a new Cam.
Some people have never changed them and they have been fine so I don't think it's anything to majorly worry about but it's defo worth checking especially if you hav an AXX engine.