Stay with Poly. They are more predictable load wise and a snap to change out if necessary.
I see no advantage to the original rubber units at all. Plus, you need to press them in and out.
It is the poly that has the reputation of squeaking but in my case I think the rubber is worse and absolutely deforms.
The graphite impregnated are the quietest and Joules recommendation of the offset bushings is a very trick and good solution to front end caster.