There is more to solid roller lifters than adjusting every year. You need to replace them every 3-4000 miles. The lash take up beats the needles rollers to death and then the roller wheel is no longer participating as the roller body digs into your cam. You made the correct choice.
No problem milling your heads for the seat depth you describe. I've done it multiple times.