kaamacat,
Use an extra wide bearing on the inside. The original inner bearing is 18mm wide, I used a double roller that is 30.2mm wide. This gives it more purchase on the shaft. You will have to get the spacer machined down to make room for it. This will give you an opportunity to clean up the mating surfaces of the spacer. You will find a burr where it has been chattering around next to the worn out bearing. Measure your axles for damage, they often need replacing after bearing failure.
The wider inner bearing is part no 3208B-2RSRTNG
The outer bearing is part no 6308DDU
Johnny