Did this last night...Could not believe how easy it was... It took all of 1 minute...literally...
Nope, You do not have to remove the drift pin to get the ball and cable out.. Only the Wheel, and wheel center pin with ""hair" clip. Once that is removed, All you have to do is goto the front of the unit where the handle is, and push up the metal "release" bar on the front bottom of the unit, right behind the handle, then it will allow you to push the handle and rod in flush all the way, thus releasing the Ball and cable.
Again, I could not believe how easy it was. I was all setup for at least a 20min-1/2 hour project, and it only took 1 minute.. What a surprise..
Now the unit is out, and I can Clean/repaint the unit. Voila!, and Hooray!
Tony K