Update: I got it out last night. I ended up having to remove the vent window assembly and the window regulator (both of which were loosely installed in the door) to have enough movement from the door glass to actually get the rollers free of the track.
I thought I would go ahead and pull the door glass out so I could clean the window track attached to the glass, but that led to another problem. No matter what, I couldn't get the door glass out of the door. I read somewhere that I might need to remove the beltline "fuzzies" to get the glass out? Is that true? In the past, I've always pretty much ruined those when removing them, but these are nice enough to reuse.