Wear the gloves, it's the lesser of many evils. I grew up driving in Massachusetts and learned to drive with gloves, still do in California. However, you might consider varnish but you will wear it off.
I just looked at the top of my gun cabinet and there's a bottle of Neats Foot Oil up there. Maybe that or a similar oil that can be absorbed would work.
Jim