It's good that you're being so vigilant and noticing changes in your glider. Usually a slight change in appetite is not a big deal but it's important to keep an eye on him. Taking him to the vet is the best idea. There could be a number of reasons for his behavior and a good exotic vet can help figure it all out and set your mind at ease. I have a lone glider as well so I'm often on the look out for signs of loneliness, illness, boredom, depression, etc. When she doesn't want to be held in my hand I take her whole pouch out of her cage with her inside and hold her close, speaking calmly and reassuring her. I still can't reach in her cage with my bare hand to get her but if I put a cotton glove on she has no problem with it. Fortunately she loves her bonding pouch but it's still nerve wracking to take her places so I understand your concern. A small travel cage/carrier can be used to transport your glider to the vet but I actually prefer a zippered bonding pouch because it feels more secure for both of us. I've taken her to the vet several times for various concerns but usually it's an easy fix. The most serious thing was a calcium deficiency (changing to a better diet cleared that up right away).
Wishing you all the best and please keep us posted!