I actually have a single sugar glider and he is perfectly happy living by himself! My glider is EXTREMELY friendly and social and loves to play with me and everyone around me--here's how I went about the bonding process. I also take my glider out during the day, and since I've gotten him used to MY schedule, it's not a problem at all (he's still mainly nocturnal, but he doesn't mind waking up to play with me during the day)
It looks like your bonding process got off to a bad start, so why don't you try re-starting? first off, don't I repeat DON'T even touch your sugar glider for AT LEAST a week, you need to get a fresh start--I waited two weeks before I tried to hold mine. Keep your glider in it's pouch and under your shirt for most of the day--the primary way for gliders to bond is by smell, so most of the bonding process should be passive. After the first few days, get your glider used to seeing your face and to seeing bright lights, so just randomly open it's pouch and peek in, just don't get too close because it'll perceive that as aggressive. After you notice it gets comfortable then you move on to putting applesauce on your finger and letting it lick it off, this is a great way to bond because it will inevitably lick your fingers and get some of your scent from that :)
after all of these routines you should notice that your sugar glider becomes a lot more comfortable around you, at this point, you can proceed to a larger "bonding pouch." wear a t shirt and a sweatshirt (preferably with a zipper) and put your glider between the sweater and your shirt, it's like a giant bonding pouch where it'll still be enveloped by your scent and it will gradually get more comfortable moving around you, especially when it realizes you're doing it absolutely no harm--and always reward good behavior with some applesauce licked right off your finger!
After alllll of these steps is when you can proceed to let your glider roam free--but you need to be confident that your glider is already comfortable with you when you try to let it out. You don't want to just let your glider wander out of its pouch because they are naturally curious animals, so when it inevitably jumps out and wants to explore, you're both going to be stressing out. I would recommend going into an enclosed space (I used a shower) and just placing your glider on your body--it is important that you are STANDING when you do this and here's why: gliders will always want to jump off you initially to go explore, so what you want to do is to gradually walk around a bit and spin (slowly) in a circle so your glider can't pinpoint an exact place that it can jump to.
The good thing about doing this in a shower is that if it DOES choose to jump off you, the floor's a little harder for it to fall to; it won't hurt your glider to fall to the tile, but that little sting of a rough landing will serve as a reminder to not jump off you!
This entire process takes from 2-4 weeks and while I understand how frustrating it is because you just want your glider to bond with you already you have to understand that rushing the bonding process will do no good to either of you in the end.
and yeah in general sugar gliders don't like to be held, they like to hold themselves! Mine loves to just go into the hood of my sweaters and will just sleep there all day. The times that he's stayed still in my hands is when I roll it out of it's pouch while he's still asleep...
Hope this helps!