It'd probably make him a bit more hesitant with you, maybe a bit sad. But it won't be the end of the world. My fully bonded glider I had to leave for 3 weeks and when I came back he was completely fine! But others may handle separation differently. When you come back just keep him in his pouch, or do whatever you do to bond with him a bit more than usual so he knows you are there for him. Maybe hand feed a few (healthy!) treats ;)
If there are any setbacks in the bonding just go backwards until he is comfortable again and pick up from there.