I haven't had that experience before. I've had setbacks in the relationship/bonds I've had with my gliders due to extreme changes over periods of time when I moved across country. We are building that back and almost like it was before I moved, so I'm hoping to strengthen even more.
It may be though that they sense you sharing your time with the others, not sure what feelings they'd have about it, but could be unsure of you and what's going on.
Hopefully someone with more knowledge on this will come forward with some insight for you. In the meantime, just do what you did with building the trust with both sets the best you can.