I have a neutered male with two females. The only time I have a problem is when the female is in heat. He still tries to mate, but he can't make babies. I just separate him for the night and then everything is fine.
I don't know how easy it will be for you to introduce. I recommend doing introductions one on one after scent swapping.