If you do swap, make sure not to change any bedding or toys, leave everything dirty with their scent so they don't stress.
Personally i would wait until the joeys are 2 weeks OOP. You don't want mom and dad distracted by the cage, feeling like they need to rescent everything. If mom is a first time mom i would be cautious.
They will be spending most of their time in the pouch with the joeys for the first 2 weeks, anyway.