You can't add a pass through PCI device while the VM is on, at least you cannot in 5.0.
Just so I am clear, you were passing through the entire USB controller (as a PCI pass through device) to the VM(s), correct? You were not just passing through an already listed attached USB device?