xhci: don't dereference a xhci member after removing xhci