usb: gadget: udc-core: fix asymmetric calls in remove_driver