at76c50x-usb: fix oops on disconnect