Doing port specific cleanup in the .port_remove hook is a
lot simpler and safer than doing it in the USB driver
.release or .disconnect methods. The removal of the port
from the usb-serial bus will happen before the USB driver
cleanup, so we must be careful about accessing port specific
driver data from any USB driver functions.
This problem surfaced after the commit
0998d0631 device-core: Ensure drvdata = NULL when no driver is bound
which turned the previous unsafe access into a reliable NULL
pointer dereference.