USB: usb-serial: call port_probe and port_remove at the right times

This patch (as1253) prevents the usb-serial core from calling a
driver's port_probe and port_remove methods more than once per port.
It also removes some unnecessary try_module_get() calls and adds a
missing port_remove method call in a failure path.

Signed-off-by: Alan Stern <stern@rowland.harvard.edu>
Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>