spi: Fix device unregistration when unregistering the bus master