powerpc/virtex/spi: Xilinx SPI driver not releasing memory

The driver was not releasing memory when it was removed or
when there was a failure during probe. This fixes it.

Signed-off-by: John Linn <john.linn@xilinx.com>
Acked-by: David Brownell <dbrownell@users.sourceforge.net>
Signed-off-by: Grant Likely <grant.likely@secretlab.ca>