Make sure we can be woken from the forced sleep that is done on errors.
Removing a card often results in -ENOMEDIUM or -EILSEQ so we previously
locked up the removal process for a second.
We could completely exit on -ENOMEDIUM, but it might be a transient
glitch so treat it like any other error.