mm owner: fix race between swapoff and exit