wl1251: fix crash on remove due to leftover work item