ALSA: seq: Fix double port list deletion