ipv4: Fix peer validation on cached lookup.