ipc, msg: fix message length check for negative values