IB/ipath: Fix some issues with buffer cancel and sendctrl register update

There was confused use of INFINIPATH_S_PIOBUFAVAILUPD (value) and
IPATH_S_PIOBUFAVAILUPD (bit position).  Also, some callers of
ipath_cancel_sends() need kr_sendctrl restored, and some want to do it
later.

Signed-off-by: Dave Olson <dave.olson@qlogic.com>
Signed-off-by: Roland Dreier <rolandd@cisco.com>