Fix ENOTEMPTY check in btrfs_rmdir

The ENOTEMPTY check in btrfs_rmdir isn't reliable. It's possible that
the backward search finds . or .. at first, then some other directory
entry. In that case,  btrfs_rmdir delete . or .. improperly.  The
patch also fixes a fs_mutex unlock issue in  btrfs_rmdir.

--

Signed-off-by: Chris Mason <chris.mason@oracle.com>