x86: Add 1/2/4/8 byte optimization to 64bit __copy_{from,to}_user_inatomic
authorAndi Kleen <ak@linux.intel.com>
Fri, 16 Aug 2013 21:17:19 +0000 (14:17 -0700)
committerBen Hutchings <ben@decadent.org.uk>
Wed, 15 Jun 2016 20:28:10 +0000 (21:28 +0100)
commit733037c3c89b259235ff5217096b1c846fe7e6e2
treeb41b8c2770e7cf8267eb104bf95738ad5323d43f
parent3cfb9a410bd0489f73c69db040f727ff05df29e1
x86: Add 1/2/4/8 byte optimization to 64bit __copy_{from,to}_user_inatomic

commit ff47ab4ff3cddfa7bc1b25b990e24abe2ae474ff upstream.

The 64bit __copy_{from,to}_user_inatomic always called
copy_from_user_generic, but skipped the special optimizations for 1/2/4/8
byte accesses.

This especially hurts the futex call, which accesses the 4 byte futex
user value with a complicated fast string operation in a function call,
instead of a single movl.

Use __copy_{from,to}_user for _inatomic instead to get the same
optimizations. The only problem was the might_fault() in those functions.
So move that to new wrapper and call __copy_{f,t}_user_nocheck()
from *_inatomic directly.

32bit already did this correctly by duplicating the code.

Signed-off-by: Andi Kleen <ak@linux.intel.com>
Link: http://lkml.kernel.org/r/1376687844-19857-2-git-send-email-andi@firstfloor.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
Signed-off-by: Ben Hutchings <ben@decadent.org.uk>
Cc: Jaccon Bastiaansen <jaccon.bastiaansen@gmail.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: mingo@redhat.com
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: h.zuidam@computer.org
arch/x86/include/asm/uaccess_64.h