minor optimization: move delay code to where delay is set and
thus where it is used vs in the main line path.
Signed-off-by: Philip Rakity <prakity@marvell.com> Acked-by: Axel Lin <axel.lin@gmail.com> Signed-off-by: Mark Brown <broonie@opensource.wolfsonmicro.com>