ASoC: twl6040: No need to change delay during HS ramp