drm/radeon: rework finding display PLL numbers v2

This completely reworks how the PLL parameters are generated and
should result in better matching dot clock frequencies.

Probably needs quite a bit of testing.

bugs: https://bugs.freedesktop.org/show_bug.cgi?id=76564

v2: more cleanup and comments.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>