FROMLIST: fix minor error in YUV2RGB matrix used in shader
The matrix used for YCbCr to RGB is listed in Wiki https://en.wikipedia.org/wiki/YCbCr;
There is minor error in the matrix constant: 0.0625=16/256 should be 16.0/255,
and 0.5=128.0/256 should be 128.0/255.
Note that conversion from a 0-255 byte number to 0-1.0 float is to divide by 255
instead of 256. That's we get 255=1.0f.
By the constant change we can see the CSC result is bit aligned with
Wiki conversion result and FFMPeg result.
Otherwise in some situation, there will be one bit difference
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=100854
Signed-off-by: Kristian H. Kristensen <hoegsberg@google.com>
(am from https://patchwork.freedesktop.org/patch/153580/)
BUG=b/35083011
TEST=android.media.cts.EncodeVirtualDisplayWithCompositionTest#testRenderingMaxResolutionLocally
Change-Id: I85c3873bc6caa8474daa051e6358de285ab14e81
Reviewed-on: https://chromium-review.googlesource.com/491967
Commit-Ready: Chad Versace <chadversary@chromium.org>
Tested-by: Kristian H. Kristensen <hoegsberg@chromium.org>
Reviewed-by: Chad Versace <chadversary@chromium.org>
Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org>
1 file changed