| Forums | Register | Reply | Polls | Search | Statistics |
Display Calibration Light Illusion Forums / Display Calibration /  

Eliminate rounding error from your old LUT

Author ebr9999
#1 | Posted: 10 Mar 2021 18:05 
With the latest CS version Legal (16-235)/Extended (16-255) rounding errors have been fixed. This thread aims to discuss how to remove those rounding errors from your LUT.
Older CS (and LS) versions generate Legal/Extend based on conversion from full (0-255 integer) levels. These levels are normalized and recorded in BCS files as stimuli. But what is sent to the display is in the 8 bits domain, so integer rounded. Hence if you apply the inverted conversion to the converted (rounded) levels, you get decimal (rational) values that differ from the original value.
That has been fixed. So, if you create your LUT with new CS betas, no issue at all. But what about old LUTs, requiring long characterization?
In principle that can be easily fixed with editing/excel functionalities: if you remind the level scale you used during the characterization, you just have to redo the conversion, convert back (without any rounding), and change the stimulus to the corresponding value. After having fixed the BCS in such a way, you can generate a 3DLUT, free of those rounding errors.
I said in principle, as you need the right tools to do that. You can see it attached. You can also note in that tool that near-black rounding errors are quite significant.
Only for testing the tool, I have run it on a 1000 points verification I run some time ago. Have a look at the related attachment. The dE2000 is very similar for both (indeed near black was not really in that patchset).

Patch Scale Fix.zip Attached file:
Tool for fixing rounding error

dE Comparison
dE Comparison

Display Calibration Light Illusion Forums / Display Calibration /
 Eliminate rounding error from your old LUT

Your Reply Click this icon to move up to the quoted message

Watch this topic / Get updates by Email
Only registered users are allowed to post here. Please, enter your username/password details when posting a message, or register first.


Online now: Guests - 1
Members - 0
Max. ever online: 264 [6 Jun 2020 12:02]
Guests - 264 / Members - 0