thoughts on hdri on stages

The article
https://www.splotchdog.com/index.php?option=com_content&view=article&id=98&catid=8&Itemid=110
was more or less a braindump, I'm not even sure if I should have posted it.

If you got questions / to want to participate, feel free to post

Seems expanding the good old lightsmith editor and that part of the sIBL fileformat is the way to go:

Advantages:
- Easy to read and adapt
- The planed data volume is accepteable for a txt file / the amount of new parameters is not that big (less than 10 if things go well)
- Saves some programming time
- Encourrage others providing using lightsmith edit to profile their lights
- Could be the first step to enhance the sIBL sets to a full set / scene environment documentation and preset system
for real and virtual environments.

Done some intial ui programming, wich is allways a good way to provide your ideas some structure.
I plan to get the fileformat and rudimentary profiling done at this and the next weekend with some
intial tests. If things work out like I think it will it would be even possible to profile gels

Still wrapping my head on the profiling system...one roadbump wich makes me worry is:
if you are using an hdri for profiling, how to make sure its a consistent input usuable for callibration.

Progress Report #2

Things progress more slowly than I thought, fighting some interface issues / setting up internal data structures.
But future me will be very thankfull in doing so.

Screenshot of the new measurement section of lightsmith edit should give you an idea of, what and how,
these stored data will be correlated with an hdri / dmx outout in future...things will change, for
example: I'll put the spectrum inside the other light channel definitions, had an idea to make it an extra defintion at the
end of the sIBL file, to help readeability, but its not worth the trouble for now.

So now a bit more than half a day is needed for the data management (add/remove/load/save) and
some interface stuff, and then we can start the fun part: The actual Calibration, but since
real life work needs my attention I will only do it next weekend if things work out.

Done loading / saving and adding / removing channel definitions, currently I do have new section and 6 parameters to the lightsmith file
(Also finaly fixed the pesky bug with my custom comboboxes and made it optionally possible to enter a different text value than the aviable choices..
needed for the channel names)


[Measurement] Start new channel set
MSname string name of the channel
MStype int Type of channel currently 0 for linear bw gradient, 1 for kelvin value, and 2 for hue channel (Required)
MSColor float,float the hue value dialed in and real output hue separated by comma. (RGB calculation is currently done in reader
       might be changed in future
MSSaturation float,float the saturation dialed and real output.
MSIntensity float,float the dialed in and actual intensity

MSspectrum float,float one set of values of the spectrum (wavelength) and intensity at that channel and inensity. The amount of MSspectrum
                       values is variable.

Each channel can be added multiple times with different values
(For example Different real output values and spectra for low and high kelvin values etc)
The dmx app will interpolate between these values. (at least thats planed)

Also my cheapest of the cheap spectrometer arrived
https://thunderoptics.fr/product/mini/?v=11aedd0e4327
its nowhere accurate,and you need to calibrate it yourself wich cost me a few hours of dev time. But it might be good enough to counter check other calibration methods Ive planed.

So I changed the look of the spectrum, and did some initial import of the spectrum. More formats will be added once things work.



and a comparison of what the spectrometer sees and what is displayed in lightsmith (manually scaled and moved)
so I'm sure there is not bug in that area.


Don't think I'll do something on it tomorrow, so next weekend I plan to do pseduo spectras from different inputs, and see how well I can calculate
the original emission color from them

Since late shift does have a great toll on my energy levels, I could not do much this weekend.

I created the initial prototype of the sIBLstage application wich has the basic
hdr load and display functionality, plus a initial implementation of the dmx control
(to see if things work)
(lightsmithedit progress happens once I can cross check the output correctly)




The controller I'm using is based on a FT232RL chip, wich is widely used, and
very cheap to get your hands on. The light is a cheap rgbwauv stage light from china
Wich is perfect, since it sets the programming path to the hardest way possible.
(HSV controled lights are easier to implment)

Next I focus on the lightsmith / normal light loading / editing, then I do some work on the dmx
side, and after that I focus on calibration and color rendition of a rgb hdri with more than 3 channels.
And last but not least I look into android codiing for the
lightcatcher app... wich I'm still not sure if this is even doable.

Here is one of the four major problems I have to solve..the rest is just "boring" coding:


The idea is simple: Get the spectral values of each patch, add them according to their reflection(=brightness)
, and you got a nice speudo spectrum for cheap.

Now look at the pink patch, see how bright it is under yellow light ?
Adding the spectrum values based on brightness for this patch would also add pink to the calculated spectrum
wich we don't want.

My guess is, not only do I have to weight in the brightness of the patch, but also its
hue distance relative to its original color. The farther awy the less impact it has on
the spectra.
Sure ugly hack, but tháts the whole idea from the beginning.
So its something like this wich I currently have in mind:

(1-fabs(HueOrig-HueNew))*(1-(LumeOrig-LumeNew)))*Spectral Dataofpatch

Of course I need to normalise the luminace based on one of the grey patches first
So a brighter yellow light does not create negative values.
(And ignore the grey patches completely for spectra calculation)

Also the whole sum of spectras will then be Normalised and cut at a user defined noise level
(Wich I also needed to do with the real spectral data...got some funky results when
I tried to calculate the hue with all data)

I know I'm currently shooting myself constantly in the knee with the integration of a
spectrum based color workflow (above my paygrade) but I'm listening to my gut wich
tells me that there is a payoff in elevating lightsmith towards it.

As said a hack, but not everyone has a spectrometer at hand, and I try to make this tool as
accessible as possible. (Also its fun solving problems)

I had the chance to do some work during the week, but I plan to keep programming this weekend short

I rewrote a complete new sIBL class for future use in sIBLedit, Lightsmith edit and future sIBLstage (or sIBLset)
This also questioned some of the parameters in wich a sIBLfile is currently structured.

I want to go away from the ENV/BG/REF/PLATE description and merge them all into one IMG tag, wich hosts all functionality
of plate and spheres. Nowadays BG sphere are rarely used, and PLATES need much better paramters / user input than
the current set provides. Also this opens the possebillity for new tags.
Same for Light. I want to kill the destiction between a normal Light and a Lightsmith light, and add
some additional parameters like orientation ro ir. Since dev on sIBL is currently relatively dead there is no harm in
doing some experiments with it.


As for lightsmith I added the reference image display (including zooming) +
color checker probe and did my first experiments with aquireing spectral
data from these images...wich is more or less a fail:

And this is the result after I tried to filter out some data.
As you can see, green is the only spectrum what has some similarities to the
measured spectrum. Red is overrepresented (was worser before filtering and initially present in all spectra)
and the most interesting spectrum, blue  (wich has become my reference), only has the peak
values and a bigger shift towards uv.

I think I put that problem into my subconscious part of my mind and look at it next weekend.
I expected that it won't be that easy, and now I'm more doubtfull that it will ever work,
...but since a lot of the code is usefull for other projects, not big loss if it does not work.

stupid me ....  
thats why I really _need_ to return to  programming in my free time,
This wouldn't happen a few years ago:

you need the full spectrum of the color checker at 1.0 output, normalise it and substract it from
the (normalised) spectrum derived from the image. Of course the sum output of the color checker is not a flat spectrum.
Still not there yet, but the output makes a lot more sense

feeling really stupid

not much done this weekend. Still hitting a wall on the spectral data. I suspect I need a more sophisticated way to calculate the color differences of the patches to the reference values.
Here is the current result of the blue led with an .5 value added to the whole spectrum to get the negative values of the substracted spectrum


As you can see,  the shorter and longest wavelengths are overrepresented in comparsion to the actual spectra
also the small peaks in the yellow / green are missing. And the peak of the blue is aso shifted too much into the shorter wavelengths.


I do think current collor difference is too crude
A simple substract between the base spectra and the calculate one does not cope it /
the "normalisation" idea is partly wrong.


Results so far (calculated vs measured spectras):

not much done this weekend (lateshift blues)

Still figuring out getting the "correct" spectral data.
Added a color from patch function (grey patch, also will be used for future aquiring the color from a grey card)
so now I can now correlate the patch color  with the color produced by the measured spectrum and the pseudo spectrum.
Wich is a good thing to sort out calculation errors..correlating two is easy to do it wrong with three you got some
security, that things are based on reality.

also did some interface cleanup


(patch color vs measured spectrum color)

again not much progress, but slowly getting there

Not much done since this was the second lateshift week in a row. So a lot of sleeping / idle time to regenerate:

"Fixed" the spectra calculation or at least got it matured to a point, that I can keep it alone for a few weeks
Some works on linear images import. (gamma correction in viewport and color checker comparison)

Still some slight differences between the patch color and what I get out of the spectra from the image / the measured spectrum.
But for now its not as drastic that I can't move on with the rest first, and investigate it later....at least the results are not completely of the rails.
And I think I got to the stage, where I trust a simple uncalibrated patch / camera less, than multiple patches wich only uses the color information indirectly.... But there is still some ideas lingering in my head to get better checker spectrum results, (like spectrum sustraction) but those can be implmented later.

As a sidenote I did my first "hello world" application under android.
Still not sure if it will work out as planed, but there seems to be no setup / hardware problems.


Next:
- Kelvin Channel measurement (wich should be easier and less error prone)
- Getting the Intensity from the checker and from a luxmeter: I expect second will be a bit harder (light size ?), and will keep me bussy the next weekend.

Then after I measured a few lights in my collection, and have the relative color and intensity values to each other,
I can finaly start putting things together in sIBLstage

So small steps, but at least some important progress, and soon I leave a terrain I have less experience in and get into the fun stuff,
so I might have something presentable soon (3-4 weekends).

Again not much done the last two weekrends...regneration and socialicing was more important


- started the floating point calculation of the light intensity.
Still a lot of reading on this one, the goal is to get values wich can correlate with a
usual hdri and set in correlation to other lights, and ideally, also be useable for 3d scenes itself across multiple renderers.

- In consequence  I also added a lux and luxmeter distance to the Measurment tag (MSLux and MSDist)
so even if I change the algorythm in future you don't have to reprofile lights. and it might be better suitable for import in other rpograms

-Additionally added some DMX Tags to the interface, loader and saver routines...needs further testing first
with real world lamps before I go into detail on this one.

Whats missing in lightsmith is now kuminance from patch, the display and editing of the channel values in the channel "import" dialog,
and maybe some pseudo import from the lightsmith hdri for the different channels. (But I might postpone this after the first release)

Then we can finaly start with the sIBLstage programming and the android app. (Wich will go hand in hand)

Sometimes its not the algorithms fault, but the data you feed into it.

A whole weekend lost, by blaming myself not getting the algorithm right,
while its the fault of the data I feed into.
The reflection values of patch 21 of the xrite checker does not correlate with
the differnt colors of the led spotlight. I also thought that it was the fault of my
cheap luxmeter, (green is reflected overpropotionally in comparison to white)

This is the third time the input values set me off.
I'll try to use use some corss reference of different patches to get better results

Anyway a little screenshot of whats happening:

This took me a lot longer than planed, but this is a good thing, but I only had one day, and there were some pesky little errors:



showcases the initital prototype of the new sIBLbrowser class
wich internally uses the new loader / saver sIBL classes wich will be used globally across all sibl editors (lightsmith, siBLset and sometimes in future sIBLedit if I find the time to rewrite it)
These allows me to easily prototype new sIBLfeatures (wich I'll propose soon) and remove errors across all editors at the same time. The original browser took a lot shorter to get to this stage, but at same time this was on the cost of maintainability expanidbillity and stabilty....and more important the new one has better structured code. So I'm currently paying my debts on this.

so yeah, not much new to see here. (Exept for the new custom popup wich is, as said before, finaly bugfixed, and the fact the new browser can handle sIBLs and lightsmiths at the same time.) but internally a lot has happened, wich will make all the comming bell and whistles faster and more important better to implement.

So yes, I'm a bit late considering my own timeplan, but next weekend is pure sIBLstage programming...especially since work / fre time starts to ballance out. again.

Still working on the browser, and why its fundamentally important to the
sIBL architecture, and why its worth time to take this detour from the inteded
usage of the sIBLset program:

The new sIBLclass loader, a preview of a new sIBL structure.

Since current dev is more a one man show, Its pretty easy to break conventions.

In the past sIBL had a clear destinction between background, environment and reflection and plate maps
This made totally sense, towards the old workflow, where all 3 spherical maps were needed to generate a
good looking environment for the traditional render engines at that time.

But nowadays, due to the introduction of MIS sampling the env map rarely makes sense (except for
fast lighting setup/browsing and background fog). Also some render engines do internally use floating point maps
anyway, wich makes the bg map rather a problem for memory consumtion than a solution.
And last but not least,due the fact of gpu rendering and its memory limitations, it makes more sense that a sIBL
consist only of one mid res hdri map for lighting and reflection and (several) background plate (image or movie) for the background.

So the new class introduces a new IMG class, with all the parameters the previous maps had (IMGthumb IMGu IMGv etc)
plus the new tags IMGbg, IMGenv, IMGref. Working as flags indicating wich render evaluation pass it is used for.
(0 for not used 1 for used)

Internally the new class loader already just uses this image class and assigns the mode automatically depending on the BG REF and ENV tag as a
fallback to the current sIBL version.

Also in future I'll break with the convention and add a IMGpos tag. Although this will introduce absolute position values to
the architecture for the first time (1.0 = 1m...but this needs to be taken care of by the loaders anyway),
it will ultimatively allow you to add multiple environment spheres to a single sIBL file, and will provide plate
positioning with the needed flexibillity in positioning.

And now making a turn why its important for the sIBLset:
also the Lightsmith tags will completely replace the current sIBL light defintions in future,
making it possible to use complex light rigs as presets.
Wich will allow you to load a sIBL with predefined lights, or add complex individual lighs to an hdri environment.

So we can go full circle: generating an virtual environment from reality and use this to light a
real environment.

Thats why the new browser is so important.

slow progress on browser:

- Added the search functionality
- Added the new Element Icons, added Plate element support / preview
- Reimplmented the location map  preview. The world map is now a global ressource wich is now constantly in ram. (no reload time..might become an option)
- Preview now uses the mulitply and gamma values of the sIBL
- Added the small preview functionality, Now displays also the actual image sice and bitdepth
- Comments are now displayed with propper wordwrap
- Displays now all relevant header information
- Some minor interface tweaks
- Removed error when no thumbnail is defined in a sIBL


todo:
- Recreate the pano and flat preicew option
- reimplment browser editing functions as buttons or a right click popup on each siblset (undecided)


First will slow me down a bit since there are some significant changes to the openGL
implmentation between the old 2.9 (wich sIBLedit is based on) and the new 3.x (siblset)

Slow progress, but as said, the more time I spend implmenting it right, the less problems I'll have
later adding new functions. (For example adding the previews of the maps to the main inetrface of sIBLedit
and siBLstage...so yeah I inderectly rewrite 2 programs and write a new one at the same time. fun !!!

Again this is more of a "still alive" post, got a few new tasks to learn at my dayjob and I
also did some intitial new camera implmentation on mitata wich I could only do on a weekend,
and my energy levels were quite low.

Ok, so currently I got the browser preview done (flat and spherical pano)

It now features dedicated multiply and gamma sliders (and yes upon opening it it will display it with the values of the sIBLfile)

Then I started on the the additional "addlight" functionality in sIBLstage (or sIBLset...still undecided) wich allows you to quickly
add a baisc light with some basic parameters (basically brightness and dmx channels) to a set.

The idea is, that if you know the wattage or the lux values you can quickly add some lights without the calibration hastle.
And assign a dmx channel to it, according to the new LStype tag wich has been added to the light class array of the sIBLloader

What is important is their relative values: The big headache, lux/watt to hdri values, is a big issue I still not sure how to tackle it the best way
Since this seems to be the black magic of each studio / hdri creator without any real standardisation.


We will see how well my ideas will clash with reality once I'm at that stage..at current state, doing it practically
seems the fastest option. sIBLstage will definitively get some global and local finetune controls..to allow personalised setups
depending on your equipment. And fundamental questions like if I need to gamma or not gamma correct a practical light
will be solved then....there is no point to waste mental ressources on this now.

Here is a list of lighttypes I added, more might come (For example none is missing for non dmx controlled lights)

Originally I wanted to do this automatically depending on the lightsmith definitions, but this would overcomplicate
the addlight dialog. Also I think that this info will be quite helpfull in the browser, so you can have a little
overview of your existing lights to choose from.
bicolor Mixdial is for lights where you can dial in the leds for low kelvin / high kelvin separately

So except for the app to position lights I got the fundamental corner parts together to solve this puzzle.

Next to do is the light canvas in sIBLset wich I do want to keep simple, and the I can actually bring together
the lightsmith data with the sIBLstage output.

small one for today

added the list view in sIBLstage for adding and removing lights
not much functionality or design wise...you know just the boring stuff you need to do to get to the goodies,
but I really like how the new sIBLclass speeds up programming.

energy levels are still low, although I got some time to relax, but
energy levels are still low.

searched for a nice cross platform library for bluetooth and implmented a test
case (Search for devoces and interface selection). I try to start the android app tomorrow, and if things go
well get the basic connection and data exchange done.


Also added the buttons for bluetooth and dmx connection

So I need to do the app in java, since what I've read its not
possible to access the bluetooth stuff in c++. So this might the
day I can't avoid to code in in this language.

Luckily adapting (not learning) another language today is much easier
than it was 20 years ago.
And if I can get some insights on a widespread language, the better.

So I created a simple app with a text field wich outputs the x accelometer
value for start.

I planed to use the cheap homtop for develeopment but as
I discovered it outputs the gyroscope values as the accelerator values.
Consistent, but still the wrong value :S, did cost me some valuable time
to discover this.

So I switched to my fire tablet, to cross check. and it outputs way more
reasonable data.

BTW As for the sensor readout I read a lot about people try to remove
gravity from the output by a fixed value.
But I decided to go a different way and also filter the readout and position
using this simple pseudo code snippet wich might be for some use for others:

Code:


float xPosition //global position value
float xReadout //local readout of the sensor
float prevReadout //global value of the last xReadout

if(( xReadout >0.1+prevReadout) ||(xReadout < -0.1+prevReadout) )
{
    xPosition += xReadout -prevReadout;

}
prevReadout = xReadout;


This not only filter out small movements, but gives me also a more consistent position readout.
And removes almost all calibration issues at the start of the readout.
Its not in the cm range, but it might be good enough for some UV Sphere coordinates positioning
on larger scale, but we'll see, can't promise anything at that stage.But it's actually in the ballpark
I expected it to be...wich is not a bad sign.

Since I allways want to make the bluetooth communication protocoll aviable to public and my java
coding skills are weak/nonexistend at this time. I decided that I will relase the lightcatcher app as open source.
So others can implement their own position methiod, or go through the code and make it
better. Also people can be sure nothing, wich oppose a security risk might happen on a phone.

Quick update, still alive, had some other projects and real life stuff stopping me from gettinmg into bluetooth programming. I doubt I'll make any progress the next 2 weeks. But no I don't have given up on it yet.

Sorry for that

Ok, I have to force myself to get some of my projects to the finish line in my limited time

to get again into it, I put the bluetooth development on halt until I can work 3 days in a row on it

One thing I find getting back into a project after a long time is to start with the braindead stuff...you know things
wich is tedious, mostly lot of mindless typing to archive.


more work on adding the quick add light functionality (still wip) for non profiled lights
color selection and positioning (and display) of multiple lights on the hdri image
multiply and gamma for the viewport view.

as said, this stuff is trivial , and no real software design problems to solve implementing them.
(most of them had been solved in sIBLedit / lightsmith)...but it has a therapeutic effect to have this stuff
out of the way, wich adds to the motivation to push this project further.

again not much done this and the last weekend. Most of the time invested in other projects I want to finish (mainly mirtata for a different camera system). Some basic concepting on the color engine. (How to transfer the rgb values to dmx values) but nothing to show yet

sorry about that,

Ok, something to show: had a few hours this weekend
Sorry for the ugly test chart, but it demonstrates what I've done:

Rewrote the simple DMX rgb color render with a slightly more advanced color enigne (lets call it cogine from now on)



As you notice, currently it desaturates to white using all channels, I will add a whitepoint specification
to the sIBL file and lightsmith wich allows you to define a custom white point by yourself.
This will allow you to calibrate your lights / correlate multiple lights. (*)
Also I'll add an additional desaturation method using the white led only.

Whats different to the first demo:
The color calculation is channel agnostic, meaning it will take the hue definitions of each channel in a sIBLfile, get the closest channel values neighbouring the wanted hue and calculates the output color.  (look at the video how sIBLstage uses the amber channel)

still A LOT to do, but at least one potential bigger mental roadblock has been eliminated. And its clear that
cogine will need a lot of coding and testing in future, but at least I got something working with clean expendeable code.

Also cleaned up the fixed dmx channel defintions, next is some interface work to actually assign the dmx channel.
So yeah...doesn't look like much, but this was fun, and as said seeing an idea work out, wich was only a concept,
is allways a motivational factor in programming

(*) wich opens an interesting box of ideas.

Not forgotten, but postponed for later in lightsmith and siBLstage when I got
everything together (little mindstorm for myself):

User defineable way to set and record one or more whitepoints (based on the kelvin input) of a light
inside the lightsmith file. To be used as reference for the color engine / other dmx software.
(Using all channels the user want to use eg: RGB RGBW RGBWW and WW)

The problem:
No idea how to do the correlation as hardware agnostic as possible if no spectrometer is at hand.

The McGuyver way I think of, without checking if possible, would be to use a grey card, dmx control and a hdmi videofeed
So you practically dial in the same kelvin on your camera and software,
and use dmx to adjust the channels individually so the grey is actually grey.
This might be even done semi-automatically. (just setup the kelvin and the area of interest in the feed)

The advantage would be that the lights whitepoint is actually calibrated to the used
camera..and if possible, to the temporal LUT used (if the hdmi output includes it).
(so both have to additionally be stored in the lightsmith file to be retrieved later)

The disadvantage is orientation between camera, light and greycard and the general usability / time investment,
and that this proposal makes me look as an unprofessional fool as I am.