Hello Mhut,
I have the same setup with rpi3, osmc and apa102 and more or less the same problems. I spend many hour calibrating but the results are not really good. I have some green color when yellow and red when it should be brown.
This evening I can upload my hyperion.config. But at first view color settings are similar to yours.
Best regards,