Yeah I've never really had a good sense of how powerful computers are when it comes to calculations like these... especially with Clickteam being 32-bit and all.
Anyhoo, I believe I fixed the shader for now, all that's in my way is the fact that it's "too complex" according to Clickteam (there are "more active values than registers", apparently).
I've tried my best to remove un-needed variables, but to no avail.
I've tested the shader where I removed the for-loop/color-index-search and instead just put a predefined index value (pointing to a specific color), which successfully colored the entire sprite in that color + offset, and the offset's limits also worked the way they should.
sampler2D img: register(s0); // Input screen (per pixel)
sampler2D Palette: register(s1); // Palette
float Xoffset; // X offset value
float Yoffset; // Y offset value
int Mode; // 0 = Normal, 1 = Rainbow, 2 = GameBoy
float4 ps_main(in float2 In : TEXCOORD0) : COLOR0
float4 Out = tex2D(img, In);
float2 PaletteCoord = (0.0, 0.0);
//PaletteCoord.x = 0.0;
//PaletteCoord.y = 0.0;
float4 PalOut = tex2D(Palette, PaletteCoord);
for (int i = 0; i < 58; i++)
if(Out.r != PalOut.r || Out.g != PalOut.g || Out.b != PalOut.b)
//if(Out.r != tex2D(Palette, PaletteCoord).r || Out.g != tex2D(Palette, PaletteCoord).g || Out.b != tex2D(Palette, PaletteCoord).b)
//if(tex2D(img, In) != tex2D(Palette, PaletteCoord))
// ^^ I tried several different ways to if-check this, the first one makes it lag horribly with an error saying
// the shader is too complex, while the second one creates the error "float expected".
PaletteCoord.x = fmod(i, 16) / 16;
PaletteCoord.y = (floor(i / 16)) / 4;
i = 58; // I tried break; but it didn't work, so this breaks the loop instead.
// There are 57 different colors, but 8 of the 64 spots in the palette are the same pink color which is used for transparency, and I have no need for 7 extra colors.
// so 56 colors + 1 transparent color (pink).
//PaletteCoord.x += (Xoffset / 16);
if((PaletteCoord.y + (Yoffset / 4)) > 0.75)
PaletteCoord.y = 0.75;
else if((PaletteCoord.y + (Yoffset / 4)) < 0)
PaletteCoord.y = 0.0;
PaletteCoord.y += (Yoffset / 4);
Out.rgb = tex2D(Palette, PaletteCoord).rgb;
VertexShader = NULL;
PixelShader = compile ps_2_0 ps_main();
The error used to say something about there being 32 values when D3D9 only allows 31, but now that I've removed some unnecessary values, it now believes there are still more than it has been able to register.
Any idea on how this could be fixed?
Edit: I just tried switching the number of times the for-loop would run, and it appeared to fix it. With the way the for-loop is designed now (+ the if-statement thingy inside) Clickteam can only afford to run the for-loop about 5-6 times... So yeah a for-loop is most likely out of the question...
But anyway, this brings me back to another method I was trying out before I saw your reply Looki, which is to use the RGB values of the input to make a unique seed, which can be used to get a pre-defined index value in an array. The only problem with it was that it was difficult to find a way to generate a seed that would be unique for each color, since all the colors Red and Green values here are divisible by 4, and the Blue values are divisible by 8, there's a lot of similar numbers. But I found a way, by writing a program in C# that would calculate each color with this seed-generator. And then use a "check list for duplicates" website.
The thing that stopped me from doing it this way before, was that Clickteam would lag horribly every time I used the seed-calculation to find the index value (which is in an INT array with 2048 spots, that btw works fine if I skip the calculation and input a certain seed-result instead). The problem persists. The seed-calculation must be too long...