The Clickteam Fusion 2.5 optimisation & performance Hard Data thread

[Volnaiskra]

@ Volnaiskra: tried your latest mfa,
got a very very small gain (+1 fps) with "transparent" ticked (from 14/15 to 15/16 fps)

What about your MFA? I got a massive improvement in that. Also, you should adjust the MFA so that you're getting a much higher FPS. If your FPS is hovering around 15, then it'll always be very difficult to reliably measure changes. Try reducing the loops/number of objects until you get at least 100FPS (you may need to disable vsync in your GPU control panel if you go that high), and then try comparing.

For example, imagine that test A is 15% faster than test B. 15% is a very significant performance increase (for a player, it might mean a very welcome difference of 52 vs 60). If you're testing at around 100fps, then you'll see it unmistakably: 100 vs 115fps. But if you're testing at around 15fps, then that same 15% should produce around 2.25fps difference. After rounding, and with some random fluctuations and background system activity thrown in, that difference would tend to float around 1-2fps, or in some cases could even be masked by something else. The result would be that a significant 15% performance increase would to you seem like a "very very small gain".

In this case, perhaps it is a very very small gain, but we can't really know unless you give your test more FPS to work with (or use a more precise measurement than FPS, such as recording the timer after each tick).

Intel®HDGraphics5000

I thought so. In my experience, integrated graphics really struggle with Fusion. A dedicated GPU is a must for any graphically heavy Fusion game (or at least Spryke).

But I'm fine with that, because I'm targeting PC gamers, the majority of whom have dedicated GPUs (or if not, then they usually know to keep their expectations low in many games). My PC (4770k @ 3.5Ghz + GTX 980ti) is on the the high-end, though it's ageing. I also test on a gaming laptop, as well as an old PC with very modest hardware (ancient core i5, budget-level GPU from several generations back) and it runs fine (60fps except in poorly optimised areas). Spryke's minimum system requirements will be similar (though probably even lower) than most standard AAA games. That perhaps makes it a little unusual in Fusion/retro circles, but well within the expected norm otherwise.

[Volnaiskra]

I've spent about 3-4 hours today trying to recreate that antialiasing test (I never saved the MFA). I've banged my head against the wall, trying every single permutation and setting I can think of: big objects, little objects, bouncing ball objects, multiple types of objects, "save background" on/off, collisions, no collisions, etc. etc. etc. I've tried different sorts of tests, and I've tried recreating my steps of the earlier test, to the best of my memory.

I simply can't recreate the results that I posted on the 2nd post on this thread. No matter what I do, the results are the same, or almost the same, whether "antialiasing" is turned on or off. That huge performance saving I registered the first time has simply vanished. I find this very odd and very perplexing. The performance saving was definitely there. Back when I did the initial test, I would have done 10 or so tests just to be sure, and triple-checked the code to make sure (since it was such an odd result). But it's gone.

There's only one explanation I can think of, apart from that I might be losing my mind. Between then and now, I've changed my nvidia driver on this machine (for an unrelated reason: I'd loaded up Borderlands 2 again and had some performance issues in it, so updated to the newest driver). So the performance gains I was getting may have been related to the old driver (can't remember what version it was). So it seems like this one is something that has a positive impact in certain rarish cases (eg. me with my old driver, and also on happygreenfrog's system in at least one game), but isn't something that can be relied upon. I'll update the original post to reflect this.

[schrodinger]

Tried my previous mfa with the "transparent" option on/off
and tailored it to run around 80 fps
(in my case, had to cut objects on screen to 200)

results:

72/68 with transparent ticked

78/74 without transparent ticked

lower figures (68 and 74) were obtained by moving the mouse,
higher figures with mouse cursor static
(just moving the cursor gave a drop of 4-6 fps)

Also, trying that mfa again for the first runs,
without modifications, I got sudden 30 fps (instead of 20)
then after a few runs heard the CPU fan going,
and the framerate dropped to 20 for all subsequent runs.
Obviously, my pc got stressed by those subsequent request,
temperature had risen and performance drained after some runs.

I'm used to test around the average use case of 30/60 fps
sure very high frame rates can detect very sensitive changes,
and easily tell if something is better-or-worse,
but I think the "how much better" percentages would be misleading,
I'm oriented towards thinking a +15% gain on a very high frame rate
might be negligible on real-world framerate.


Btw, in my talk about "not using a top-notch" machine for developing/testing game, totally agree with you,
I was not referencing Spryke, which as you confirmed falls in the NASA-pc category
(joking - well, it somewhat does, for my standards, I probably qualify as Fusion/retro circle, and not even a gamer)
Using a crap pc is totally not a good idea when developing a HD game with juicy smooth graphics and animations,
for a (probably very wide!) target audience of gamers with a mid-to-high-end rig.


On your latest post - not the slighest doubt you had those gains with antialiasing ticked,
perhaps you might had some other process going on in background, some you weren't even aware of
(and since you are on a Windows machine, it's well known how it's particularly good in running
background processes without you knowing it - and possibly without any good reason )
Your GPU likely makes things still more difficult to discern.
I guess having a final word on these kind of optimizations would imply knowing in detail the way of
graphic processes, how CPU and GPU coexist, how their threads are handled, and the very high number
of tasks a computer does in every moment.

In case you'll run into that again, I'll be happy testing on my mid-to-crap machine

[Volnaiskra]

So you got better performance with transparent ticked on your mfa? Blimey, that's the opposite of me. I think the most rational explanation is that you're in the Northern Hemisphere, so things are naturally upside-down there. Would you agree?

It's beginning to prove frustratingly difficult to get reliable results for some of these things. I hope more people keep contributing their results to the thread so we can eventually find clearer patterns.

[schrodinger]

So you got better performance with transparent ticked on your mfa?
Blimey, that's the opposite of me.
I think the most rational explanation is that you're in the Northern Hemisphere, so things are naturally upside-down there. Would you agree?

That's most likely the cause
We even have bathtub drains going counterclockwise so it's certainly worth considering!
(..ok, that's not entirely true either..)

[Volnaiskra]

I'm used to test around the average use case of 30/60 fps
sure very high frame rates can detect very sensitive changes,
and easily tell if something is better-or-worse,
but I think the "how much better" percentages would be misleading,
I'm oriented towards thinking a +15% gain on a very high frame rate
might be negligible on real-world framerate.

Well, you were testing at around 15fps, and I suggested you test around 100fps. I would say that, of these two, 100fps is more the 'real-world framerate'. Like many modern gaming monitors, my monitor is 144hz, and so 100fps is in fact lower than my preferred fps in most games (I like a solid 120fps if I can get it - though in more demanding games, I often need to set my sights lower and aim for at least a solid 60+fps). Whereas I think that for most people, 15fps would be getting into uncomfortable 'slideshow' territory.

Also, remember that we're just testing for one thing at a time (in this case, it was the "transparent" setting), so except in extremely remarkable circumstances, we're not expecting any one thing to make a dramatic impact on a game's performance. The most we can hope for in most situations is that some optimisation technique gives us 1 or 2 FPS in a real-world setting, if that. But if we manage to find a bunch of things that give us a small but tangible increase, then they could cumulatively add up to very real savings.

[Volnaiskra]

I just made a few tests regarding animations. First, I wanted to see whether the speed of an animation made any impact on performance. According to my tests, it doesn't. I had a lot of objects (250 objects, though I also tried 2500 in another test) and moved them around in a fastloop. Whether they had a speed of 100 or 20, they always took the same time to process (100ms)




I also did a test to the difference between:
A) many objects that all have an identical looping animation
B) many objects that all have an identical looping animation, but the position of each object's animation was randomised at creation (force animation frame to random(20); restore animation frame) so that they all play at the same speed but out of sync with each other.

In each case, I had 9000 objects onscreen, being shifted around in a x10 fastloop, just to slow things down enough to expose the time difference better. As you can see, scenario A ("uniform") is 31% faster.



I also tested the speed with which Fusion can switch frames, directions and animations. I had 5000 objects onscreen at once. First: changing frame vs changing direction.

In test A, I randomly changed animation frame in a fastloop. In test B, I randomly changed animation direction in a fastloop. Changing directions seems to be about 3% quicker.



But while changing frame was more expensive, changing animation sequence was more expensive still. Randomising the frame (which, remember, was slower than randomising direction) was 11% faster than randomising the animation sequence.

Obviously, neither of the previous two results is going to be particularly relevant unless you're frequently switching frame/direction/animation in very many on-screen objects. But one can easily imagine a particle system or some sort of swarming enemy mechanic that would do just this.



Finally, I was curious whether having lots of unused directions and animation sequences inside an Active Object would somehow bog it down. Thankfully, it doesn't seem to. In this test, I had 5000 objects on screen at a time. In each case, the object was sitting on frame 1 of the "Stopped" animation. But in one of the tests, that frame was the only frame that existed in the object ("minimalist"), while in the other test ("hoarder"), there were also 31 used directions in the "stopped" animation, as well as another 12 or so animation sequences (with one frame each in them).




Summary

-The speed with which an animation plays doesn't seem to have any impact on performance.
-Multiple playing animations will cause significantly less performance strain if their animation frames are in sync
-Switching frames is significantly faster than switching animation sequences. Switching animation directions is faster still, albeit only a little bit
-Whether an object has many unused directions and/or sequences in it appears to have no impact on performance

[Volnaiskra]

I did some tests, comparing the speed of positioning an Active using "set position" (rightclick>position>select position) and "set X/Y position" (rightclick>position>set X/Y coordinate). From these results, it appears that:

-using "set X/Y coordinate" is slowest
-using "select position" to set an object's position relative to another object is faster
-using "select position" to set an object's position to an absolute X+Y position is fastest

[Volnaiskra]

-Loop optimization for moving objects-

Normally, whenever you make multiple objects that require precise movement, you loop through each one with a foreach and then start a different set of actual movement fastloops for each object. The process looks something like this:

-objects that need to move are looped through with a foreach loop
-the ID of each object is recorded on this foreach,
-movement fastloops are run for each individual object and its ID is compared, to move it pixel-by-pixel according to its velocities, also checking for collision

Let's assume that you have 5 objects moving at 5 pixels per second on the X axis. Using the above method, that is 25 Fastloops required for movement. As your object count grows, so does the number of fastloops, until eventually, the sheer amount of loops bogs your application down. My personal limit on my (old and very weak) laptop is about 170 moving objects with collision before the FPS drops low enough to become unplayable. That adds up to at worst 850 fastloops fired. I could probably manage more objects just moving around if they didn't require collision, but in action games collision is usually a necessity.


Naturally, the limitations of the traditional method with fastloops got to me, and I sat around brainstorming for an hour or so. The following method is the result of that, and astoundingly simple:

-objects that need to move are looped through with a foreach loop
-the highest absolute velocity of the objects is recorded (which would be 5, like earlier)
-movement fastloops are run only 5 times
-each object decides to move by checking whether its absolute velocity is >= to the move loop index

Using this method, you don't need to specify which object is currently moving, because Fusion does it for you: any object with an absolute velocity higher than the loop index is scoped. To remove an object from the loop, you would set its absolute velocity to 0, which means it wouldn't be scoped whenever the loop fires. Thanks to the comparison to the loopindex, you can also have objects that move at independent speeds. If an object only has a velocity of 1, it will remove itself from the scope after one loop, whereas another object would keep going through iterations until it reaches its cap.

Now lets go back to our earlier scenario, of 5 objects moving at 5 pixels per second. With the new method, that amounts to only 5 fastloops vs the old 25! Starting to look like an improvement, but let's check what my new object limit is using the new method: 2000

Now there is a significant improvement! The second method yields a total object gain of 1830! All moving simultaneously, all with collisions working! And there are still only 5 loops running each frame. The old method would be using 10,000 loops each frame, and probably crash the application on even the strongest computer!

You don't need to worry about objects having independent movements, like an enemy that uses a certain behavior or the player's standard movement. All you need to do is handle physics separately, before the movement loops happen. I added a couple different behaviors in the following example for you to play around with:

Attachment 22862

(I threw the "un-optimized" examples together quickly, so there are leftover values from the normal examples in there and the debug string isn't accurate. Ignore them. Fine detection is also OFF by default. Turning it on doesn't change much, though. My FPS dropped from 45 on average to 30-40 on average in the optimized bench.)


I don't think I need to explain the "Real world application" benefits for this method, do I? Practically everyone will benefit, especially mobile users where loops completely wipe their performance.

I finally found the time to have a look at this. It's an absolutely brilliant solution. Kudos, and thanks for sharing it!!

By the way, I've got a pretty decent computer, and your unoptimised bench frame (with 2000 objects = 10,000 loops) doesn't crash it. It does however, produce a framerate of......1 FPS. Your optimised bench frame on the other hand produces 134FPS. That's a 13,400% performance increase!!

[Volnaiskra]

I just did a test that confirmed what I was already 99% certain of: using the "flipped horizontally" shader is, surprisingly, very expensive. When Spryke moves, she generates dust clouds. Naturally, those dust clouds must move left when Spryke's moving right, and vice versa. To save on RAM and effort, I originally did only one version of the dust clouds: she would generate the vanilla dust clouds when moving right, and those same dust clouds, with the "flipped horizontally" shader applied, when she was moving left. But I noticed, even with just the naked eye, that FPS was significantly hurt whenever she was moving left.



This test confirms it: simply having objects onscreen that have the "flipped horizontally" shader applied can cause noticeable slowdown. The difference is pretty huge. It'd obviously be much cheaper (though more time-consuming, and will use a bit more RAM ) to make flipped versions of the dust clouds in an added direction of the animation editor. I don't know anything about shader language, but flipping something horizontally seems like it should be pretty simple (I mean, you're just subtracting a number from each pixel's X position, right?). We're not talking about one of those shaders that makes complex curved distortions or mimics algorithmic Photoshop blending modes. So I have to assume that most, if not all, shaders are going to also be significantly expensive, no matter how simple they appear to be.

I want to reiterate that the event shown in Test A above created visually noticeable performance drops with regular usage, and not just in the extreme test scenario (111 fastloops per frame). In my game, I was creating just a few of those dust cloud objects per second, and even then, I could visibly see an FPS drop if they were "flipped horizontally" (and checking FRAPS confirmed this).



_______________________________________________

While I'm on the topic of shaders, keep in mind that setting a shader effect is particularly expensive. Whenever possible, try to only do it once (eg. when you create an object). In the test shown below, the only difference is the order of the set effect to "Flip Horizontally action. In test A, it is only applied to the newly created object. In test B, it is applied to all instances of that object (since the newly created instance for that event only occurs afterwards). On the one hand, it makes sense that applying an effect to every object, on every loop, would be taxing. But you might also suppose that setting an effect that is set already would not make much difference. Well, it does.




This even applies to setting the effect of "none". I had assumed that setting the effect of "none" would have to be cheap, and could be thrown into certain frequently-occurring events without needing to think twice. But actually, it's quite expensive too. In this test, each object has the effect "none" applied either once (test A) or each loop (test B)



For comparison, here's the same setup, but with turning objects invisible. Repeatedly telling objects to be invisible (even when they're already invisible) is also somewhat expensive, though not quite as expensive as repeatedly telling them to have no shaders. So, I guess the moral of the story is to not do anything repeatedly that you don't have to. Though this seems to go doubly for shaders.