Beta 12: Benchmark results and framerate problems!

[netninja]

I made a benchmark app in which I tested iPhone games made with the MMF iPhone Runtime on an iPhone 4 and an iPhone 3GS. Screenshots taken on my iPhone 4, all games used a resolution of 480x320. These were the results:

Benchmark 2.mfa


32x32 standard active objects with bouncy ball movement. Bounce on leaving the screen.

iPhone 4:
* 230 objects, constant 50fps
* 250 objects, signs of struggle but basically constant 50 fps
* 280 objects, low 40s in fps
* 300 objects, high 30s in fps

iPhone 3gs:
* 150 objects, constant 50fps
* 160 objects, spiking but still constant
* 180 objects, heavier spiking
* 190 objects, 40s in fps

Benchmark 3.mfa
Same setup as Benchmark 2.mfa except that the objects didn't have bouncyball movement, they were moved pixel by pixel (x position = xposition + speed) and wrapped when they reached the edge of the screen. This resulted in a performance hit, roughly halving the amount of objects in use as compared to the previous test.

iPhone4: 160 objects, consistently bad (low 30 fps)
iPhone3GS: 80 objects, consistently bad (low 30 fps)

Benchmark 6.mfa


64x64 pink boxes created, no alpha channel. Bouncy ball movement. Same as first example but with gradient background. (Yay for awful colours )

iPhone 4:
After 35 objects there is a serious performance hit with constantly lower than 35 fps.

iPhone 3GS:
untested (I assume it would be worse).

Summary
From this data I am really quite disappointed with the performance of 64x64 images (or larger) in the iPhone runtime. The iPhone 4 can hardly handle any at all and with alpha channels combined with more than a few lines of code, the engine becomes very restricting. And of course an iPhone 3GS or earlier has no chance to compete, given that it performed roughly half as well in the tests above.

So could you please look into some way of improving performance with these larger objects? Benchmark 2 has a promising amount of active objects on screen without slowdown, but most games require larger graphics! :eek:

It is also good to note, for the user, that built in movements are more efficient than custom coded ones (as seen in benchmark 3).

To the testers: What is your experience with slowdown and performance issues? What do you think of these results?

[Francois]

The gradient background is taking a lot of processor time : it has (as all backdrop objects in HWA) to be redrawn at every loop. I'm sure the benchmark would be better with a normal background.
Come on, when have you (or anyone) used a gradient backdrop object in a game?

[Anders]

The quick backdrop should not be the cause.
From what I can see it is the constant updating of the string object that is the cause. Modifying strings seem to take 97% of the CPU time (on my 3GS iPod Touch).

I will see what can be improved there. I haven't experienced such a slowdown on other strings before.

Francois: I often use it for the sky in my small games (when I get around to making some hehe)

[Anders]

When I changed the events in benchmark 6 to only update the string object every second, the FPS only started to drop slightly once reaching 160 pink objects.

[netninja]

Oh wow. I have a string object on my current game to let me know how many objects are on screen, and the framerate. I never considered that it might be responsible for the low FPS! I'll test this on my actual game and see how it runs.

And yes sorry for the inconsistency with the gradient background - I put that in when I was running some tests with alpha channels. I only put it in so that I could see the colours underneath, but I have similar slowdown on an actual game I am working on - it slows down with 60 objects or less moving on an iPhone 4. It's the one I've sent you Francois.

[netninja]

When I changed the events in benchmark 6 to only update the string object every second, the FPS only started to drop slightly once reaching 160 pink objects.

That is great news! I'll test my latest game on my iPhone right now.

[Anders]

The quick backdrop gradient used 57% CPU on the emulator versus only 0.2% CPU time on the device itself. The emulator is a really bad indicator on how a game performs on an actual device as it uses a software renderer and fakes pixel shaders.

I will see what I can do to optimize the text drawing on the device itself (as it is blazingly fast on the emulator for some reason)

[BHGames]

It's the drawing of text and or the creation of strings themselves?

[netninja]

The emulator has always run my games quite slow. I'll do a real hardware test now on an iPhone 4.

[Anders]

It is the "set string" action that takes so much time. Drawing them is very fast.
Note: The set string action is only slow if the string is different from the previous string displayed. If you always set the string to the same it should not be fast.

In the case of the FPS counter it was a funny example. If the FPS varies just slightly the string changes. The change of the string is expensive CPU wise and further changes the FPS, which again changes the string -> changes the FPS -> changes the string. The FPS ends up going completely nuts because of this lol

Creating the string takes around the same amount of time as setting the string to something new.