Tokenizing strings goes too slow

[Karzon]

I have a string, let's say it looks like this (in reality the thing is a bit more complicated):
"if{2+4 == 6}"
Using several String Tokenizer Objects, I divide it several times in one loop, using respectively " ", then "{", then " " again, then "+". If the first division gives 3 elements, Tokenizer 2 divides element 0, then 1, then 2, and so on, and the same thing is done by other Tokenizers using delimiters I gave above. Despite being performed in a single loop, it goes really slowly. When the number of elements reaches 40, the 40th division with "+" is done after about 5 seconds. Is it my mistake, an MMF speed limit, or an extension issue?

Don't ask me for the purpose of this scripting engine.

[Joshtek]

You could try String Parser and see if its faster.

[James]

MFA example?

You could also consider using Lua instead of creating your own scripting language.

[Nifflas]

Joshtek: I've benchmarked String Parser 2 and Jamie's String Tokenizer, and Jam's extension is the faster one.

Karzon: This has to be one of your mistakes. My latest level editor parses a whole bunch of polygon strings with up to 50 elements each, all together in a fraction of a second. It uses 2D tokenization with Jamie's string tokenizer. You aren't doing anything dodgy, such as re-tokenizing on every loop when you iterate through it's elements?