Think of it this way... The MORE animations you have that you want to swap colors, the lower the FPS will drop. The LESS animations you have, the FPS shouldn't drop that much.
When you call the RGB color swap to change colors of an active object, if that active object has a lot of animations, I'm guessing that MMF2's events pauses until all the color swap thing is done. I'm guessing this is why FPS drops so much.

May I ask what kind of game you are making that you need a color swap? I think I have an idea what you're doing, but I want to make sure.