Public Testing of EDT Extensions Proposed Again

[Novabrain]

I am addressing this to all the Devs, but in particular, I wanted to relate the idea to Sphax.

We have a good group of testers, but time to help out can be a luxury for some. It seems that development would accelerate if more testing, (and even suggesting) was done. It would be nice to assure that extensions get released before one of the EDT moves on or has major distractions, so time is of the essence, in my experience.

Now that Fusion Updater is in the picture, I see a better potential, (we have discussed public testing in the past) for opening up testing to registered users.

What I am thinking is that we could use the Test Tank just to make sure that an extension is relatively stable and to perhaps consider any important features. We could also check spelling, consider the menu layout, etc. We could call them Alpha on our end to distinguish them as pre-Beta release.

Once the extension has reached a certain point, it could then be made available with the updater.

My concern is that they would be Beta extensions and that means that this would have to be clearly represented. We wouldn't want users to download and install, (especially an upgraded version of a previous extension) without some significant way to clearly warn them and assure that no mistake is made about the status of the extension.

In other words, if we did agree that the process would benefit from this, then it might take either a special section, or special version, of Fusion Updater. Strong warnings and an explanation of what the user is doing is important:

1) They have to be warned not to use the extension to develop a serious project until it is final.

2) They have to be given the "use at your own risk" and agree to it, probably by checking a box for every Beta they download.

3) They have to most certainly be warned if the Beta is for a currently existing extension that will be replaced. We could get around that by either exclusively testing those in the Tank, or some other way. Maybe a renamed Beta that would be removed after testing?

There are various ways to deal with this. We should be able to remove/expire Betas should the project be suspended or canceled. That is why a naming convention might be a good idea. The final release would use the real, final name for the .mfx.

If we did agree on this idea, I would also suggest a special, generic icon for Betas that would make them stand out in the list and even a prefix: Beta"Extension Name" that would keep them in one place for testers. The final version would have the extensions actual icon.

[Sphax]

I think it's a bad idea to open the beta testing for all users.

Open it to qualified users (developers, professionals or maybe long time community members) is a good thing but open it for all users will introduce more problems and security issues than expected.

I think FusionUpdater should only be used for the final versions of the extensions and not for the beta versions because a beta version needs to be tested and needs feedbacks that FusionUpdater can't do.
I think the test-tank forum should be opened for selected users only (more than before if you want) but not for everybody.

Also, a beta version should be flagged as a beta with a MessageBox at the start of frame (which can't be removed) and protected against the compilation to prevent the use of it in a compiled product.

[AndyH]

I think public testing is a good idea once the developer is happy that they have done enough to release it to the wild. That stage could be differrent for each extension and developer, as they have to be happy about doing it in the first instance.

I don't see any problems with it as long as it's done sensibly. What springs to mind is :-

1) Extensions in test state should have the warning popup appearing when running a frame with the extension in it as mentioned above.

2) Feedback for test extensions would be in the public Extension forums - up to the developer if/how they read/respond to any posts there. It should not cause too many head-aches, no more so than what you'd get after releasing an extension. It comes down to when a developer decides they want to move to public testing of course.

3) It would be great if FusionUpdater could just have a category titled "Future releases" or something similar, with a suitable note to explain the extensions are beta's to re-enforce the pop up messages when using the extension.


All the points raised about getting extensions into a release state as quickly as possible can only be a good thing. Even though you can get a lot of noise on the forums/pm's, you'll also get a lot of valuable feedback and a wider spread of testing. Noise can be ignored don't forget

So I'd say yes to public testing, but leave it to the developer to decide when public testing starts, and still use the private testing as we do now. It would be great to have FusionUpdater helping with this too, so users don't have to go near the extension folders ever again

[James]

1) Extensions in test state should have the warning popup appearing when running a frame with the extension in it as mentioned above.

I have said countless times that this is not an acceptable protection method as it is so easy to simply NOP the call to MessageBox with a debugger.

[Jax]

If you have it in public beta, people will not really test it in the way required. I'm not sure much is gained from it.

[Sphax]

I agree with Jam.

That's why, a protection against the compilation is necessary. That could be cool too and secure, to update MMF2 to let it not compile at all the edit-time extensions. That will secure the beta extensions against the compilation (if protected) and ensure that developers compile an optimized runtime version.

[James]

Which is easy.. just add some parameters to the GetDependencies function in General.cpp (Francois recommends using __asm int 16h instead, but I guess you could quite easily NOP that too).

[AndyH]

Is there any reason why, when it should reach a public beta stage, that you are worried about people hacking their version of a beta extension to remove a pop up?

I see the pop up as a warning to not use in a finished product, as features may change making it incompatible with future versions or may contain instabilities. It's a warning to the user not to start using it until it is ready. If you were selling your extension it's a different matter.

And why do you think people will not test it in the way required? I think you'll get a lot of people who will say they will test but don't give you any feedback at all, but you may find one or two who will love it and give you lots of feedback.

The way I see it you are only delaying the public beta to a full release, and then you have to release additional release versions. At least in beta, you stand a better chance to fix more bugs before it goes to version 1.0.

I think you are missing an opportunity.

Still, the fusion updater will make it easier to get the latest versions.

[James]

If it's just a warning, why make it into an annoying nag at the start of the frame? Simple popup when inserting the extnesion into the frame editor, perhaps?

[Novabrain]

It is good to hear all the points considered.

We could leave this as an open option and let the developers and Sphax decide if it should be given a try, at least.

I like the idea of a "pilot program" where users could feel a part of trying out a new idea. If it works well and there is good feedback, participation, even intelligent suggestions for new features, it might prove the concept.

If not, we could pull that idea and try to recruit more internal testers. My post about that in the public area has received about five responses to date.