How do binary data elements work/which extensions

I've been experimenting with the data elements tab to try to internalize some of the massive file structures that go along with my game, and I was wondering a bit about the efficiency of packing them into the executable as binary data.

I'm still not exactly sure how the data elements work, though. When I load files into them, they have the file address on my machine (ie C:/mmf2/stuff.png), but my application loads all external files based on the folders file structure, for example a file might be loaded into the active picture object as:

Code:

---

Apppath$+"GameData\Levels\"+Level_Name+"\Main.lvl"

---

So just loading the files into the data elements seems to pack them internally fine, but I'm curious how its referencing these files when it calls them? Because I can compile the .exe, delete the folder the files were in, and they'll still be loaded fine- thats good. But when I move the .exe to a different folder, they are still loaded fine which makes me thing, how are they being loaded.

Is MMF2 just searching over the internal data elements based on the filename and not path when it is loaded in? Since my game is trying to load "C:/Application Folder/Data1/Stuff.PNG", but the file was saved under the address "C:/MMF2/Data1/Stuff.PNG". Is it just searching for "Stuff.PNG" in the internal elements? Is there some specific syntax I should use to more easily reference files in the internal structure, like skipping the whole file address stuff?

And if so, does this create a problem with multiple files with the same name? I mean, if I had multiple "Main.ini" files in different folders, loaded into the INI object, and tried to load them, would it know to search the pseudo-folder of the internal structure or would it just pick the first match?

Also, what sort of efficiency issues would I have with binary data. I gather that some extensions load it directly from the .exe, and some first unpack the data to a temporary folder then load it. In the former, there should be the same disk call time of just having the file on your hard drive, but in the second there should be both a read and write time (much slower), right? And I'm guessing from my brief messing around that binary data is *not* loaded into the RAM during runtime until it is called- it wouldn't inflate my memory footprint to stuff everything in the .exe, right? Well it shouldn't, I suppose.

And also, is there any sort of compression applied to the binary data? Does it need to be decompressed when called? Because I imagine that would (marginally) slow down performance over just loading via disk, while lowering filesize.

Lastly, is there any list of which extension *can* pull data out of the compiled .exe, and relatedly which extensions can pull files but need to cache them on the disk first?

Thanks for any help!



*oh yeah, also, whats the interaction with the File Object? Specifically, if I had a condition that said "Does file exist? {directory}", would this search for internal data elements too, if they matched the string?

*and also, how would I go about playing a sound effect from an internal data element? I mean, I have all my sound effects stored externally to the game, and it precaches and plays them via the "play external file" thing, but this does not appear to work with embedded files. I know this question might seem odd since I could just store them internally via the *normal* MMF2 play sound method, but in this case I need to be able to dynamically play files based off of their names, and you can't do that with the normal play sample options.

the "playing sounds from data elements" thing has really got me stumped. I'm not sure if theres any way of actually pulling this off (besides unpacking them all, which is fruitless)

Blah, that sound bit really has me stumped. Maybe theres a way to use an extension like the old DMC object did, but then I'd have the whole host of problems of playing via an external sequencer. The other nice fix would be if there was some way to play files in MMF2 based on the name of the sample to the internal MMF2 sounds, not just external files.

Um, did you ever use the extract/release binary file actions in the Special Object? After extracting, you can use BinFileTempName$(Apppath$+"GameData\Herp.Derp") to get a temporary path to the extracted file. I think this works best or at all when the files are relative to your mfa/exe. That's all I can say, really. I've always used them like this.
I think extensions could be hardcoded to read binary files without extracting temporarily, but I don't think there is an extension which does that - Except a one I wrote myself, but for the Flash runtime. FlashFx can read pixel shaders from binary data.

well the thing is, theres no advantage to using extracted temporary files over just storing them as a file structure packaged with the application- its actually worse due to the write times. The idea being that I'd like to keep data inside of the .exe itself, protected by being packaged within it. So I could have a folder full of sounds, or I could play sounds from directly inside of the .exe, but theres no point having the sounds inside of the exe and then unloading them and reloading them into it.

But from what I gather with the documentation, each extension can be specifically coded to use binary data when loading external files. I know the Active Picture object is an example of this- you can load images from the data elements without having to unpackage them first. But most extensions don't do that.

I see - Thanks for the Active Picture hint, wasn't aware of that.
My Sound Player object which is currently in beta (and put aside for more important things), but despite that quite stable, extends MMF's sound player by ONLY using the SDK's sound library - no overhead at all. It allows you to add named sounds at runtime and play the inbuilt samples by specifying their name in an expression. The feature that you might be interested in is that you can add sounds from a memory address and size in any file format that MMF supports, ie wav, ogg, etc. - So I just would have to write a small extension to access the address of binary files, it seems. How does that sound for you?

EDIT: Hm, I had a look at the SDK, looks kind of confusing. I can't just read the files like in Flash. I'll have to talk to Yves or Francois.

Oh yeah, that would be excellent! I'm still interested in how the binaries work in general and what extensions already accommodate them
but you know, as long as you can have the same controls over internally packed sounds as you can over external ones, theres no real reason to bother with loading them from binary data. Right now the only reason I'm doing them externally is because I needed that critical "play sound based on string input" function

Im still really curious how the data elements work in the first case though. I plan on doing some experimenting once I'm done with my research paper the next week, but besides just the sound effects, I'm also puzzled how its referencing the elements. The fact it actually pulls images onto an active picture object even with a different path name makes me wonder if its only based on the filename, in which case I'll have to make sure no names are duplicated

Well, you can use my extension if you only need named samples.
It's based on the file name, yes. I'm pretty sure it's the relative path to the mfa, e.g. "Data\Test.bin".

Oh yeah, I've been checking out your sound player and it looks perfect, thanks very much! I figured I can solve most of my external data elements on an ad hoc basis by internalizing each one differently- sfx by storing in MMF2's sound player and using yours, xlua by using its internal editor (can't pull data elements directly), active pictures with internal data elements, so on and so on.

I just like to have a solid handle on how it works, but once I've got the time I'll experiment and post my findings, might be useful to other people. If its the relative path then it should be gravy, but if its indexed by filename there would be overlap problems.

But I'm curious- for loading sounds from memory with your sound player, how the heck would you know the memory address/bytesize? I'm assuming thats from those loaded into application memory (RAM), not in the internal hard disk storage (Hard Disk), right? So I guess you'd have to have already loaded it from a file or the internal player. But that seems perfect for what I needed since I can just embed them in the internal player anyway. Thanks!



...and I have a nasty feeling the only way I'm going to be able to load sound effects is to manually insert each individual one into a dummy code event, aren't I? Thats going to take some effort, hah

Yeah the memory loading function is a bit advanced.

You could load a file into the Binary object, and then pass the Binary object's address and size for example (rather pointless).

What I would have done is give you the ability to get the adress and size of the embedded file, so you don't have to extract it.

To insert internal samples, do this:

Never
* Play sample Bleh
* Play sample Derp

etc., works fine.

Yeah thats what I was planning, I'll just take to inserting every single one :) I can't see any advantage of putting them in the data elements storage anyway since its just another arbitrary internal element either way. I guess I'll play around with the other things to see how data elements work and when they dont

Figured I'd mention, XLua can pull scripts from data elements (using the 'require' statement in-script), but it is incredibly picky about where it is instructed to search. I believe you must specify the full pathname to package.path and package.cpath (C:\\path\\to\\my\\scripts\\?.lua), double-backslashes and all, or it will fail. Even though you're specifying all of this as an absolute path on your own filesystem, it's only being used as a key internally, so it doesn't matter when it's run on someone else's computer.

Of course you are just as well with the internal script editor. I don't like the design of the data elements facility in general.

Yeah that does sound a bit unwieldy. I guess I'll stick to the internal editor for the xLua object. But next up on my list to figure out are Array & INI++ objects. Former promising, latter doubtful.


But I confirmed that it does indeed handle relative file paths, not just filenames. I tried using two different active pictures pulling external files from different folders with the same filename both stored as binary data, and it worked fine.

That's because it uses the full path as the key. As to what a particular object is willing to accept is up to implementation of that object (I think ultimately they must all be converted to an absolute path one way or another for a successful lookup).

Well I'll do some testing it out with xLua, but for now I've found ways to do internalize arrays, INIs, sounds/musics (wav/ogg) and active picture tilesets/backdrops (.png). And xLua could always just have its stuff in the script editor in the worst case.


I've found that the absolute path lookup didn't actually work for the array/active picture objects when I compiled them. IE if I had it open C:/stuff/image.png, it wouldn't work when I moved the application to a different folder with the relative file deleted (but would if it stayed in that folder). But when I had it use (apppath$ + "image.png"), which is how the rest of my game is already coded, it actually did work in whatever folder or drive.

Mind for the ini object the only real solution was to extract/pull/release the embedded INIs which is some extra disc writing, but I think thats affordable for level changing (whereas every time a sound effect is used, would not)

So thanks for all the help, its shaping up to store 99% of my content within the .exe. I'm keeping the file structure around so that people can mod it and configuring the game to use both sources of input for everything, though.

This is very interesting. My projects currently use external data elements (all files needed are stored in Appath$ + "data\"). But internalizing them would be interesting. The whole internal binary data elements tab has been an utter mystery to me.

I'm very curious as to how you internalized so many different types of data. It would be nice if you could post an internalization example file some time.

I'm sure it would help a lot of people understand the mysterious Data elements tab.