Reading a Unicode text file?

netninja · May 24, 2012 at 9:17 AM

I'm trying to simply load a UTF-8 encoded text file into MMF2, however it still seems to display it with ANSI encoding.

I have this simple code for a textbox:

"start of frame - load text file 'book.txt'"

and it displays — and “ as â€.

If I manually edit a string to have these characters, they display normally, but I can't find a way to load textfiles that read them properly. I noticed that these characters even work in the normal version of MMF2 anyway. I know that my font supports these characters.

How do I get a text file to load and display these characters?

Phi · May 24, 2012 at 11:52 AM

UTF-8 and ANSI are the same until you reach the bytes worth 128+.
It's likely the open command doesn't specify the encoding to be utf-8, in which case the object will need to be rewritten.

netninja · May 24, 2012 at 12:11 PM

I've tried the Edit object, File Stream and the List object and none of them work!

I'm finding this quite frustrating because I bought Unicode MMF with the intention of being able to read Unicode characters!

Are there any extensions which can read large UTF-8 encoded text files (or otherwise specify the encoding) and display them somehow in MMF?

Fernando · May 24, 2012 at 3:48 PM

Hi,

sorry to bump here.

did you try to use webview?

I will like to mention that some UTF-8 files downloaded or taken from the web using Chrome or Firefox are diferent.

Please login to see this link.

in this link you will find a demo file if you download using chrome or firefox this is not going to work but if you download using IE you will see that works correctly, if you use from internet directly it will always works .

also i add both files see (in a HexEditor i am using HxD ***free***) one start with the unicode byte mark (IE ===> EF BB BF) and for Chrome and Firefox NO (a single 0A).

Hope it helps

EDIT: i see you have developer you may use web control object.

netninja · May 24, 2012 at 4:09 PM

I opened your example .mfa but it crashed when I tried to open a file. I didn't have the WebView extension, and your program required "Webview2.mfx" which I thought may have been a duplicate?

I know that my text files are encoded as UTF-8. If I open the two sample files that you included with Notepad++, it detects the encoding of one as ANSI and the other UTF-8, as you said. Similarly, with my own text file, it detects it as UTF-8.

Is webview able to display UTF-8 encoded files properly on Windows?

Fernando · May 24, 2012 at 4:31 PM

WebView2 is the correct name this object have the same name as the one from iOS.

here is the link

Please login to see this link.

but since you have developer use web control object more powerful, when you open a document that is signed in UTF-8 using the menu you will see the encoding also you may read the encoding using the extension.

webview and web control object can show correctly the UTF-8 on windows.

netninja · May 25, 2012 at 3:43 AM

I installed the version of the extension that you specified, however when I run the example, first the .exe crashes due to "KcWebX.mfx", and then MMF itself crashes, also due to "KcWebX.mfx"

I'm using MMF2 Dev Beta 3 after build 254 and Windows 7 64-bit.

Thanks for taking the time to try and help though.

Ideally what I'm trying to do is simply display a sentence from a UTF-8 text file, word by word, in a very large font on-screen.

Yves · May 25, 2012 at 5:47 AM

Quote from netninja

I've tried the Edit object, File Stream and the List object and none of them work!

I've just tried the Edit and List objects, they can load Unicode text files...

These files must start with a special mark that says how they are encoded. Save Unicode text files with Notepad to see what I mean.

Phi · May 25, 2012 at 11:45 AM

That mark is known as BOM (Byte Order Mark), but it is not necessary to include it Yves, you can use the "wb, encoding=..." instead of just "wb" in the fopen() call.
Please login to see this link.

You can add the UTF-8 BOM mark by prepending 3 bytes with values 0xEF 0xBB 0xBF to the start of the file. In decimal terms, that's 3 bytes, 239 187 191. You can add it with the Binary object. Or a CT dude could add that functionality to an extension

Fernando · May 25, 2012 at 1:35 PM

Quote from netninja

I installed the version of the extension that you specified, however when I run the example, first the .exe crashes due to "KcWebX.mfx", and then MMF itself crashes, also due to "KcWebX.mfx"
I'm using MMF2 Dev Beta 3 after build 254 and Windows 7 64-bit.
Thanks for taking the time to try and help though.
Ideally what I'm trying to do is simply display a sentence from a UTF-8 text file, word by word, in a very large font on-screen.

perfect you may use the web object and after that use the find text features and send to clipboard after that you may use inside string or edit box.

download the object from my web page, the last version is 3.1u

netninja · May 25, 2012 at 1:46 PM

Quote from Phi

You can add the UTF-8 BOM mark by prepending 3 bytes with values 0xEF 0xBB 0xBF to the start of the file. In decimal terms, that's 3 bytes, 239 187 191. You can add it with the Binary object. Or a CT dude could add that functionality to an extension

Prepending it if it doesn't exist is a good idea! It's just annoying to have to do all these 'workarounds' for something that seems so simple.

In the messages I've exchanged with Yves it seems MMF Unicode can't read UTF-8 encoded files, it can only read "Unicode" or "UCS-2 Little Endian".

Phi · May 25, 2012 at 2:29 PM

Oops, yeah, you want UTF-16 for MMF Unicode. My bad.
Just prepend 0xFF 0xFE then (aka 255 254).
UCS-2 was deprecated by UTF-16 in July 1996, but it holds 96.9% similarity to UTF-16.

netninja · May 25, 2012 at 2:33 PM

Thanks Phi, that's a really good fix! I'll give it a go

Phi · May 25, 2012 at 3:34 PM

*1 hour later, Phi notices the finger-shape holes in his desk :o

Phi · June 9, 2012 at 10:44 PM

* 2 weeks later, Phi is still waiting for netninja to tell if it worked and is now very hungry

netninja · June 16, 2012 at 9:31 AM

I haven't had a chance to try it yet! I went on holiday for 3 weeks overseas while you've been waiting haha. I'll get back to you

Phi · June 25, 2012 at 8:05 AM

* Phi is now dead from starvation

netninja · June 25, 2012 at 8:12 AM

If it's killing you, why don't you try and make an example of it to test?

Phi · June 25, 2012 at 12:05 PM

I just assumed that you'd keep your word and get back to me...

Log in or register!

Looking for Customer Support?

Reading a Unicode text file?

Participate now!

Useful Links, Resources & Information

Fusion 2.5 Resources

Crowdfunding Focus