[blongo]

I have a problem with extended ascii characters like à ò è ì (I hope you can see these on your browser).
When I read or write messages using the Communication Centre (a PC Suite component) everything works fine.
But if I export messages in .csv format, the special characters are written on the file as couples of wrong characters.

I'm using:
. Windows XP Pro, italian, SP2
. PC Suite 7.0.7.0, italian
. Nokia 7310 Supernova, italian language, V 08.20

Any idea?

Thanks
Giovanni

[petrib]

What are the wrong characters, exactly (and what should they be)? How do you determine that the characters are wrong (maybe it is just the wrong encoding for the application you are trying to use the data in)?

[blongo]

I try to explain better with an example.

I write a message containing the latin small letter a with grave (dec 224 in extended ascii). I see this character correctly both on the phone and in the Nokia Communication Centre.
Now I select the message in Communication Centre and I export it in .csv format.
If I open the file using UltraEdit, Notepad or MS Excel I always see the latin capital letter A with tilde (dec 195 in extended ascii).

With some letters, the original character is trasformed in a couple of characters.

[scossa]

I have same problem with

. Windows XP Pro, italian, SP3

. PC Suite 7.0.9.2, italian
. Nokia 6630

any solution?

thanks

[darkhorsedre]

It sounds like a problem with either:

1. The phone and nok comms centre not using the correct ASCII code for the character, but notepad etc. does, or,
2. The phone and nok comms centre IS using the correct ASCII code.... but the others are not.

Assuming Ultraedit, notepad and MsExcel support extended ASCII, then I would think that option 2 is unlikely.

 You would think that every phone and application would encode characters the same, but it is not so.

(We in the English speaking world dont experience this as all our characters are in the standard ASCII set.)

Can you extract the RAW (exact) message with NOKI or another tool and see what ASCII code is seen?  or try to export into another format?  (it is my guess that the export feature is breaking your encoding)

Please tell us what language the notepad/Excel/Ultraedit is, as if you are using UK/US versions of these, then we need to ascertain if these apps support the ASCII set you are using on the phone..

If you could attach an example message here, we could download it and open it in various apps to see what we see.

[scossa]

I rename export Nokia 6630.csv in Nokia 6630.txt

correct message text of the export in attach is "Sì grazie ciao"

I can see it with notepad

with excel 2000 I see "Sì grazie ciao"

I'm using Italian versions of notepad and Excel

I think ASCII code of "ì" is 236

[petrib]

Note that the exported data is in all likelihood not ASCII at all, but Unicode (probably UTF-8 encoded).

[darkhorsedre]

Ok thanks scossa.

I used the following applications:

Textpad (www.textpad.com) - supports UTF8/Unicode - displayed incorrectly
Notepad (English, xp sp2) - supports UTF8/Unicode - displayed correctly
Wordpad (English, xp sp2) - supports UTF8/Unicode - displayed incorrectly
Firefox 3.06, English - supports UTF8 - displayed correctly

A point to note about UTF-8/Unicode, as stated in the help files on notepad et al:
"Notepad allows you to create and open documents in several different formats: ANSI, Unicode, big-endian Unicode, or UTF-8. These formats allow you to work with documents that use different character sets.
...
Unicode is a superset of all the major scripts of the world."

So I doubt the problem is Unicode/UTF8, as it is a superset of the other scripts and some of the failures came from clients that support these - if Nokia was exporting Unicode, I'm thinking Unicode capable clients mentioned in this entire thread should display correctly.

Textpad is a editor I use alot as it is quite flexible, but it failed to show your 'ì' correctly.
The help file of Textpad says:
"TextPad automatically detects 16-bit Unicode and UTF-8 encoded characters, when opening files. Unicode characters may be in "little endian" (Intel) or "big endian" (RISC) order, and the order is preserved when a file is saved.

Internally, these files are converted to single or double byte characters (DBCS), using the locale corresponding to the font script selected for the document class. For example, if the screen font for the Text document class is MS Mincho, with the script set to Japanese, Unicode characters in *.TXT files will be converted to the corresponding DBCS characters in code page 932.

WARNING: This means that it is only possible to edit, without data loss, files containing characters from the implied code page. Other characters will be converted into a system default character (normally "?"), if you confirm that is what you want to do."

So it is doing some conversion that may impact the displayed character, although I doubt this is the problem with this client as Italian/Latin fonts are closest to the US/UK English fonts and always supported.  Indeed, I just copied and pasted "Sì grazie ciao" into Textpad and it displayed correctly.

This must mean that Textpad CAN display "ì" when copied from the clipboard, but when opening your file it cannot, probably because the Nokia program encoded it incorrectly when it wrote the file.  Further more, when I save "Sì grazie ciao" in Textpad, it is re-opened as expected.

This is not unusual, as you can get the character "ì" (and others) to display with many methods, not just ASCII/Unicode etc.

so I think we have proven that it is the format of the characters that are saved that is the problem, not the clients that display them - Nokia must be using a format that is different from something universal like Unicode.

If I were you I would raise this to Nokia support and have them look at it - its a common problem when standards aren't used 100% and easily detectable by developers.  Whether Nokia will fix this, or fix it soon is a completely different matter

I hope this helped,

DHD

[al_lea]

When copied around, Windows may convert the characters to proper format.

[darkhorsedre]

Al, "Windows" does not interfere with files, as each application owns the file handler it uses when openeing the file. Dont forget that they also lock files when they open them, hence temporary ownership from the OS.  The applications above show that they behave differently, even "windows" apps like Notepad Vs. Wordpad.

  Clearly what you describe is not happening here as has been demonstrated.

To put it plainly, the only thing we havent verified, and still is in doubt, is the way the original .txt file (SMS) is written.

I have shown that if you create a file with these apps, they will save it using a standard encoding, and theyre help files specifically state that they ALL support the ability to display AND create these (simple) characters.

Ive worked with international O/S's for years and havent seen a problem with windows apps displaying latin fonts since windows 3.1/95!

Given that Nokia can encode characters any way they choose, and given the many simple problems their software introduces (on this forum!!!!), I wouldnt be surprised, given the evidence, if it was Nokia's applications that were not encoding the characters properly.

Although any theories that can disprove the evidence thus far is welcome - Im sure these guys need an answer!

(although I still think they would be better off reporting this to Nokia support as no-one here seem's to 'know' the answer for sure)

DHD