TD-26657: Change the current implementation of UTF8 source support in TD IDE

[Dave Rabelink]

Ticket CS0065577
TD-26657: Request for change UTF8 source file handling in TD IDE


Current implementation:
The TD IDE has in the preferences (general tab) these options:

Use UTF-8 encoding: For Save
Use UTF-8 encoding: For Read

TDPreferences_UTF.png

These settings are global, not project specific and managed in the registry.
When saving a TD source, it will take the preference.
This applies for new files and for opened existing files.


Problem with current implementation:
It is not clear to the developer now which encoding the opened file is using.
Before saving, to be sure that a source is saved as UTF16 or UTF8, the preferences screen has to be opened.
This can easily be forgotten.

The problem with this is that specially GIT handles UTF8 and UTF16 files differently.
UTF16 is by default seen by GIT as binary and does no auto-merge. But UTF16 has difficulty in diffs.
So a repository should be configured to manage UTF8 or UTF16 encoding correctly.

If for instance a repository having all UTF16 TD files is used by a TD developer and has the TD preferences set to saving as UTF8,
this developer will unknowingly open UTF16 files and just by saving it convert the file without notice to UTF8 and put that into the repository.
Later on, GIT will auto-merge this file which will result in corrupt TD files as TD files have hex blocks which cannot be merged.

Also diffs will show up in version history because of this encoding change.

Now with the current feature to save UTF8 it is very dangerous to have such a hidden option which affects saving of sources.
I already made mistakes as I was playing around with UTF8 saving, but forgot to set the option back to save to UTF16.
Repositories are afterwards set to the unwanted encoding by accident.

Also when different repo's are used where the one you want UTF8 and the other UTF16, switching between repo's need changing the TD IDE preference every time.

Also that it is not visible to the developer immediately which encoding is used except by opening the preferences makes mistakes very common.

The current implementation does not take the overall approach in other software and even with other TD settings.

Request to change the way TD handles UTF16 vs UTF8 which offers better control to the developer, more visual cue's and will not suddenly change encoding automatically.


Requested change:

The preferences setting should be only for new files. Just like the setting which extension to use and which format the source is saved in is only for new files.
Not for opened existing files.

When an existing TD source file in UTF16, opening the file in TD will keep the UTF16 encoding when saving back to disk.
When an existing TD source file in UTF8, opening the file in TD will keep the UTF8 encoding when saving back to disk.

When creating a new file in TD (by using the new option in file menu OR when opening TD without a file), when saving the file, it will take the preference setting.
So new files will automatically be saved as in preference UTF8 or UTF16.

The developer should be able to choose which encoding will be used in the Save-As dialog.
Just like in notepad, when doing save-as, you should have the option to change the encoding for that file.

SaveAsEncoding.png

So when a developer opened an UTF16 file and the TD preference is set to save to UTF16 it can only change the encoding to UTF8 in the Save-As dialog and change the encoding in the save options. Visa versa for UTF8.

The user can use this encoding selection feature in save-as dialog to quickly check the encoding used instead of first going to preferences.
The save-as dialog overrules any preference setting and only applies for that specific file.

Question: why do we need a Read UTF8 option at all. It seems a bit unneeded. Why would you block TD opening a UTF8 file when in actually can open it?
Why have an explicit preference for that?
Better to remove the option "Read for UTF8" as opening a file should always be possible if TD is able to. And with the described requested changes above it will keep the encoding as-is when saving and can only be changed in the save-as dialog when needed.

Maybe even show the used file encoding in the lower right corner of the TD IDE status-bar for quick reference.
As example in notepad:

ShowEncodingStatusbar.png

[Dave Rabelink]

This request for change was already implemented for the next release, TD 7.5:

TD-26235: Request to add an option to change UTF-8 setting at file level instead of IDE

UTF-8 Encoding options are removed from Preference because this options conflict with Save As command behavior:

TD_Preferences.png

Save As command is enabled to specify text encoding (UTF-16 or UTF-8) for individual application file:

SaveAs.png

SaveOutlineAsUtf8Text function is added to cdkApplication class in CDK.apl.
It saves application as text format with UTF-8 encoding. SaveOutlineAsText function still uses UTF-16 encoding.

So this to be expected in TD 7.5.

[Dave Rabelink]

This is now implemented in TD 7.5.0