Won't Fix TD-17726: Sybase Error Unicode conversion failed

Post found bugs and possible workarounds.
Brian Howard

TD-17726: Sybase Error Unicode conversion failed

Post by Brian Howard » 01 May 2012, 23:23

Hi All,

We get error 10446 - "Unicode conversion failed" ( "Failed to convert input( or output) buffer to (or from) Unicode format." )

I have attached a repro case that hopefully allows you to reproduce this. To run this test case create the table per bh_cc.dml then populate it with data per bh_cc.dml.
In case it is relevant the following are the settings from the TD SQL.INI file for

Code: Select all

[sybgtwy]
longbuffer=32650
locktimeout=0
sybautocommit=on
yieldonservercall=0
sybworkstationname=workstation_name
sybapplicationname=CENTURA Sybase App
checkexists=off
fetchrow=20
sybmaxmessages=100
enablemultipleconnections=on
closecursorateof=on
This error occurs with Sybase client 12.5 and 15.0 client. We are using Sybase on a Sun box.

This error occurs when char_convert is on, this is the default for our clients servers and they are not willing to change this. If we change the connection properties to char_convert off, then we have other issues with non printable characters such as 0xFE

Could you please look at this and advise if this is a defect.

thanks

Brian
You do not have the required permissions to view the files attached to this post.

Jeff Luther

Re: TD-17726: Sybase Error Unicode conversion failed

Post by Jeff Luther » 02 May 2012, 16:37

Brian
The correct way to handle this issue. Unfortunately, I also read what you wrote...
This error occurs when char_convert is on, this is the default for our clients servers and they are not willing to change this.
FYI, a couple links by way of background on the subject of "sybase char_convert":
http://manuals.sybase.com/onlinebooks/g ... 54;pt=2689
http://manuals.sybase.com/onlinebooks/g ... View/29082
http://infocenter.sybase.com/help/index ... ag1474.htm

I'll have to ask internally about this and will look at your test case.

Jeff Luther

Re: TD-17726: Sybase Error Unicode conversion failed

Post by Jeff Luther » 02 May 2012, 17:36

Back after running your test case...
Note I've attached a mod. to your test. Take a look at this one, since my changes are essentially to make the test 'self-contained' without having to run external DML or DDL files. This helps dev. and QA automate their testing. You'll see I added to form create the DDL and enough DML rows to get some data in the DB table, plus on destroy drop the table. If you use this model for test cases for us, it really helps. Thanks.

I also saw this file was missing: C:\!Dev\Phoenix Dev\Centura60\txt6.0\dasybsal.apl
and I included TD's sybsal32.apl -- If your DA version has the TD APL as a File Include, no problem. But if that DA apl has included sybsal32.apl in it, I'd caution you that this could be a problem because future TD releases might make changes to sybsal32.apl and if you are using a hard-coded version in DA then you'd miss any changes. (I've seen this a number of times with users, especially for the TD APLs, where declarations changed from old 4.2 with ANSI 'A' funct. decls. to newer Unicode 'W' decls. and then wondered why their VT calls were failing.)

So, to your test case. I do not get any error when I run that with our Sybase v15, either with char_convert ON of OFF. In SAM_Create you'll see I turn this ON before the DDL and DML statements, then keep it ON when I click the Go button. No error.

Couple thoughts and questions:
** What version of the Sybase DB server is your customer on?

** Is their Sybase DB Unicode-enabled?

** I talked with our TD/router developer who believes as I do: It seems that they do not have a Unicode DB which is why this conversion is failing. Our Sybase v15 test DB at the office is. Since you still use the 'Centura' word even for TD v6 it seems your app. is being ported from earlier TD prior to 5.1, when TD used ANSI/single-byte strings.

** The dev. also mentioned that this char_convert issue might be entered in as an enhancement request, but this is not a bug. Sounds harsh perhaps, but TD since 5.x and later is now in Unicode, fundamentally there are 2 basic options:
1 - configure/set up the DB so it can handle Unicode strings. Or add code to Set char_convert off, for example. You have a way, based on that previous forum thread, except your users don't want to change their DB for char_convert.

2 -- (more difficult) might be that you would have to convert your Unicode strings to ANSI (using SalStrToMultiByte) before the INSERT/UPDATE calls, and then on fetch back the ANSI string must be converted back to Unicode (using SalStrToWideChar) if you want to use and display the string in TD. Like in the table window of your test case.

Last, Sybase NVARCHAR (your test case uses VARCHAR) -- I recommended this in that other forum thread, info. for this type here:
http://dcx.sybase.com/1200/en/dbreferen ... rying.html

NVARCHAR is the correct DB column type for Unicode data, as the link says: "The NVARCHAR data type stores Unicode character data, up to 32767 characters."

I suggest you change the test to do a DLL call with nvarchar(30) and see if that works for you.
You do not have the required permissions to view the files attached to this post.

Brian Howard

Re: TD-17726: Sybase Error Unicode conversion failed

Post by Brian Howard » 02 May 2012, 23:49

Hi Jeff,

The dasybsal.apl simply included sybsal.apl. It also had a Connect function tha handles the char_convert setting. Sorry my bad for not pulling this out of the example.

You asked what version of Sybase
Select @@version returns
"Adaptive Server Enterprise/15.0.3/EBF 18467 ESD#4 ONE-OFF/P/Sun_svr4/OS 5.8/ase1503/2768/64-bit/FBO/Thu Nov 18 20:03:17 2010"

You asked is Unicode enabled
These are the settings I can find using sp_configure that include the word "unicode"
95 enable unicode conversions 1 0 1 1 switch dynamic
96 enable unicode normalization 1 0 1 1 switch dynamic
294 Unicode 0 0 0 0 not applicable [NULL]

I understand what you mean about making test cases self contained i.e. include ddl / dml, but could you please insert ALL rows not just enough to get a result. This makes a difference.

I use the word 'Centura' because I have been using 'Gupta' since the 16 bit version about 4.1 i.e. about 1995. And although we are porting from TD 3.1 the repro case contains only a pasted table definition from ported code.

This is where I might sound a bit harsh. Our clients servers are not just used by TD applications, there are hundreds of other non TD applications that use these servers and any server level changes will require full regression testing of ALL applications, including other TD 3.1 applications during the migration period. This would require months of testing. Now I freely admit I have not read every word put out by Unify regarding migrating from non unicode versions of TD to a unicode version of TD, but I do not recall reading anywhere where Unify advises that the back end database configurations have to be changed to also be unicode. i.e. should this not all be in the router?

I'm not sure how we can use SalStrToWideChar in a SalTblPopulate :0)

I recreated the test table with nvarchar and the same issue occurred.

We have worked around this issue by using the char_convert on our connections, but when I came across the issue of the 0xFE not inserting correctly, I though it prudent to spend some more time trying to reproduce this issue.

Anyway, if you could please advise which db settings you believe we need to make, with some assurance that at least these changes will not affect non unicode versions of TD, then I can pass these on to the relevant authorities, who I expect, will laugh at me and say 'Not likely'.

regards

Brian

Jeff Luther

Re: TD-17726: Sybase Error Unicode conversion failed

Post by Jeff Luther » 05 May 2012, 01:20

Without sound glib back, Brian...
if you could please advise which db settings you believe we need to make
I thought we went through this and found a DB side and TD code side configuration/coding where you could get your 0xFE string in and read back OK with no errors.
...with some assurance that at least these changes will not affect non unicode versions of TD.. .hundreds of other non TD applications that use these servers...including other TD 3.1 applications during the migration period
That's a BIG 'assurance' and I cannot give that to you because I don't know. Users have been dealing with TD and Unicode since 5.1 originally came out, so perhaps a search of the 5.2 et al sections can help you glean some info. as to the issues they faced and overcame.

OTOH, you are (apparently) trying to upgrade the front-end TD v6 app and keep your (old) Sybase DB and its configuration. I'll pose your issue internally and see if anyone has any suggestions.

As for "every word" take a look at the PDF page for older white papers
esp. top one on 5.1 and Unicode. Section toward the bottom has some DB tidbits -- that were relevant as of 5.1, note.

Brian -- I strongly urge you to get a test DB, couple of old 3.1 apps. working and see how a 6.0 test will perform for you. I understand your frustration but an assurance for some special configuration and mandate from me for your clients' special setup just isn't something that I can provide you. That's why special-case testing on your end is what I recommend.

Brian Howard

Re: TD-17726: Sybase Error Unicode conversion failed

Post by Brian Howard » 06 May 2012, 23:08

Hi Jeff,

With reference to
\\
I thought we went through this and found a DB side and TD code side configuration/coding where you could get your 0xFE string in and read back OK with no errors.
//
Just because I can work around this issue does not mean it is not an issue. I have noticed over the past year or two Unify have been more an more taking the attitude, that if there is a work around, then there is no issue. The problem with this, is that we can not pass this on to our customers.

The only mention of Sybase is
"UTF-8 encoding must also be set for Sybase databases by editing the file “local.dat” under \Sybase\local sub-directory when dealing with Unicode strings.
o This is not necessary if using TD5.1 latest Service Pack"

As this change is not required since TD5.1 latest Service Pack, I would expect there are no required changes for 6.0, and this setting is in the local.dat anyway.

In my last post I asked you, "could you please insert ALL rows not just enough to get a result. This makes a difference." Did you do this? I understand as a test case you do not want to have a seperate script, but please at least try what I asked. If you do reproduce the issue, I will update the test case to include all of the inserts for you.

regards

Brian

Jeff Luther

Re: TD-17726: Sybase Error Unicode conversion failed

Post by Jeff Luther » 07 May 2012, 21:37

I/we need to back up a bit. I have updated the test case I mod'd: cc_test-JLmod-3-AttachToForum.zip
with all the row from your DML file.
...insert ALL rows not just enough to get a result. This makes a difference." Did you do this? I understand as a test case you do not want to have a seperate script, but please at least try what I asked. If you do reproduce the issue,
Done. Now it is your turn to run this mod-3 test case and describe for me exactly what this issue is you find. My testing shows 75 rows fetched ON or OFF.
You do not have the required permissions to view the files attached to this post.

Brian Howard

Re: TD-17726: Sybase Error Unicode conversion failed

Post by Brian Howard » 08 May 2012, 02:19

Hi Jeff,

I have run the adapted code and still get the same error.
UnicodeConversionError.jpg
I don't know where we can go from here as you are suggesting we have to change our server, and I believe the router should handle this, so our only way to use UNICODE TD is to code around the short comings of the TD product.

regards

Brian
You do not have the required permissions to view the files attached to this post.

Jeff Luther

Re: TD-17726: Sybase Error Unicode conversion failed

Post by Jeff Luther » 09 May 2012, 00:12

"the short comings of the TD product' -- LOL, well that's one place to put the blame :?

How about we take a breathe and start with that error msg. box:
* Which column is the one causing the Unicode conversion failure in that error msg? Probably DESC2, since it is a varchar column in the DLL statement of my modified test case I attached earlier:

Code: Select all

"CREATE TABLE bh_cc
(
    RIM_CLASS_CODE    smallint    NOT NULL,
    DESCRIPTION       nvarchar(30) NOT NULL,
    GROUP_CHARGE_CODE smallint    NOT NULL,
    DESC2             varchar(30) NOT NULL
)"
How about doing some testing/debugging and changing the test case DB table column DESC2 to NVARCHAR as well and testing? NVARCHAR is the correct column type for Unicode:
http://dcx.sybase.com/1200/en/dbreferen ... rying.html
I believe the router should handle this
Out of curiosity, how do you figure the router should be handlling this? Auto-convert the Unicode string to ANSI before INSERT/UPDATE? If so, what about the 0xFE char in your prior test? That's a 2-byte/Unicode character -- what should the router do about that? It can't convert it, otherwise that 2-byte char. is lost forever.

Brian Howard

Re: TD-17726: Sybase Error Unicode conversion failed

Post by Brian Howard » 09 May 2012, 01:08

Hi Jeff,

You state
How about we take a breathe and start with that error msg. box:
* Which column is the one causing the Unicode conversion failure in that error msg? Probably DESC2, since it is a varchar column in the DLL statement of my modified test case I attached earlier:


I possibly have not been clear enough. This test case is the first time I have been able to create a scenario that will always fail. If there were less rows it might not fail. If there were less columns it might not fail, but under this scenario it always fails.

You asked
How about doing some testing/debugging and changing the test case DB table column DESC2 to NVARCHAR as well and testing? NVARCHAR is the correct column type for Unicode:
As stated in my message on May 03 "I recreated the test table with nvarchar and the same issue occurred." I have now also changed the test case as supplied by you so both again are nvarchar and the problem still exists.

Finally you asked
Out of curiosity, how do you figure the router should be handlling this? Auto-convert the Unicode string to ANSI before INSERT/UPDATE? If so, what about the 0xFE char in your prior test? That's a 2-byte/Unicode character -- what should the router do about that? It can't convert it, otherwise that 2-byte char. is lost forever.
That's funny Jeff, are you now asking me to write your router for you. I am your customer, TD 3.1 works perfectly well with the issues I have raised. Unify changed the client application to be UNICODE. We should not have to change our customer's servers so that the new TD client application does not error. It is my belief that 0FE is only one byte long and as such would be 0xFE00 in UNICODE and 0xFE in ANSI, so I fail to see why this cannot be translated, and yes, as you suggested, I believe the router should translate to ANSI if the backend server is ANSI.

Would it be possible for you to change your server settings temporarily to an ANSI configuration, to see if you can then replicate this issue?


cheers

Jeff Luther

Re: TD-17726: Sybase Error Unicode conversion failed

Post by Jeff Luther » 14 May 2012, 18:41

One thing for you to check, before proceeding, Brian: Recall I'd mentioned I had to comment out a "dasybsal.apl" include and add sybsal32.apl, like this:

! File Include: C:\!Dev\Phoenix Dev\Centura60\txt6.0\dasybsal.apl
File Include: sybsal32.apl

Well, t/s manager opined whether you might not have the correct v6.0 sybsal32.apl included and are still using an old version? And that perhaps this is an issue?

Do what I did above -- comment out your DA file and include sybsal32.apl, being certain it's the correct v6.0 one, try again and let me know. There have been many cases with customers who Merge Libs. back in an older TD and now when upgrading to v6, for example, they're still using the old APL without knowing it. And when Unicode in TD was released the old APLs often did not work correctly.

Brian Howard

Re: TD-17726: Sybase Error Unicode conversion failed

Post by Brian Howard » 14 May 2012, 23:01

Hi Jeff,

We are deninitely using the correct sybsal.apl and sybsal.dll. I previously advised that we also get this error when using the updated test case provided by you. But here is a screen shot showing the wide string parameters to a syb function call.
Sybsal.jpg
Have you tried this on s sybase instance that is not set up as unicode?

cheers
You do not have the required permissions to view the files attached to this post.

Jeff Luther

Re: TD-17726: Sybase Error Unicode conversion failed

Post by Jeff Luther » 16 May 2012, 18:08

Thanks for verifying this. Development needs to look at this and with that in mind I have added new TD defect TD-17726 for this issue, Brian.

"This issue" is actually more complex than it sounds and this defect, while titled:
"SYBASE non-Unicode DB: User cannot convert the database and with Unicode string data causes Sybase #10446 SQL error: "Unicode conversion failed" "

actually is 2 related parts, based on your previous forum thread about 0xFE char and this thread and the SQL #10446 error you are reporting. For the moment, both issues are the same TD defect. Development may think differently and/or perhaps determine that the 0xFE issue (with char_convert OFF only 3rd row inserts the 0xFE char. correctly) is not a bug. TBD at the moment.

Here's my description in the defect for the 2 parts:
#1 - This is the issue where 75 rows are inserted and fetched and where user is getting the #10446 error because of non-Unicode SYBASE when the 's' column data are inserted.

#2 - This is the issue [you] first reported problem with non-Unicode Sybase. This is a related issue to #1 where the user cannot change his customer's char_convert setting to ON and with it OFF there are several modes of binding etc. what fail to insert char. 0xFE.

Jeff Luther

Re: TD-17726: Sybase Error Unicode conversion failed

Post by Jeff Luther » 26 Jul 2012, 20:12

BOTTOM LIN: Our TD/router developer looked this and cannot reproduce the issue you reported. As he commented:
I tried creating a Database in Sybase, and wasn't prompted for any unicode / non-unicode or locale choices, so I'm not sure what he might be running that would be different from what we have. If he's using a default Sybase install of 12.5 or 15, our environments shouldn't be that different with respect to Unicode (ours if 15.5). Sybase first introduced unicode support in version 11, and improved it in 12.5
(source: http://www.sybase.com/content/1026310/u ... ort_wp.pdf).
Based on his comment, and based on your last comment in your other forum thread:
We have to always have char_convert on and to be safe we also always bind.
I am going to at least update the defect info. with your comment and note that this does not appear to be an issue for you any longer. As of right now the verdict is: N.A.B. (not a bug)

Return to “Bug Reports”

Who is online

Users browsing this forum: [Ccbot] and 0 guests