Solved TD-23116: TD build option: add/edit custom manifests

Missing or incomplete OpenText Gupta product features? Discuss it here!
Dave Rabelink
Founder/Site Admin
Founder/Site Admin
Netherlands
Posts: 3384
Joined: 24 Feb 2017, 09:12
Location: Gouda, The Netherlands

TD-23116: TD build option: add/edit custom manifests

Post by Dave Rabelink » 11 Apr 2019, 10:00

Ticket # 2757730
Registered on : 10-10-2016

REMARK: this feature request was officially registered years ago. So to inform about the existence of such a feature request, I place it here.


**************************************************


In earlier versions of Windows it was possible to have external manifests for TD applications by placing the manifest file along with the application. The external manifests had priority, so windows would use custom ones when deployed with no problem.

With newer versions of Windows (at least in Win7 and up) the priority has been changed in Windows. It now will use the internal one and ignore external manifests.

The only way to get the old way working is to change/add a registry key in Windows to prefer external manifests over the internal ones:

Code: Select all

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\SideBySide]
"PreferExternalManifest"=dword:00000001

But this is a system wide setting and it may be prohibited by companies to change this setting.
This means that TD developers are cut off from a basic Windows feature to use manifests.

In other dev environments (eg Visual Studio) you are able to edit/add your own manifests and embed it as internal.

Why we would need this?

* Setup side-by-side installations
* Adding PATH settings to the executable without using batchfiles and/or changing PATH system variable
* COM components usage without registering them on the system

Specially the last feature is very powerful and I would like to emphasize this feature as major advantage:

You can xcopy any COM based components on a system, have multiple versions of COM interfaces to different dll versions on a system without registering them. TD is able to use the COM components as if it was registered and on the fly change the version to use.

An example:
Add this line to the exe manifest:

Code: Select all

<dependency>
<dependentAssembly>
<assemblyIdentity type="win32" processorArchitecture="x86" name="Philips.SpeechMike" version="3.5.350.05" />
</dependentAssembly>
</dependency>

(stating to use speechmike COM component specific version)

and then (excerpt) the manifest for the COM component:

Code: Select all

<assemblyIdentity
processorArchitecture="x86"
version="3.5.350.05"
type="win32"
name="Philips.SpeechMike">
</assemblyIdentity>

<file name="SpMikeCtrl.dll" hashalg="SHA1">
<comClass
clsid="{4A0948DA-2C0A-4995-9679-4C609B2E3F01}"
tlbid="{90D8C17A-63DC-489C-9742-0EAA2F48D236}"
description="BarcodeKeyboardSettings Class">
</comClass>
<typelib
tlbid="{90D8C17A-63DC-489C-9742-0EAA2F48D236}"
resourceid="1"
version="1.0"
helpdir=""
flags="HASDISKIMAGE">
</typelib>
</file>
<comInterfaceExternalProxyStub
name="IBarcodeKeyboardSettings"
iid="{35EA0B74-5BC4-11DC-8314-0800200C9A66}"
tlbid="{90D8C17A-63DC-489C-9742-0EAA2F48D236}"
proxyStubClsid32="{00020424-0000-0000-C000-000000000046}">

Here you can specify all the interfaces to the dll with the needed GUIDs and classnames.

A TD application can then use this dll as COM without registering.

I ask for a feature to be able to view/edit/add your custom manifest so it will be compiled as resource in the exe/dll.

In build settings I imagine a way to view the current manifest, manually edit it or import an existing from a file.
Then when TD builds the exe it will use that resource.
Regards,
Dave Rabelink

Image
Articles and information on Team Developer Tips & Tricks Wiki
Download samples, documents and resources from TD Sample Vault
Videos on TDWiki YouTube Channel

Dave Rabelink
Founder/Site Admin
Founder/Site Admin
Netherlands
Posts: 3384
Joined: 24 Feb 2017, 09:12
Location: Gouda, The Netherlands

Re: TD-23116: TD build option: add/edit custom manifests

Post by Dave Rabelink » 11 Apr 2019, 10:05

Now follows some information exchange with Gupta (at that time).
(See next postings)

Related to TD-23203: Request for info on internal manifests
Regards,
Dave Rabelink

Image
Articles and information on Team Developer Tips & Tricks Wiki
Download samples, documents and resources from TD Sample Vault
Videos on TDWiki YouTube Channel

Dave Rabelink
Founder/Site Admin
Founder/Site Admin
Netherlands
Posts: 3384
Joined: 24 Feb 2017, 09:12
Location: Gouda, The Netherlands

Re: TD-23116: TD build option: add/edit custom manifests

Post by Dave Rabelink » 11 Apr 2019, 10:17

Dave Rabelink on sept 2016 wrote: We are in need to add our own custom manifest to TD build executables in our project.
The manifest contain path redirections and most importantly COM component GUId descriptions. This is to support installation of 3rd party components without registering them on the system.

The enhancement request is asking for a default TD option to be able to change/add custom manifests in TD builds.

This ticket is about getting information how we could add manifests manually.

To quickly explain what we need:

- TD adds a manifest to executables (under the hood)
- Windows (by default) will use this manifest first. Even when external manifest file is present, this internal one will be preferred.
- To use a custom internal manifest we should then replace the TD manifest by our own
- Visual Studio offers a way using the mt.exe tool to add/change the internal manifest.
- The mt.exe tool uses the Windows API to access the resources of an executable.

Problem:
- mt.exe can not be used on TD build executables. After mt processed replacing the internal manifest, the TD executable is truncated. All custom code in the PE header of the executable is cutoff and we get a exe file of a few kilobytes.
(the manifest is added but the rest of the TD executable file is ignored and stripped).

As we have an automated build process, we need to add the manifest after the exe has been build.

I tried the WIndows API but have the same issue with the truncation of the TD executables. Probably I'm using the same API as mt.exe does. So we can not use this API to call in the automated build process.

Fortunately we found another tool : Resource Hacker.
This tool is able to replace the manifest with our custom one correctly. The TD executable is not cutoff and works like expected in our system.

Issue with this:
- Resource Hacker is a 3rd party tool not from Microsoft and offers issues here to get officialized.
- To add this in our automated build process, we must call this Resource Hacker using command line and offers difficulty to error check.

What we need:

A way to use Windows API or other programmatic ways to add a manifest on TD executables.

We ask here for more detailed technical info how TD executables are build up and how we could be able to programatically (using normal API's from TD or WIndows) add our own manifest to existing TD executables.

How does the TD compiler add the internal manifest and is there a way for us to mimic this. This could be c++ functionality.

Is the TD executable structure documented in any way so we could

- load the existing TD exe from disk
- replace the existing manifest with our own manifest which is also on disk
- Change the PE of the executable in such a way this manifest and the rest of the TD structure keeps intact.

Joe Bunger wrote: The executables built by Team Developer are generated from a stub that is built and then embedded in TD’s IDE as a resource. The stubs have their manifests embedded when they are built prior to the stubs themselves being embedded in the IDE’s resource, that resource (stub) is then used to build the executable. There are three different stubs we build from for win32, one for a standard exe, one for an out-of-proc COM server, and one for an in-proc COM server (.dll).
This design is not really equipped to allow for manifest editing, however what we might do is give the customer a copy of the stub they are interested in modifying. They could then edit the manifest in the given stub and we could give the IDE a mechanism to then use that stub instead of the embedded one. The manifest editing would be done by an external tool such as Resource Hacker.
The least impacting mechanism would be a registry entry that points to the file to use as the stub. This gives the opportunity to customer to test the idea without risking regressions for any other builds. We might then evaluate cost/benefits and perhaps give an easier solution from the perspective of the TD developer.

SetResource: works in TD42, not in TD63

I totally forgot, but I have found an old article explaining TD resources.
Also a tool was supplied to manage the resources, called SetResource.
So this is exactly what we need.

I have adapted the sources such that it runs in TD42 and TD63.

Based on the info in the article, it seems that TD stores the offset of the TD custom code part in a resource.
The tool is actually doing what we need: read the correct offsets, manipulate the resources using WinAPI and construct a working EXE having the resources changed.

The tool works perfectly in TD42 (any ANSI version for that matter).

But the converted TD63 version does not.

I checked that when running the tool in TD63 and manipulate a ANSI EXE (eg created in TD42), the tool is creating a working EXE in the end. So the TD63 port of the tool is correct.

But the tool is not able to manipulate the same action (adding manifest) on UNICODE sources (TD63 EXE).

The resulting EXE is somewhat smaller than the original and can not be started.

So there was an internal change in the EXE structure (PE header) so that the tool is unable to create a working EXE.

I gathered everything into the ZIP.
It contains the original article (PDF).
The folder contains a TD42 version of the tool (SetResource42.apt)
And a TD63 version (SetResource63.apt).

SetResource.zip

Run the TD42 tool.

- Press the open button
- Select the file: SomeApp42.srf
- A manifest is to be added (srf file contains the info on the action)
- Press the button to execute the manipulation (the gears icon)
- Now, SomeApp42.exe will be manipulated. It is a TD42 build executable (in fact it is the SetResource application).
- See afterwards that the EXE contains the manifest and can be run

I have added SomeApp42_original.exe. After manipulation of the EXE you can start over by copying the original EXE to SomeApp42.exe.

Run the TD63 tool.

Do the same for the TD42 EXE and see that the resulting EXE works on TD42. This means that the TD63 tool is working as intended.

Run the TD63 tool but now select SomeApp63.srf.
This will use this EXE : SomeApp63.exe
The result is a non working EXE having the manifest in place.
(also the original file is supplied, copy to SomeApp63.exe to start over)

Looking at the core of the tool:

frmSetRes.GetEndOfFileOffset( )

This function gets the offset out of the PE header.
This offset is the same as the resource info #1 (on TD42) but not on TD63.

frmSetRes.SetResource( )

Is using the offsets and writes a EXE and is using WinAPI to manipulate the resources.

The result is incorrect in TD63.

So, how to get this working in TD63?
What header changes are done between ANSI and UNICODE EXE's?
Should the tool use extra or less offset calculations? Could be the ANSI version is not taking extra offsets into account.
I tried to figure this out using HEX editors and compare tools, but I could not find a logical place to calculate the offset.

So in the end my conclusion is:

ANSI TD versions do have the PE header and info such a way we can use WinApi and some custom code to add manifests.

Probably it could also be done on UNICODE versions, but the tool is not completely correct in getting the offsets right.
You do not have the required permissions to view the files attached to this post.
Regards,
Dave Rabelink

Image
Articles and information on Team Developer Tips & Tricks Wiki
Download samples, documents and resources from TD Sample Vault
Videos on TDWiki YouTube Channel

Dave Rabelink
Founder/Site Admin
Founder/Site Admin
Netherlands
Posts: 3384
Joined: 24 Feb 2017, 09:12
Location: Gouda, The Netherlands

Re: TD-23116: TD build option: add/edit custom manifests

Post by Dave Rabelink » 11 Apr 2019, 10:25

Gupta dev wrote: The executables built by Team Developer are generated from a stub that is built and then embedded in TD’s IDE as a resource.
The stubs have their manifests embedded when they are built prior to the stubs themselves being embedded in the IDE’s resource, that resource (stub) is then used to build the executable. There are three different stubs we build from for win32, one for a standard exe, one for an out-of-proc COM server, and one for an in-proc COM server (.dll).

This design is not really equipped to allow for manifest editing, however what we might do is give the customer a copy of the stub they are interested in modifying. They could then edit the manifest in the given stub and we could give the IDE a mechanism to then use that stub instead of the embedded one. The manifest editing would be done by an external tool such as Resource Hacker.

The least impacting mechanism would be a registry entry that points to the file to use as the stub. This gives the opportunity to customer to test the idea without risking regressions for any other builds. We might then evaluate cost/benefits and perhaps give an easier solution from the perspective of the TD developer.

Dave Rabelink wrote: Hi,

I have read the notes and was thinking about a sound way to cope with manifests in TD and have a easy solution for TD developers.

I think I have a design suggestion. I will create a Word document explaining this. Maybe this solution can be implemented.

Let me create the document and post it here. I'm currently busy on this and when finished will attach the document here.

Here it is:

Design_TD-Manifest_support.pdf
You do not have the required permissions to view the files attached to this post.
Regards,
Dave Rabelink

Image
Articles and information on Team Developer Tips & Tricks Wiki
Download samples, documents and resources from TD Sample Vault
Videos on TDWiki YouTube Channel

Dave Rabelink
Founder/Site Admin
Founder/Site Admin
Netherlands
Posts: 3384
Joined: 24 Feb 2017, 09:12
Location: Gouda, The Netherlands

Re: TD-23116: TD build option: add/edit custom manifests

Post by Dave Rabelink » 11 Apr 2019, 10:29

Gupta dev wrote: Development appreciates the effort you have put into the implementation document and after review they have concluded that a full implementation as suggested in the PDF would need to be considered for TD7.2.
Support would be happy to log a feature request for TD7.2 based on the doc provided.
In the meantime, would Dev's suggested implementation - even though partial - suffice? If yes, they will go ahead with it.

Gupta dev wrote: About TD-23203 this will not get a FIX this in light of TD-23116

Reason for this comment from DEV for TD-23203: " TD does not add a manifest at build time, instead it uses pre-built exe stubs that are built from Visual Studio projects and embedded in the IDE resources. The IDE takes those stubs and injects the pcode (compiled SAL code) as a resource into the exe, not touching the manifest at all. These tools seem to not keep that resource in place, which results in the truncated exe."

So the ticket 2849540 specific about truncation will close as defect won't fix.

About TD-23116 this is a feature request still open
Regards,
Dave Rabelink

Image
Articles and information on Team Developer Tips & Tricks Wiki
Download samples, documents and resources from TD Sample Vault
Videos on TDWiki YouTube Channel

Dave Rabelink
Founder/Site Admin
Founder/Site Admin
Netherlands
Posts: 3384
Joined: 24 Feb 2017, 09:12
Location: Gouda, The Netherlands

Re: TD-23116: TD build option: add/edit custom manifests

Post by Dave Rabelink » 25 Apr 2023, 11:16

Custom manifests are now implemented in TD 7.5
Regards,
Dave Rabelink

Image
Articles and information on Team Developer Tips & Tricks Wiki
Download samples, documents and resources from TD Sample Vault
Videos on TDWiki YouTube Channel

Return to “Enhancement suggestions”

Who is online

Users browsing this forum: [Ccbot] and 1 guest