Ordinal growth and the extension DLL pattern

Symbian OS provides a combination of compatibility with customisability. This means that Symbian must maintain binary compatibility in its APIs, while at the same time allowing derived platforms and products to add innovative and differentiating functionality to these APIs in order to customise the OS. The extension DLL pattern is a method that allows licensees to extend APIs without introducing binary compatibility problems.

• Potential problem of ordinal growth

• The Extension Dll pattern

• Extension pattern example code

The figure below shows a small part of the actual Symbian OS "family tree". The key point about this family tree is that there is no single linear evolution path from one release of an entity to the next. Symbian OS evolves from 6.1 to 7.0s; derived platforms such as Series 60 or UIQ evolve from one version to the next; and product "x" evolves into product "y". This is because at each level of the family there is requirement to add functionality to the basic platform. There is an additional requirement to maintain that functionality in future versions of that level of the family. This means there are multiple routes through which functional growth can occur: for example a particular product will need to inherit additional functionality from the Symbian OS version, from the platform version, and from its product predecessor.

Multiple functional inheritance routes cause immediate platform fragmentation due to the function ordinal linking scheme used by Symbian OS. Each function exported by a DLL is numbered sequentially. A client of the DLL calls the functions by this ordinal number. The lookup between functions and ordinals is defined in the .DEF file of the component concerned. This sequential numbering scheme means that if two parties make an addition to the same API without knowledge of the other, the functions added will be given the same ordinal number (at the end of the range). This means that code built against the modified DLL will use that ordinal number to access that functionality. The problems begin if that code is ever run against the other version of the platform; since the function at that ordinal position is not the same. Once this problem has occurred in released products, there is no way to correct it without breaking binary compatibility in one or the other platform. The diagram below illustrates what happens when an API is extended by two parties in parallel, and how the resulting ordinal space cannot be resolved.

The issue of multiple routes of functional inheritance does not pose a problem so long as each API has a single inheritance route. In practice this means a single owner. This owner is allowed to add to an API; nobody else is. This means that the API evolves in a linear fashion. For example, DLLs owned by Symbian can only be extended by Symbian, and cannot be extended by derivative platforms or by products. However, in practice this ideal of not allowing extension to interfaces creates friction with the key Symbian OS advantage of customisability. This allows products to add functionality to interfaces owned by others. This could be a platform adding an API to Symbian libraries, or products adding an API to platform libraries; the problem is the same in either case.

It needs to be understood here that the problem is not that compatibility with the underlying platform has been broken. In each case the product is compatible with the current Symbian OS/UI platform. The problem is that if a future revision of the product is made, based on a later version of a Symbian OS/UI platform, it can only be compatible with either the new version of the platform, or the previous version of the product, but not both. However, compatibility with both is essential in order to maintain compatibility across the platform.

In summary:

• Symbian OS is adding substantial functionality to its APIs in new versions of the OS

• UI platforms such as UIQ or Series 60 are adding considerable functionality to their APIs in new versions

• Products are adding functionality to Symbian OS and UI platform APIs in their products, in the same ordinal space as the Symbian OS and UI platform additions above.

This provides potential for problems with products/SDKs with conflicting ordinals. The extension DLL pattern described next enables customers to add functionality to APIs in a safe, extensible way without endangering future compatibility.

The objective of this pattern is to enable a third party to add functionality to an existing API, which is linked by ordinal, in a way which does not compromise future evolution of that API. A second objective is that any future evolution of the original API which does occur will not compromise the third-party added functionality.

This pattern is applicable to any situation where a third party needs to add functionality to an existing API which they do not own. It does not apply when the owner of an API is extending the API, as by definition they are the maintainer of the API. It is not a substitute for good binary compatibility maintenance practice, as the pattern applies only to maintaining the ordinal space of the API. API additions that have already been made can be reverse engineered and re-implemented safely using this pattern.

The pattern is not intended to allow wholesale modification of APIs by third parties. This technique is intended to be applied when analysis determines that the only effective architectural means to implement desired functionality is to add to existing APIs. In general, additional functionality should be added in new, third party owned DLLs. Even where this is not possible, additional functionality should be added in the least intrusive way possible. This means only the very minimum change should be made to the existing DLL: any other functionality should go in a new DLL. This pattern is also not a substitute for the use of a change management process to influence the owner of a DLL to add required functionality.

The figure below shows an application linking against the LIB of a platform DLL. The application uses the functionality via the LIB file ordinals, therefore when calling OriginalFunction 1, ordinal 1 of the lib is linked to, and so on. This is the standard pattern of DLL linkage employed in Symbian OS.

The next figure illustrates the Extension DLL pattern in use for some additional ordinals which have been added to the platform DLL by a product. The application that needs to use these additional ordinals uses them via an extension DLL. It is not permitted for the application to use any additional ordinals from the platform DLL directly. The most practical way to achieve this is to implement the product additional functions as private members of classes. This means that the compiler will enforce that client code must use the extension DLL. The extension DLL need not be purely an adapter; it is anticipated that it can contain considerable functionality. This is in line with the requirement that the additions to the Platform DLL must be as small as possible. Also, the Extension DLL is not specific to a single Platform DLL. There could be just a single "Product Extension" DLL in the ROM of a device that provides the interface to the additional functionality of many Platform DLLs.

As shown in the figure, the third party application is linked to Ordinals 1 to 5 of Platform.lib, and ordinals 1 and 2 of extension.lib. As far as the application is concerned, it has no visibility of ordinals 6 and 7 of platform.lib. As a result, the third party SDK for the product does not even need to ship the extended version of platform.lib. The safest solution is to substitute the original (5 ordinals) version of platform.lib in the SDK, and only use the extended (7 ordinals) version for internal development. This means that there is no possibility of the third party code using the extra ordinals.

This section contains a simple demonstration of how to code for the pattern.

Header

The following code shows how a function could be added to a platform DLL's header file. There are 2 functions added to the header. The public ExtensionFunction() will be implemented in the Extension DLL. It will therefore not be exported from Platform.dll, but from Extension.dll. The private DoExtensionFunction() will be implemented in the Platform.dll and therefore will be exported from Platform.dll. This function is made private so that the compiler will prevent third party code from calling this function through its ordinal export.

//PlatformDll.h
Class CMyClass : public CBase
    {
public:
    // The supplied API of the class
    …
    // Product additional functions
    // This public exported function will be implemented in the extension DLL
    IMPORT_C    void ExtensionFunction ();
private:
    // This private exported function will be implemented in the platform DLL
    IMPORT_C    void DoExtensionFunction ();
    }

Original implementation

The implementation in the platform.dll is to add the code for DoExtensionFunction(). This should contain the minimum of code necessary to be in this DLL, so as to minimise the amount of change required. This function will be exported in Platform.lib.

//PlatformDll.cpp
…
CMyClass::DoExtensionFunction()
    {
    …
    // implementation of the required functionality
    …
    }

Extension implementation

The purpose of the implementation of the extension.dll is to enable the call through to the private DoExtensionFunction() in platform.dll. DoExtensionFunction() is private, so can only be called from within the same class. Note that the call to DoExtensionFunction() is surrounded by other code which does not need to be in platform.dll, and therefore is implemented here to minimise the impact on platform.dll.

//ExtensionDll.cpp
CMyClass::ExtensionFunction()
    {
    …
    // implementation of functionality that does not have to be in platform.dll
    …
    DoExtensionFunction();
    …
    // implementation of functionality that does not have to be in platform.dll
    …
    }

The following are some guidelines to follow when using the pattern:

Extend the platform DLL as little as possible

This ensures that only essential changes are made to the platform DLL. All other functionality should be implemented in the Extension DLL. This means that there is less possibility of clashing functionality later on, and thus future integration of newer platforms will be less troublesome.

Extend the platform DLL with private functions

The extensions in the platform DLL should be made as private functions. This makes it more difficult for an external client to inadvertently use an extension function in the original API rather than in the extension DLL. The compiler will then help to enforce the rule.

Consider other aspects of binary compatibility

This pattern only addresses the issue of ordinal space management when making additions to interfaces. There are many more aspects of BC which are just as important, for example enum ranges, vtable layouts, class sizes, function signatures etc. It is important not to cause a BC break in these other areas.

Include the originally delivered platform lib files in any public SDK

A publicly released SDK should only include the original LIB files as supplied. This means that third party code cannot possibly call any of the product additional ordinals, as they are not present in the LIB files. They therefore have no choice but to access the extension functionality through the extension DLL, and therefore will not encounter any problems with future platforms.

Ensure no code links against additional ordinals, except the extension DLL

By making extension functions private, and excluding the modified libs from the SDK, third party code is prevented from calling the product additional functions. However, it is also good practice to ensure that ROM code also uses the extension DLLs. Although ROM code can be rebuilt in the future against new LIB files, it is always best to maximise binary compatibility wherever possible, and the extension DLL mechanism enables this.

Notify the API owner of changes where possible

Where functionality needs to be added to a supplied DLL, this generally indicates that the DLL is not fully fulfilling the platform's requirements, or that it is being used for a purpose that the original designer did not foresee. Therefore, if a DLL needs to be extended, it is good practice to notify the owner of the interface and request that the required functionality be added to the platform.

Use tools to verify compatibility

The BC Comparator tool can be used to assist in the verification of compatibility, and check that no potentially future damaging changes have been made.

The examples in examples\Basics\ExtensionPattern\ provide buildable projects that shows the pattern being used. There are six components. Four are built from the group directory:

• OriginalDll: an original implementation of a trivial DLL that contains a class that stores 2 numbers

• OriginalClient: client code that links against OriginalDll.lib and demonstrates the functionality

• ExtensionDll: an extension DLL that adds 2 functions to the class in OriginalDll to add & multiply the 2 numbers together

• ExtensionClient: client code that links against OriginalDll.lib and ExtensionDll.lib and demonstrates the new functionality

The remaining two projects are built from the group_v2 directory:

• OriginalDll_v2: the same DLL as OriginalDll, but with some new functions added by the supplier

• ExtensionDll_v2: the same DLL as ExtensionDll, with identical code.

OriginalDll has six functions in its DEF files, but its header file has been modified to add:

• Two public exports, the AddNumbers() and MultiplyNumbers() functions.

  The implementations of AddNumbers() and MultiplyNumbers() are in the ExtensionDll project, and therefore the exports appears in the ExtensionDll DEF files.

• a private export, the DoMultiplyNumbers() function. This function is made private, so that it cannot be called by clients of OriginalDll. The DoMultiplyNumbers() function is necessary because it needs access to a const, KMagicMultiplyer, which is defined in originaldll.dll and therefore can't be implemented elsewhere. This is an example of an intrusive function.

It is possible to use the 6 original exports in OriginalDll by simply linking against OriginalDll.lib, as shown in OriginalClient.

To use the additional functions, the client must link to both OriginalDll.lib and ExtensionDll.lib. This is shown in ExtensionClient.

The OriginalDll_v2 and ExtensionDll_v2 projects demonstrate what happens when a supplier releases a new version of their DLL with new ordinal functions. OriginalDll_v2 includes some new supplier functions, which means that the extension functions move in its DEF file. This is taken care of by the ExtensionDll, which is rebuilt. Once this is done, the client code will work exactly as before, without needing a rebuild. Therefore the extension to the functionality is achieved without causing any BC breaks.

To build OriginalDll, ExtensionDll, OriginalClient and ExtensionClient, go the group directory and enter:

> bldmake bldfiles

> abld build

To build OriginalDll_v2 and ExtensionDll_v2, go to group_v2, and enter:

> bldmake bldfiles

> abld makefile

> abld reallyclean

> abld build

The cleaning step is necessary to remove the previous versions of OriginalDll and ExtensionDll.

The following gives a brief description of each example file:

eabi\EXTENSIONDLLU.DEF - ARM DEF file for ExtensionDll

eabi\ORIGINALDLLU.DEF - ARM DEF file for originalDll

eabi\ORIGINALDLL_V2U.DEF - ARM DEF file for OriginalDll_v2 (containing new functions)

bwins\EXTENSIONDLLU.DEF - WINS DEF file for ExtensionDll

bwins\ORIGINALDLLU.DEF - WINS DEF file for originalDll

bwins\ORIGINALDLL_V2U.DEF - WINS DEF file for OriginalDll_v2 (containing new functions)

group\bld.inf - Makefile for OriginalDll, ExtensionDll, OriginalClient & ExtensionClient

group\ExtensionClient.mmp - Project definition for ExtensionClient

group\ExtensionDll.mmp - Project definition for ExtensionDll

group\OriginalClient.mmp - Project definition for OriginalClient

group\OriginalDll.mmp - Project definition for OriginalDll

group_V2\bld.inf - Makefile for OriginalDll_v2 and ExtensionDll_v2

group_V2\ExtensionDll_v2.mmp - Project definition for ExtensionDll_v2

group_V2\OriginalDll_v2.mmp - Project definition for OriginalDll_v2

include\NumberStore.h - header file for CNumberStore class

include\NumberStore_v2.h - header file for version 2 of CNumberStore class

src_extension\NumberStoreExtensionImplementation.cpp - Source for ExtensionDll

src_extensionclient\ExtensionClient.cpp - Source for ExtensionDll client console app

src_extension_v2\NumberStoreExtensionImplementation_v2.cpp - Source for ExtensionDll_v2

src_original\NumberStoreOriginalImplementation.cpp - Source for OriginalDll

src_originalclient\OriginalClient.cpp - Source for OriginalDll client console app

src_original_v2\NumberStoreOriginalImplementation_v2.cpp - Source for OriginalDll_v2