Design and implementation of STDLIB

This is a description of some of the main features of Symbian OS's implementation of the C Standard Library (referred to as STDLIB). It provides information on how it differs from other such libraries and descriptions of some of its main features. It does not attempt to document the library's entire API. For such documentation, see the following:

• for documentation of the ANSI C and POSIX specifications, see the relevant organisations' web sites (http://www.ansi.org and http://standards.ieee.org/index.html).

• For literature on ANSI and POSIX specifications, see POSIX Programmers Guide— Donald Lewine O'Reilly & Associates, Inc.— ISBN 0-937175-73-0

The following are some of the reasons behind the implementation of STDLIB:

• to address the porting requirements of C software engines to Symbian OS.

• to provide an ANSI C library with associated POSIX system calls.

• to support both pure C programs and mixed C/C++ programs.

• to provide the standard include file structure with the standard #defines and function prototypes.

The following are not among STDLIB's design goals:

• to support pre-ANSI C compilers.

• to pass ANSI or POSIX conformance test suites.

• to support POSIX-style tools executed from a shell. Symbian OS does not provide such a shell, nor does it provide a sophisticated text console.

• to be the C API for Symbian OS. For example, STDLIB does not provide any new C functions for supporting threads or active schedulers.

• to provide new C++ interfaces. The public API for STDLIB is the C functions themselves, and almost no aspects of the underlying C++ implementation are exposed, even to C++ programmers.

• Many ANSI standard functions have been implemented, including all those in stdio.h and math.h. The functions which have been implemented are those in the STDLIB header files whose prototypes are preceded by the IMPORT_C macro.

• STDLIB's maths library, math.h, provides the mandatory ANSI maths functions. Most of these use the underlying Symbian OS maths functions, encapsulated by the Math class.

• Many functions, including bsearch(), qsort(), memcpy(), and tolower() are little more than a call to a corresponding function provided by Symbian OS. In many cases, this function may be found in Symbian's User class. For example, malloc() and free() simply call User::Alloc() and User::Free() respectively.

• The STDLIB implementation of printf() does not take into account the format specifier for floating point values. Instead, the implementation uses TDes8::Format() with a Symbian OS format specifier which is derived from the style, width and precision of the original printf() specifier. The result does not obey the rules for ANSI printf() for some combinations of number width and precision, and formats the numbers using the thousands separator taken from the Symbian OS locale; see class TLocale.

• Symbian OS identifies some errors as being fundamental programming errors and will abruptly terminate the offending thread (referred to as a panic). POSIX never does this and will always return error conditions instead (though some errors will result in signals, e.g. segmentation faults).

• POSIX processes exist in a hierarchy with children inheriting resources from their parents — the "fork and exec" model of process creation. In Symbian OS, processes have no association whatsoever with the thread which created them. They are created by constructing a fresh instance of an executable image.

• Symbian OS provides limited text console support, intended for debugging of low-level components.

• An open file in the POSIX system is a process wide resource that can also be inherited by child processes. In Symbian OS each thread within a process has separate resources within the shared process address space and open files are private to a given thread.

• Open POSIX files are inherently shareable, so multiple processes can write to the same file simultaneously. In Symbian OS, open files are inherently non-shareable, and even the thread which has the file open is only allowed one outstanding write operation at a time. For more information on files, see the RFile class.

• Symbian OS does not support global data in DLLs. To overcome this, STDLIB uses a structure, _reent (declared in reent.h) an instance of which is allocated to each thread to hold the errno value, some "static data areas" used by functions such as ctime(), and the STDIO data structures.

• Symbian OS has different client classes for each type of resource: files, sockets and the console are all different and have different APIs. POSIX file descriptors can refer to all of these different resources and have a common API (though many functions will actually only work on socket file descriptors).

• Symbian OS processes do not have a notion of a current working directory. The closest notion is the session path, as set and retrieved by class RFs.

• Symbian OS does not have a way of pre-empting a thread and causing it to execute a different piece of code, so there is no true equivalent of a POSIX signal or signal handler.

Simple C programs may not need to support file descriptors shared between threads, or the execution of sub-processes. Such programs may use the default implementation of STDLIB, in which the library code opens files, sockets etc. in the context of the calling thread, and provides a per-thread table of open file descriptors.

Multiple threads may still be used, but each thread's resources are private and cannot be shared with other threads. For example, a setenv() call in one thread will not be seen by a getenv() call in another and each thread will have a separate console window (created on demand when the console is first read from or written to).

More complex programs may need to use process-wide resources. This is often true of programs which assume the existence of support for multiple threads within a POSIX process. To meet this requirement, STDLIB can operate in a mode in which all shareable resources are owned by a single CPosixServer thread. In this mode, library routines such as open(), read() and write() operate by passing an appropriate request to the CPosixServer.

The program's mode of operation is determined when it first tries to use STDLIB's services. If a CPosixServer is running for the Symbian OS process, the thread will use it; otherwise the program will operate in the single-threaded mode.

The CPosixServer is a Symbian OS active object, and can be started either in the context of an existing active scheduler, or by spawning a separate thread to run an active scheduler. The functions for doing this have a C++ interface, defined in estlib.h. For more information on active objects in Symbian OS, see active objects.

Communication between CPosixServers is used to establish the POSIX process hierarchy and to communicate resources from parent to child. Programs which require multiple processes must use the multi-threaded mode of operation.

Most error codes reported by STDLIB functions correspond to the standard C error code values. Some of these are identified within STDLIB, which produces the correct errno value directly; most are reported by translating the Symbian OS error codes into equivalent errno values using a translation table. Occasionally, a Symbian OS error code may be reported untranslated. In this case, it will have a negative value. For the Symbian OS error codes, see KErrNone etc. STDLIB does not usually attempt to detect inputs which will cause a panic, (for example attempting to accept() on a stream socket which has not been bound). Such a panic will terminate the offending thread, which may in turn result in the termination of the whole process.

STDLIB provides the following CRT0 libraries:

Simple Mode (thread local mode) ecrt0.lib EABI Emulators Narrow char main (int argc, char *argv[], char *envp[]) wecrt0.lib EABI Emulators Wide char main (int argc, wchar_t *argv[], wchar_t *envp[]) Complex Mode (multi threaded mode / POSIX server mode) mcrt0.lib EABI Emulators Narrow char main (int argc, char *argv[], char *envp[] wcrt0.lib Emulators Narrow char main (int argc, char *argv[], char *envp[]) with stdxxx redirection to Win32 using RWin32Stream server wmcrt0.lib EABI Emulators Wide char wmain (int argc, wchar_t *argv[], wchar_t *envp[]) wwcrt0.lib Emulator Wide char wmain (int argc, wchar_t *argv[], wchar_t *envp[]) with stdxxx redirection to Win32 using RWin32Stream server Other estw32.dll Emulators: Provides the RWin32Stream server redircli.dll EABI Emulators: Provides application specific redirection (e.g. to Java RT or test applications)

These static libraries provide the compiler runtime glue code that runs from the Symbian E32Main() entry point for regular Symbian EXEcutables into the standard C main() entry point.

These executables are considered to be C programs hosted in a Symbian executable image. The glue code has a number of functions to perform, for example:

• marshalling the process command arguments in the types used by main()

• initialising the C Standard library context and mode of operation e.g. calling SpawnPosixServerThread().

A normal Symbian OS executable provides an E32Main() function which is called by the operating system to start the program. The file ecrt0.lib provides an E32Main() function for projects which use STDLIB. This function prepares the traditional argc and argv arguments and passes them to main().

For a simple example demonstrating how to link to ecrt0.lib, see the description of the project specification for "Hello World" in Porting. The user of STDLIB does not need to use ecrt0.lib, and may provide their own E32Main(). In this case, a CTrapCleanup pointer should normally be provided. This pointer is required because although STDLIB does not call Symbian OS functions which can leave, it does use some functions which require a cleanup stack. Code in ecrt0.lib provides such a cleanup stack ("TheTrapCleanup"), but programs which do not link with ecrt0.lib will need to supply one directly.

For more details on the motivation behind Symbian's use of the cleanup stack, see Cleanup support.

Note, however, that if your program is a UI application, i.e. one that includes the following code in one your source files:

GLDEF_C TInt E32Main()
    {
    return EikStart::RunApplication(NewApplication);
    }

then, the CTrapCleanup pointer is provided by the application framework.

STDLIB provides a POSIX-like abstraction of file descriptors which unifies the different types of resource and permits a single API to be used across all of them. This is a significantly different approach from Win32 and Symbian OS, both of which have separate APIs for each distinct type of resource.

STDLIB supports files stored in the file system, sockets, a console, and a /dev/null device. The first time STDLIB initialises its internal file descriptor table it creates an emulated console device and attaches it to descriptors 0, 1 and 2. The emulated console device will appear as a window when it is first used (i.e. when the program writes to or reads from the console).

The open() function recognises the following names:

• CON: is taken to mean a newly-created console. This will never be the same console as the one automatically associated with 0, 1 and 2.

• NUL: is taken to mean a /dev/null device.

• TMP: is taken to mean a temporary file, which will use the underlying Symbian OS file system facilities to create a uniquely-named temporary file, and will cause the file to be deleted after it has been closed cleanly.

• COMx: is the serial port where x is a number from 1 to 9. COM1: corresponds to serial port zero and so on.

• IRCOMx: is the serial port where x is a number from 1 to 9.

The number of open files in the file has no explicit limit.

Symbian OS resources such as RFiles and RSockets are derived from class RSubSessionBase, so are thread specific. This means they cannot be used by any thread other than the one which opened them. In STDLIB however, the CPosixServer, if running, controls the master file descriptor table. In this case, all STDLIB threads in a process may share their resources, because the STDLIB implementation forwards all I/o requests to the resources owned by that process's CPosixServer thread. If no CPosixServer is running, each thread has a separate file descriptor table and the resources are not shareable.

A single CTtyDesc object is created per thread and creates an instance of the basic console object. The CTtyDesc object is attached to stdin, stdout and stderr.

Requests made on stdin, stdout and stderr are passed to CTtyDesc.

There is a separate console per thread. Therefore, output is spread over a number of different consoles.

The Posix server must be started in the crt0 glue code.

Either a single instance of CW32StreamDesc or CReDirDesc is created in the Posix server. This is attached to stdin, stdout and stderr and is used by all threads. Requests made on stdin, stdout and stderr are passed to the attached descriptor object.

CW32StreamDesc is used for the emulator, but only if the RWin32Stream server was started in the crt0 glue code.

If CW32StreamDesc cannot be used or code is compiled for hardware then CRedirDesc is used instead.

CReDirDesc connects and passes the request to a redirection server. If connection fails the output to stdout and stderr is lost. Nothing is read from stdin.

The client application (e.g Java RT or the Test application) starts the redirection server. The client also provides an implementation of CStreamFactoryBase2. This implementation deals with standard I/O data in a customised way.

Stdin/stdout on emulator in complex mode

A text EXE running on the emulator can either use the wcrt0.lib or wwcrt0.libcrt0 libraries. These automatically start the RWin32Stream server to redirect stdin, stdout and stderr to the Win32 console.

Stdin/stdout on hardware in complex mode

A test EXE running on target hardware has no default console. The test application requires application/client specific C++ code to be written to create a new thread and call CRedirServer2::NewL() to start the redirection server along with client specific derived classes to plumb the I/O to wherever it is needed. The Posix Server will then connect to this server when stdXXX streams are used.

The STDLIB console (encapsulated by class CTtyDesc, defined in fdesc.h) is a client of the CConsoleBase class implemented by econs.dll. It provides very simple text input and output with no support for embedded control sequences. When STDLIB receives a character from the console it prints it out to the same console, providing a "local echo" facility to make simple command-line interfaces possible.

STDLIB does not provide any sort of terminal driver or line-discipline. In particular there is no support for local processing of backspace, nor any line buffering. Neither does it provide termio or termcap facilities. Symbian OS is a graphics-based system and it is recommended that C code be ported into a Symbian program which uses a graphical user interface.

All STDLIB I/o operations are blocking; that is they will suspend the calling thread indefinitely until the I/o completes. So in general, STDLIB I/o should not be used in a Symbian OS active object because it will cause the entire active scheduler to block. A possible way to avoid this problem might be to use fcntl() for individual file descriptors, but STDLIB does not currently implement this function.

Asynchronous I/o may be achieved using a set of C++ functions provided by STDLIB which implement a per-file-descriptor equivalent of the POSIX select() function. These functions provide a form of the ioctl() function which takes a TRequestStatus& as a parameter, together with functions for completing the ioctl() operation once the status has been signaled or canceling the pending ioctl. This scheme can be used within an active object to wait for a socket to become ready for reading or writing, so that the subsequent i/o does not block the whole active scheduler. See estlib.h for the interface to these functions. For more information on active objects and the active scheduler in Symbian OS, see active objects.

Note that there are no such blocking problems with i/o to local files, which is essentially a synchronous operation.

The POSIX interface is designed for a single thread of execution within a process. Many aspects of this interface do not apply to a typical Symbian program in which multiple threads of execution share the same address space. For information about threads and processes, see Threads and processes.

STDLIB allows for multiple threads, but each thread owns its own instance of the _reent structure which contains private data such as the thread's errno variable. Each thread's STDIO FILE structures are completely separate from other threads', even if those structures eventually share the same underlying file descriptor. A consequence of this is that different threads will buffer their output to stdout separately, even though the eventual output will be combined together when the STDIO layer flushes the buffers out to the corresponding file descriptor.

It is unclear how some POSIX functions should be used in a multiple thread environment. An example is the exit() function. Although each thread should have separate atexit() processing, which should include closing all open STDIO files, it is unclear whether closing the STDIO file should also close the underlying descriptor. STDLIB's current implementation is to close the files, as would be expected to happen in a normal POSIX process. However, this implementation may be changed. Note that exit() does not attempt to free memory which was obtained by malloc().

The user of STDLIB can take control over thread termination by implementing exit(), _exit(), abort() and _assert() in their own program, so that all of the user's own code which calls these functions will invoke the user's routines instead of the STDLIB versions. A helper function, _atexit_processing_r(), can be called from the user's version of exit() to do the normal atexit processing, if desired. See stdlib_r.h for details.

The Symbian OS file system APIs require all filenames to be fully specified, possibly by using default components in the session path. For more information, see the documentation on the file server (File server). In particular, the file system does not support the notion of a "relative path" which uses "." and ".." to navigate relative to a "current working directory". STDLIB does support this and does not require fully qualified paths, with the constraint that any pathname beginning with "<letter> :" will be treated as an absolute path from the root of the specified drive. STDLIB allows the drive letter '?' to mean "any drive". This can be useful when a file or directory is known to exist but it could be on any drive. STDLIB allows both "/" and "\" as directory separators, unlike the underlying Symbian file system which recognises only "\".

The current working directory is a process-wide resource (similar to the file descriptor table), so the CPosixServer will provide a single process-wide current working directory. However, when using the "single-threaded" mode of operation, each thread will have a separate current working directory.

The following table gives some examples, all based on a working directory of c:\documents\stdlib

Pathname in Symbian OS In STDLIB c:\documents\stdlib\ current directory current directory c:\documents\stdlib illegal (not a file) current directory . illegal current directory .. illegal c:\documents examples filename c:\documents\stdlib\examples ..\examples illegal c:\documents\examples d:examples illegal d:\examples ?:\system\data\ illegal c:\system\data (if file exists on c) or d:\system\data (if it exists on d but not on c) or z:\system\data (if it exists in the ROM).

In summary, the POSIX and STDLIB handling of pathnames is DOS-like, with the exception that there is not a separate working directory per drive.

The Symbian file system supports DOS-like attributes, rather than POSIX-like permissions. STDLIB cannot therefore provide the full POSIX-like handling of file attributes and implements only "user read permission".

STDLIB's implementation of fseek(), unlike some other implementations, does not allow you to seek beyond the end of a file and expand the file to the new position.

The Unicode changes for Standard Library come in two groups:

• impact on the existing "char*" interfaces

• addition of new "wchar_t*" functions which use Unicode directly

Symbian OS is a Unicode-based operating system, so all operating system services which use text require that text to be presented in the 16-bit Unicode character encoding known as UCS-2. For the Standard Library on Symbian OS, this is most significant in dealing with the names of files and directories, all of which are now Unicode sequences.

To minimise the impact of this change on existing narrow C code, the Standard Library has adopted the policy that all such names in char* interfaces will be interpreted using the UTF-8 standard for encoding Unicode strings as 8-byte sequences. UTF-8 is a no surprises encoding and matches the 7-bit ASCII encoding for character codes 0 to 127, so existing string handling code will work without modification.

The wchar_t type is defined and ISO-C standard wide character constants are supported. The wchar_t definition chosen is unsigned short to match the use of UCS-2, and a range of relevant functions now have wide character analogues which use wchar_t* in place of char*, for example:

FILE * fopen   (const char *_name,    const char *_type);

FILE * wfopen (const wchar_t *_name, const wchar_t *_type);

and

DIR *     opendir  (const char *);

WDIR * wopendir (const wchar_t *);

Where such a pair of functions exists, the char* interface is implemented by converting the UTF-8 parameters to Unicode and calling the matching wchar_t* interface.

The mbtowc family of conversion functions is provided to convert between UTF8 and Unicode, but there is no additional support for locales or other forms of multibyte encoding; to convert from encodings such as Shift-JIS, programmers are recommended to use the CHARCONV conversion routines via C++ wrapper functions callable from C.

There are no implementations of wchar_t* versions of STDIO functions such as fputc.

Symbian has no plans to provide the following in the C Standard Library:

• float-sized maths operations. POSIX requires only double-sized operations. Primitive arithmetic operations on float values can be carried out, and all of the standard printf format specifiers including %E and %e are supported. However, none of the floating point library functions declared to take float-sized arguments are supported. For example;

  double sqrt (double d);    // Supported by STDLIB
float sqrtf (float f);    // Omission

• mapping the ANSI locale functions onto the Symbian TLocale functionality. Instead the locale is a fixed "C" locale. Therefore STDLIB engines may not be localised. It is recommended that localisation be implemented at the Symbian OS application level.

• select(). The POSIX select() function is not implemented in STDLIB because Symbian OS does not in general support asynchronous "ready to read" or "ready to write" notification.

• signal(). A Symbian thread cannot be made to spontaneously execute a "signal handler", so signals have not been implemented.

• fork() or exec() functions

• IPv6 support