def-foreign-call

Arguments: name-and-options arglist &key call-direct callback convention returning method-index release-heap release-heap-ignorable arg-checking optimize-for-space strings-convert error-value

This macro creates the specification which allows Lisp to correctly call non-Lisp code. Like other defining forms, its macroexpansion clearly shows what will occur and at what eval-when times. The execution of the expanded form always returns the Lisp name being defined. The definition that is installed is a Lisp function that serves as a wrapper and initiates the foreign call.

Macroexpansion of a def-foreign-call form provides useful information about how the call is interpreted. See below.

The following table shows the arguments. The first two entries are the required arguments and the remainder are keyword arguments.

Note 1: Argument Specifications

Argument specifications are available with a rich set of syntax and defaults which allow for a C "feel" while still retaining the Lisp semantics and power.

The basic argument list is simply a list of names, e.g. (a b c d), whose types default to C int types converted in the usual manner from Lisp integer types. Any one argument can be specified more fully, as a list of the name optionally followed by the foreign type optionally followed by the Lisp type (optionally followed by a translation specification, which has not been implemented). Here is an example:

  (a (b :int fixnum) (c :lisp))

The foreign type can either be a built-in foreign type or a type defined by def-foreign-type, or it can be one of the keywords :lisp, :foreign-address, or :single-float-no-proto (deprecated). The value :lisp means the argument will be passed unchanged, analogous to the returning type :lisp. The lisp type (the optional third element of the list) can be any valid lisp type.

If the foreign type is not specified, it defaults to :int (and the Lisp type thus defaults to integer). If the foreign type is specified but the Lisp type is not, a reasonable Lisp type is chosen to correspond to the foreign type. If both foreign and Lisp types are chosen, they will be checked for consistency and a warning might be given. A foreign type specification that includes a reference spec such as (& :int) will be interpreted as a pass-by-reference argument.

For boolean values, specify the argument (argname :int boolean). Then any non-nil Lisp value (including 0) will be converted into a C value of 1, and nil will be converted into a C value of 0. For returned values, a C value of 0 is converted into nil and a non-zero C value is converted into t.

The special Lisp type specification :no-proto is provided for use with a :float foreign type; it is equivalent to the :single-float-no-proto. Note that both of these specifications are deprecated though they will work. The preferred specification is (:double single-float).

Specifying the value (* :float) or (* :double) as the type of an argument is not recommended. The function will expect a foreign address (not a Lisp address) and it will not pass the address of a Lisp float is a Lisp float is given as a value.

See A note on foreign addresses in foreign-functions.htm for a discussion of foreign addresses and what is expected when :foreign-address is specified.

Note 2: Possible Values for Convention

Other than :c, (the default and suitable for most purposes), the :fortran convention is defined. This convention generally causes a conversion of most atomic arguments to pass-by-reference.

:fastcall convention on Windows does not work

:fastcall is a special convention used by some Windows operating systems to speed up some calls, by passing two arguments in registers. However, it does not work with Allegro CL. In Allegro CL on X86 architectures, the first two arguments are usually passed in registers anyway in Lisp, but they are different registers than used in fastcall. Also, since the calling sequence itself overshadows the speed that would be gained by saving a couple of push instructions, the foreign call to a fastcall function would not in fact be very fast at all.

Note 3: the release-heap keyword argument

The native-threads implementation of Allegro CL changes some basic assumptions of the foreign functions user interface. There is always exactly one native thread per Lisp Process, but there is not necessarily a Lisp process for every thread. Threads are free to run whenever they want; however, only one thread at a time can access the Lisp heap (for read or write); a thread cannot access the Lisp heap unless it has "acquired" the heap, which is only possible after another thread has "released" the heap.

See Releasing the heap when calling foreign functions in foreign-functions.htm for more information on this point.

def-foreign-call allows for the specification of whether to release the heap or not during a call. The possibilities for the release-heap keyword argument are:

• :never - This is the default and is compatible to the original defforeign interface; the caller necessarily has the lisp heap, this call will not release it. Note that if the foreign code being called spawns any new threads, or if it allows another thread to run, and the other thread attempts to call back into lisp, it will have to wait for the lisp heap. The danger is that the original thread may be waiting for results from its partner thread, but it has not yet given up the heap (this constitutes a deadlock situation). If this situation holds (foreign code does spawn new thread which call back into Lisp), :when-ok is the appropriate value for the release-heap argument.
• :always - The foreign call always gives up the heap before making the transition into non-lisp execution. If the form is being called within the dynamic context of a without-interrupts form, an error is signaled.
• :when-ok - The current dynamic context is examined, and if without-interrupts is in effect, the heap is not released; otherwise the heap is released.

Note 4: String conversion when no arguments are specified

(ff:def-foreign-call foo ()) has no arguments specified, meaning that any number of arguments of any type will be passed. If any of these arguments are strings, it may be that string conversion should be performed.

String conversion is done by default. The above def-foreign-call form will generate a warning message (to indicate that string arguments will be converted). To suppress the warning but still convert the strings, specify the value t for the strings-convert keyword argument. To suppress the warning and to suppress all automatic strings conversion, specify the value nil for the strings-convert keyword argument.

Note 5: More on the error-value argument

When considering specifying a non-nil value for the error-value argument, care should be taken to assess whether or not the error value is needed. If an error value will not be used, then it should not be specified in the def-foreign-call form, because the foreign call is less efficient when retrieving the error value. However, if the error value from a foreign call is needed, then error-value is the only guaranteed option. One could define a foreign call to GetLastError or to get the current value of errno variable, but this cannot be guaranteed to work. The last errors might be overwritten between the time of the foreign call and the time of the retrieval. The wrong error code will make debugging harder, so the price of the overhead of a non-nil value for the error-value argument will be less than the price of wrong error codes.

There are two non-nil values for error-value: :errno and :os-specific. All UNIX platforms are consistent in always returning the error in or through the errno variable. But many Windows calls (e.g. LoadLibrary()) require you to call GetLastError() in order to get an error value. Other Windows functions, particularly those which emulate libc.a/libc.so functionality (e.g. open()), set errno values like UNIX. When using the argument on Windows, be sure to specify the appropriate value.

Note 6: More on the method-index argument

This argument is supported on Windows only. If a non-nil value is specified for the method-index keyword argument, then the value must be a vector whose first value is the integer index into the virtual table of the C++ class. And then, when the foreign function is called, the first argument of the call must be the vector whose first element is an integer which is a pointer to the table of method addresses. The arguments specified to def-foreign-call follow that first argument. Note that the function name specified in the def-foreign-call form is used only for the Lisp function name, and does not refer to any function in a shared library. Instead, the function's address (as a table address and an index into the table) is passed when the function is called.

Note 7: Potential problems with foreign functions returning :int in 64-bit Lisps

def-foreign-call is defined to default its :returning keyword argument (specifying the expected type of value returned by the foreign call) to :int. :int corresponds to C's int type. However, if the foreign function does not actually return an int, subtle bugs could be introduced in programs, particularly if the C function returns a long, an unsigned long, or a pointer of some sort. In 32-bit Lisps, returning those values is not a problem (when :returning :int is specified or defaulted to) because int is always 32 bits on every architecture we support. But on 64-bit Lisps, if a 64 bit value is returned, the upper 32 bits are lost. If the value was not correctly sign-extended by the foreign code, a negative value in the foreign code could be seen by Lisp as a large positive value. Also, the :long or :unsigned-long type is an inadequate specfication because on Windows long types are always 32 bits. So on either 32 or 64 bit lisps, for portability, use :returning :unsigned-nat when the return value is some kind of pointer. (See The :nat and :unsigned-nat types in implementation.htm for a description of the :nat and :unsigned-nat types.) When the returned value is an integer value, be sure to use the correct type and be sure that the foreign code actually produces that type.

Note 8: Returning booleans

def-foreign-call can accept :returning :boolean as a return type, and it will be automatically translated to a canonical form which is :returning (:int boolean). This works with some foreign languages (where 0 is considered false and anything else is considered true - it translates 0 into nil on the lisp side and anything else to the value t), but in some cases (C++, for instance) the boolean type is equivalent to signed-char type, and so it must be specified in the def-foreign-call as :returning (:char boolean).

Examples:

(def-foreign-call add2 (x y))

The symbol add2 will have a function definition calling the foreign function probably named "add2" in C, whose first arg is named "x" and is an integer in Lisp and which is converted to an int for passing to C. If the integer is larger than can be held in a C int, it is truncated. As with the first arg, the second arg named "y" is an integer converted to a C int. The return value is interpreted as a C int type, and is converted to a Lisp integer (which may either be a fixnum or consed as a bignum).

We say the foreign function is "probably" named "add2" because since no specific name or conversion function is specified, the default system conversion function is used. It depends on the platform and platform-specific rules but typically downcases the symbol name.

(def-foreign-call t_double ((x :double)
                            (y :double single-float)
                            (z :int fixnum))
  :returning :double))

Call a function, probably named "t_double" in C (again "probably" because the actual name depends on platform-dependent defaults), whose first arg is a double-float both in Lisp and in C, and whose second arg is a single-float in Lisp but is converted to double float for passing into C (this is the calling convention used by some non ANSI C compilers and by others when the arguments are not prototyped), and the third argument is a fixnum Lisp passed as an integer to C. The function returns and boxes a double-float value to Lisp.

(def-foreign-call (t-float dash-to-underscore) ((x :double)
                                                (y (:float :no-proto))
                                                (z :int fixnum)
                                                (w (* :char) string))
  :returning #-(or (and sun4 (not svr4)) sun3q) :float
             #+(or (and sun4 (not svr4)) sun3q) (:double single-float)

(def-foreign-call (t-float "t_float") ((x :double)
                                                (y (:float :no-proto))
                                                (z :int fixnum)
                                                (w (* :char) string))
  :returning #-(or (and sun4 (not svr4)) sun3q) :float
             #+(or (and sun4 (not svr4)) sun3q) (:double single-float)

These two examples do the same thing: call a function, named "t_float" in C (assuming in the first case proper conversion by dash-to-underscore, which must already be defined and should downcase the symbol name and replace dashes with underscores), whose first arg is a double-float both in Lisp and in C. Like the previous example, the second arg is a float in Lisp, and is converted to double float for passing into C. The third arg named "z" is a fixnum passed as an int, and "w" is a (null-terminated) Lisp string, whose first-character-address is passed to C (beware, the string may move if a gc is allowed). Depending on the architecture, the C function will return either a double (from older C compilers) or a float, each interpreted and boxed as a Lisp single-float value.

We give both examples to show how a lisp name (the symbol t-float) is converted to a foreign name ("t_float"). You can either specify a function that takes a symbol as an argument and returns the correct string (so (dash-to-underscore 't-float) returns "t_float") or you can simply specify the correct string. Note again that dash-to-underscore must be already defined when the def-foreign-call form is evaluated.

Macroexpansion of a def-foreign-call-form now provides more useful information

def-foreign-call has enhanced the macroexpansion to give useful information for users. Here is an example:

cl-user(11): (pprint 
               (macroexpand '(ff:def-foreign-call foo (x (y (* :char))))))

Warning: A runtime with-native-string call is being generated for argument `y'
         to the foreign-function `foo'.  The with-native-string macro can be
         used for explicit string conversions around the foreign calls.  This
         warning is suppressed when :strings-convert is specified in the
         def-foreign-call.

(progn (eval-when (:compile-toplevel)
         (excl::check-lock-definitions-compile-time 'foo 'function
           'foreign-functions:def-foreign-call (fboundp 'foo))
         (push 'foo excl::.functions-defined.))
       (eval-when (compile load eval) (remprop 'foo 'system::direct-ff-call))
       (setf (fdefinition 'foo)
             (let ((excl::f
                    (named-function foo
                      (lambda (x y)
                        (excl::check-args '((:int (integer * *))
                                            ((* :char) (array character (*))))
                                          'foo x y)
                        (with-native-string (#:g34798
                                             (if*
                                              (stringp y)
                                                then y
                                                else ""))
                                            (unless
                                             (stringp y)
                                             (setq #:g34798 y))
                                            (symbol-macrolet
                                             ((y #:g34798))
                                             (system::ff-funcall
                                              (load-time-value
                                               (excl::determine-foreign-address
                                                '("foo" :language :c)
                                                2
                                                nil))
                                              '(:int (integer * *))
                                              x
                                              '((* :char)
                                                (array character (*)))
                                              y
                                              '(:int (integer * *)))))))))
               (excl::set-func_name excl::f 'foo)
               excl::f))
       (record-source-file 'foo) 'foo)
cl-user(12): 

The effects of the new, longer array implementation on def-foreign-call

The new array implementation is discussed in the Arrays and short arrays section in implementation.htm. In brief, standard Common Lisp arrays now can be significantly larger than in earlier releases, while the new short arrays implement the old arrays (the same size limitations but also the same type codes and structure).

In this discussion, `array' refers to the newly-implemented arrays, while `short array' refers to the old implementation, preserved as short arrays in 7.0.

Foreign calls are made with arrays as arguments by passing the address of the first value. In the new implementation, simple-arrays always have exactly the same first element offset (but some short arrays are aligned to the next higher word boundary so that the elements within are naturally aligned). This sometimes-difference between arrays and short-arrays poses an extra burden on the ff interface, in that the arrays must be distinguished between themselves at runtime.

It is now possible to declare an argument a (short-simple-array ... (*)) and the interface will generate code as it did before for normal arrays, passing the address of the first argument.

For 7.0, a declaration of (simple-array ... (*)) actually generates code that tests at runtime whether the argument is a short array or a normal array. So in effect, a short simple-array passed in as if it were a normal simple-array will be properly passed.

Note that with this setup, if argument checking is specified and a short-array is passed in, the check will fail, because a short-array is not a subtype of simple-array. But for 7.0, if you suppress this argument checking, the interface will pass either array correctly.

However, programmers are urged to provide correct declarations and pass the correct type of array even though 7.0 allows sloppiness. In a future release, we anticipate adding a (dual-simple-array ... (*)) declaration to the direct-call foreign interface, and to move that functionality from its current place in the (simple-array ... (*)) declaration. This would then mean that the (simple-array ... (*)) declaration would only pass simple-arrays, and not short simple-arrays. This change will also make more consistent the arg-checking feature of the direct-call interface, since the dual-simple-array type is defined as an or of simple-array and short-simple-array, so a short-simple-array being passed will pass the test of being a dual-simple-array.

See ftype.htm for information on foreign types in Allegro CL and foreign-functions.htm for general information on foreign functions in Allegro CL. See particularly def-foreign-call in foreign-functions.htm.

Copyright (c) 1998-2009, Franz Inc. Oakland, CA., USA. All rights reserved.
Documentation for Allegro CL version 8.1. This page was not revised from the 8.0 page.
Created 2009.10.15.