First, each platform-independent module has zero or one build directory per architecture. An architecture in this case can be a machine (sun4u, i86pc) or a processor architecture (intel, sparc). The path to this location is always usr/src/uts/<platform>/<module>. The module name in this case is what's found in /kernel/drv or a similar location, and in the case of device drivers or STREAMS modules should always match the name of the man page describing that driver or module.

The only files normally present in this directory for a clean tree are makefiles. After a build, these directories contain one or more of obj32, obj64, debug32, and debug64 directories. These directories contain the object files and finally linked modules that are later installed into the kernel directory and other locations in the prototype.

"Implementation architecture"-independent modules are produced in individual directories (one per module) under the "instruction-set architecture" directory (i.e.: sparc). Similarly, "implementation architecture"-dependent modules are produced in individual directories under the "implementation architecture" directory (i.e.: sun4, sun4u, i86pc).

Platform-dependent modules (including "unix") may be built more than once in different locations. Platform-dependent modules are discussed in greater detail in <section>.

The sources are not contained in these build directories.

Most libraries and some commands and daemons are built using makefiles very similar to those used to build the kernel. Accordingly, intermediate objects, shared objects, libraries, and executables are built by small makefile fragments and placed in dedicated ISA-specific subdirectories.

Other commands' build systems place objects in the same directories as the sources. See sections 3.2.2.3, 3.2.3.2, and 3.2.5 for more information on how commands are built.

Most of the cmd tree is directly based on the original System V Release 4 source, which uses strategy (b) described above. Since most commands and daemons do not need to provide both 32- and 64-bit versions, or do anything special when building for different architectures, this strategy is adequate and appropriate for most commands and has been applied even to new subdirectories. In situations in which architecture-dependent build options are needed or multiple ISA versions of a program must be delivered, this strategy is unworkable and the more general approach of multiple per-ISA object directories must be used instead. This latter approach is similar to the approach used for kernel modules.

The lib hierarchy is somewhat simpler; nearly all subdirectories must use per-ISA object file locations and makefiles.

A few directories do not appear to follow any rule or pattern, such as cmd/cmd-inet and cmd/agents. These are primarily historical artifacts of Sun's internal project organization.

The driving makefile for any module is located in the leaf directory (build directory) where the module and its component objects are built. After a 'make clobber' operation, the makefile should be the only file remaining in that directory. There are two other types of makefiles in the tree: suffixed and non-suffixed. Common definitions and rules needed by all leaf makefiles are contained in the suffixed makefiles; these are included by leaf makefiles. Non-suffixed makefiles generally invoke multiple lower-level makefiles with the same target so that many modules can be built with a single make invocation.

Issuing the command 'make' in the uts directory will cause all supported, modularized kernels and modules to be built.

Issuing the command 'make' in a uts/ARCHITECTURE directory (i.e.: uts/sparc) will cause all supported, "implementation architecture"-independent modules for ARCHITECTURE to be built.

Issuing the command 'make' in a uts/MACHINE directory (i.e.: uts/sun4u) will cause that kernel and all supported, "implementation architecture"- dependent modules for MACHINE to be built.

The makefiles are verbosely commented. It is desired that they should stay this way.

Most command and daemon subdirectories follow one of two general layout rules, depending on where object files will be located (see sections 3.2.2.2 and 3.2.2.3). For ISA-dependent programs, the layout is similar to that used by the kernel. Programs which do not need ISA-dependent build behavior use a simplified makefile layout. In the description here, we use the example of a generic command called "foocmd" whose sources are located in usr/src/cmd/foocmd. The makefiles relevant to building foocmd are:

Most library subdirectories follow the same general layout, which is similar to the command layout. Unlike commands, most libraries are built for both 32- and 64-bit architecture variants, so ISA-specific directories will almost always be present. Therefore, the overall build structure for each library is similar to that of the kernel. See 3.3.3.3 Adding New Libraries for more detailed information on library makefiles.

There are two parts to the make dependencies: a set of targets and a set of rules.

The rules are contained in the Makefile.rules files in each directory under uts. They cover how to compile a source file in some other directory (common/io for example) into an object file in the build directory described above. They also describe which sources are needed to build a given object file. The order in which they are specified determines which source file is needed if there are multiple source files matching the same basename. The key to all this is understanding that all rules and targets are static and module-independent, but variables set in each module's build directory (leaf) makefile define the module-specific build information.

The targets are of two kinds: the first are "make targets," generic directives you can give make like "all" or "install" which are executed in each build directory. The build directory makefiles in turn include other makefiles that contain rules for building objects and modules. The second type of target is more interesting - they are the actual module names (directories, in fact) that must be built. These variables are called KMODS and XMODS.

If KMODS contains, for example, "aac ata asy", then for each of the aac, ata, and asy build directories, the make system will cd to that directory and rerun make with whatever target (all, install, etc) is given. The rules for that module are derived from all the included makefiles; the crucial variables to look for are of the form XXX_OBJS, where XXX is the module name. More than one makefile may contain definitions adding to each of these object lists, because building a module may require different objects on each platform.

XMODS is used only for building export versions of the commercial Solaris product; it is not used in building ON.

So for example we see:

common/Makefile.files:ASY_OBJS +=      asy.o

This indicates that the asy.o object is needed to build the ASY module. The makefile in the <platform>/asy directory will reference ASY_OBJS and thus know what to build. So in intel/asy/Makefile we see:

MODULE          = asy
OBJECTS         = $(ASY_OBJS:%=$(OBJS_DIR)/%)

So, what good is this? Plenty. Because our ALL_TARGET is also defined:

ALL_TARGET      = $(BINARY) $(SRC_CONFILE)

Makefile.uts defines BINARY as follows:

BINARY             = $(OBJS_DIR)/$(MODULE)

And OBJS_DIR is the individual module build directory I described at the beginning. So, make knows we need to build usr/src/uts/intel/asy/obj32/asy. Makefile.targ defines the rules for doing so, too:

$(BINARY):              $(OBJECTS)
	$(LD) -r $(LDFLAGS) -o $@ $(OBJECTS)
	$(CTFMERGE_UNIQUIFY_AGAINST_GENUNIX)
	$(POST_PROCESS)
	$(ELFSIGN_MOD)

We're almost there - OBJECTS have to come from somewhere. We find this defined in intel/asy/Makefile:

OBJECTS         = $(ASY_OBJS:%=$(OBJS_DIR)/%)

And that's that. We have a module (asy) that has a BINARY (obj32/asy) which depends on OBJECTS made from ASY_OBJS. The rules for each entry in ASY_OBJS is in one of the Makefile.rules files; they look something like this:

$(OBJS_DIR)/%.o:                $(UTSBASE)/common/io/%.c
	$(COMPILE.c) -o $@ $<
	$(CTFCONVERT_O)

Each object is built in accordance with these rules. The rule for $(BINARY) then links them into the module.

We can now put it all together. The extensive use of macros and local per-module definitions provides much greater flexibility, but in the example given, the equivalent makefile fragment would look very familiar:

KMODS +=	asy

all:	$(KMODS)

asy:	obj32/asy.o
	$(LD) -r $(LDFLAGS) -o $@ obj32/asy.o
	...

obj32/asy.o:	../../common/io/asy.c
	$(CC) $(CFLAGS) $(CPPFLAGS) -c $< -o $@
	...

Command makefiles vary widely, as described above. Therefore it is impossible to provide a "representative example" of a command makefile. However, understanding the material in sections 3.2.4.1 and 3.3.3.3 will provide you with the necessary knowledge to work with command makefiles. As always, if you are looking for information about a specific command, it is usually best to use the mailing lists and other community resources.

Section 3.3.3.3 describes library makefile operation in detail.

Commands (normally executable programs or daemons) usually do not depend on the ISA on which they are executed; that is, they will be built and installed in the same way regardless of the ISA they are built for. Some commands, however, mainly those which operate on other processes or manipulate binary objects, must behave differently depending on the ISA(s) supported by the system and/or their targets. The /usr/lib/isaexec wrapper program provides this behavior in the following way:

First, one or more ISA-specific programs are installed into ISA-specific directories in the system. These are subdirectories of those directories which ordinarily contain executable programs, such as /usr/bin. On SPARC systems, ISA subdirectories might include /usr/bin/sparcv7 and /usr/bin/sparcv9 to contain 32- and 64-bit binaries, respectively. Other ISAs may be supported by other systems; you can find out which your system supports by using isalist(1).

Next, a hard link is installed into the ordinary binary directory (in the example above, /usr/bin) targeting /usr/lib/isaexec. This will cause isaexec to be invoked with the name of the ISA-dependent program the user has executed.

When this happens, isaexec will select the most appropriate ISA-specific program installed in the first step and exec(2) it. This is transparent, so a user will not notice that a separate program has been invoked.

A similar mechanism (/usr/lib/platexec) exists for platform-specific programs; these are much less common even than ISA-specific programs.

Because it needs to interact both with user processes and kernel data structures, dtrace(1) is an example of a command that must provide multiple ISA-specific versions.

cmd/dtrace contains two makefiles: Makefile and Makefile.com. It also contains a single source file, dtrace.c, and four ISA directories: amd64, i386, sparc, and sparcv9. Each contains a single makefile. The top-level makefiles should look very familiar after our tour of the uts makefiles; Makefile is mainly boilerplate, customized only to specify that the program produced is ISA- specific and therefore a link to the isaexec program needs to be installed into /usr/sbin in the proto area:

install:        $(SUBDIRS)
	-$(RM) $(ROOTUSRSBINPROG)
	-$(LN) $(ISAEXEC) $(ROOTUSRSBINPROG)

Otherwise, it simply leaves all building and installation to the makefiles in the ISA subdirectories. Makefile.com is also boilerplate; it specifies the name of the program to build and the objects from which it must be built:

PROG = dtrace
OBJS = dtrace.o

It then transforms the objects into sources:

SRCS = $(OBJS:%.o=../%.c)

and includes the generic top-level Makefile.cmd fragment to pick up build rules.

Compilation and linkage flags are specified:

CFLAGS += $(CCVERBOSE)
CFLAGS64 += $(CCVERBOSE)
LDLIBS += -ldtrace -lproc

and the remainder consists of a handful of generic build rules. One feature is noteworthy: an extra ../ is prepended to all paths in this fragment. This is because it is always included by makefiles interpreted from the ISA-specific build directory, which is one level deeper than Makefile.com itself.

The makefiles in the ISA-specific build directories are even simpler; they each consist of only two lines:

include ../Makefile.com

install: all $(ROOTUSRSBINPROG32)

The first line picks up all common definitions in the makefile we just examined, and the second specifies (indirectly) the location in which the program is to be installed. To complete the picture, we need to briefly examine usr/src/cmd/Makefile.cmd, which defines the various locations in which programs can be installed. Since we have not defined ROOTUSRSBINPROG32, it must be defined there, and indeed it is:

ROOTUSRSBIN=    $(ROOT)/usr/sbin
ROOTUSRSBINPROG=$(PROG:%=$(ROOTUSRSBIN)/%)
ROOTUSRSBIN32=  $(ROOTUSRSBIN)/$(MACH32)
ROOTUSRSBINPROG32=      $(PROG:%=$(ROOTUSRSBIN32)/%)

Note that the program is actually installed in /usr/sbin/$(MACH32), which is not normally in a user's search path. But remember that we also created a link from $(ROOTUSRSBINPROG) to $(ISAEXEC); expanding these variables shows that we will actually install two separate files:

$(ROOT)/usr/sbin/dtrace		-> $(ROOT)/usr/lib/isaexec
$(ROOT)/usr/sbin/$(MACH32)/dtrace	(our executable program)

Since the isaexec program is responsible for selecting an ISA-specific program from among several options depending on the system's supported instruction sets, we can see that a typical system will have one or more dtrace(1) binaries installed and that when dtrace(1) is run by the user, isaexec will select the appropriate one from those installed in the ISA-specific subdirectories of /usr/sbin. As we might expect, other ISA-specific makefiles under cmd/dtrace install their binaries into ROOTUSRSBIN64, also defined in usr/src/cmd/Makefile.cmd.

Libraries, unlike commands, must almost always be built for all supported ISAs and have separate ISA-specific versions installed on, at minimum, every system that supports that ISA. In practice, libraries for each ISA in the system's processor family are installed. This allows developers on that system to write programs using any supported ISA and link them with the appropriate system libraries (it is not possible, for example, to link 32-bit object files and 64-bit libraries, nor vice versa).

Libraries use a build system very similar to that used by commands which must provide multiple ISA-specific executable programs. We examine here the makefiles used by dtrace(1)'s library counterpart libdtrace(3LIB) and how they support building and installing multiple ISA-specific versions of the dtrace library.

lib/libdtrace contains two makefiles: Makefile and Makefile.com. It also contains a directory for common code and four ISA directories: amd64, i386, sparc, and sparcv9. Each ISA directory contains a single makefile and in some cases some ISA-specific sources. The top-level Makefile should once again look very familiar; is very similar to the uts and cmd makefiles and is mainly boilerplate. There are a few customizations to add a yydebug target and to tell top-level makefiles that lex and yacc generated files should not have cross-references built against them.

yydebug := TARGET = yydebug
...
lint yydebug: $(SUBDIRS)

and

XRDIRS = common i386 sparc sparcv9
XRDEL = dt_lex.c dt_grammar.c Makefile*

Otherwise, it simply leaves all building and installation to the makefiles in the ISA subdirectories. Makefile.com is somewhat more interesting. It first specifies the name of the library to build (in static format; shared library names are translated as we shall see below), and the library version. These are common to all library makefiles. Next, we see long lists of source files used to build the library, and an OBJECTS specification. This particular library also has some unique characteristics, including the D runtime init code, built as a separate object file (drti.o), and a number of D "header" files which are installed into a separate directory /usr/lib/dtrace. You can see these unique features here:

DRTISRC = drti.c
DRTIOBJ = $(DRTISRC:%.c=%.o)

DLIBSRCS += \
	errno.d \
	io.d \
	procfs.d \
	regs.d \
	sched.d \
	signal.d \
	unistd.d
...
ROOTDLIBDIR = $(ROOT)/usr/lib/dtrace
ROOTDLIBDIR64 = $(ROOT)/usr/lib/dtrace/64
...
$(ROOTDLIBDIR):
	$(INS.dir)

$(ROOTDLIBDIR64): $(ROOTDLIBDIR)
	$(INS.dir)

$(ROOTDLIBDIR)/%.d: ../common/%.d
	$(INS.file)

$(ROOTDLIBDIR)/%.d: ../$(MACH)/%.d
	$(INS.file)

$(ROOTDLIBDIR)/%.d: %.d
	$(INS.file)

Note that some of the D "headers" are in turn generated from other files; this makefile includes rules for performing these transformations.

Once the LIBSRCS and OBJECTS have been defined, the generic library makefile fragment Makefile.lib is included to provide target and directory definitions.

Binary objects are built in the pics/ subdirectory. pics stands for position- independent code(s), referring to the fact that dynamic shared libraries are built in such a way as to be loaded at any address. This is an historical artifact; static linking of system libraries is not supported by OpenSolaris-based distributions, so the non-PIC objects which would have been used to build an archive-style library are not built. The rules for building are given as:

pics/%.o: ../$(MACH)/%.c 
	$(COMPILE.c) -o $@ $<
	$(POST_PROCESS_O)

pics/%.o: ../$(MACH)/%.s
	$(COMPILE.s) -o $@ $<
	$(POST_PROCESS_O)

%.o: ../common/%.c
	$(COMPILE.c) -o $@ $<
	$(POST_PROCESS_O)

Note that because this makefile fragment is included by each ISA's makefile, the objects produced will be placed relative to that makefile, in each ISA directory. Once again, this also means that included makefile fragments include an additional ../ component.

The libdtrace Makefile.com also contains a large number of custom rules to accommodate building a lexer and parser from lex(1) and yacc(1) definitions; these sources are then compiled specially using the custom rules and are removed when the clean or clobber target is executed.

The makefiles in the ISA-specific build directories are once again very simple. The SPARC-specific makefile, for example, defines additional assembler flags, specifies the directory where the library interface map is located, and then includes the common makefile:

ASFLAGS += -D_ASM -K PIC -P

MAPDIR = ../spec/sparc
include ../Makefile.com

The makefile may also add a few ISA-specific sources to the build list; for example, sparc/Makefile includes:

SRCS += dt_asmsubr.s
OBJECTS += dt_asmsubr.o

And once again, a separate install target is provided to indicate the directories where these specific libraries and headers should be installed:

install yydebug: all $(ROOTLIBS) $(ROOTLINKS) $(ROOTLINT) \
	$(ROOTDLIBS) $(ROOTDOBJS)

To support building a separate set of 64-bit objects, sparcv9/Makefile includes slightly different definitions and a different install target:

MAPDIR = ../spec/sparcv9
include ../Makefile.com
include ../../Makefile.lib.64

CPPFLAGS += -D_ELF64
...
install yydebug: all $(ROOTLIBS64) $(ROOTLINKS64) $(ROOTLINT64) \
	$(ROOTDLIBS) $(ROOTDOBJS64)

Note the additional inclusion of Makefile.lib.64 and the additional flag for 64-bit ELF targets. Also, the 64-bit-specific versions of the installation files are used, which as we can see in Makefile.lib cause them to be installed into /usr/lib/sparcv9:

ROOTLIBDIR=     $(ROOT)/usr/lib
ROOTLIBDIR64=   $(ROOT)/usr/lib/$(MACH64)
...
ROOTLIBS=       $(LIBS:%=$(ROOTLIBDIR)/%)
ROOTLIBS64=     $(LIBS:%=$(ROOTLIBDIR64)/%)

The linker (ld(1)) and runtime loader (ld.so.1(1)) know which of these directories to search for libraries depending on the bitness of the objects being linked or the program being run, respectively. For more information about 32- versus 64-bit linking, consult the Solaris Linkers and Libraries Guide, which can be found at http://docs.sun.com/db/doc/816-1386.