The support for USB testing provided by the eCos USB common slave
package is somewhat different in nature from the kind of testing used
in many other packages. One obvious problem is that USB tests cannot
be run on just a bare target platform: instead the target platform
must be connected to a suitable USB host machine, and that host
machine must be running appropriate software for the test code to
interact with. This is very different from say a kernel test which
typically will have no external dependencies. Another important
difference between USB testing and say a C library
strcmp test is sensitivity to timing and to
hardware boundary conditions: although a simple test case that just
performs a small number of USB transfers is better than no testing at
all, it should also be possible to run tests for hours or days on end,
under a variety of loads. In order to provide the required
functionality the basic architecture of the USB testing support is as
follows:
There is a single target-side program usbtarget. By default when this is run on a target platform it will appear to do nothing. In fact it is waiting to be contacted by another program usbhost which will tell it what test or tests to run. usbtarget provides mechanisms for running a wide range of tests.
usbtarget is a generic program, but USB
testing depends to some extent on the functionality provided by the
hardware. For example there is no point in testing bulk transmits
to endpoint 12 if the target hardware does not support an endpoint
12. Therefore each USB device driver should supply information about
what the hardware is actually capable of, in the form of an array of
usbs_testing_endpoint data structures.
There is a single host-side program usbhost, which acts as a counterpart to usbtarget. Again usbhost has no built-in knowledge of the test or tests that are supposed to run, it only provides mechanisms for running a wide range of tests. On start-up usbhost will search the USB bus for hardware running the target-side program, specifically a USB device that identifies itself as the product "Red Hat eCos USB test".
usbhost contains a Tcl interpreter, and will execute any Tcl scripts specified on the command line together with appropriate arguments. The Tcl interpreter has been extended with various commands such as usbtest::bulktest, so the script can perform the desired test or tests.
Adding a new test simply involves writing a short Tcl script that invokes the appropriate USB-specific commands. Running multiple tests involves passing appropriate arguments to usbhost, or alternatively writing a single script that just invokes other scripts.
The current implementation of usbhost
depends heavily on functionality provided by the Linux kernel and in
particular the usbdevfs support. It uses
/proc/bus/usb/devices to find out what devices
are attached to the bus, and will then access the device by opening
/proc/bus/usb/xxx/yyy and performing
ioctl operations. This allows USB testing to take
place without having to write a new host-side device driver, but
getting the code working on host machines not running Linux would
obviously be problematical.
The target-side component of the USB testing software consists of a single program usbtarget which contains support for a range of different tests, under the control of host-side software. This program is not built by default alongside other eCos test cases since it will only operate in certain environments, specifically when the target board's connector is plugged into a Linux host, and when the appropriate host-side software has been installed on that host. Instead the user must enable a configuration option CYGBLD_IO_USB_SLAVE_USBTEST to add the program to the list of tests for the current configuration.
Starting the usbtarget program does not require anything unusual, so it can be run in a normal gdb session just like any eCos application. After initialization the program will wait for activity from the host. Depending on the hardware, the Linux host will detect that a new USB peripheral is present on the bus either when the usbtarget initialization is complete or when the cable between target and host is connected. The host will perform the normal USB enumeration sequence and discover that the peripheral does not match any known vendor or product id and that there is no device driver for "Red Hat eCos USB test", so it will ignore the peripheral. When the usbhost program is run on the host it will connect to the target-side software, and testing can now commence.
Note: In theory the host-side software should be built when the package is installed in the component repository, and removed when a package is uninstalled. The current eCos administration tool does not provide this functionality.
The host-side software should be built via the usual sequence of "configure/make/make install". It can only be built on a Linux host and the configure script contains an explicit test for this. Because the eCos component repository should generally be treated as a read-only resource the configure script will also prevent you from trying to build inside the source tree. Instead a separate build tree is required. Hence a typical sequence for building the host-side software would be as follows:
$ mkdir usbhost_build $ cd usbhost_build $ <repo>packages/io/usb/slave/current/host/configure (1) (2) <args> (3) $ make <output from make> $ su (4) $ make install <output from make install> $ |
--prefix= to specify where the executables
and support files should be installed. The only other parameter that
some users may wish to specify is the location of a suitable Tcl
installation. By default usbhost will use
the existing Tcl installation in /usr,
as provided by your Linux distribution. An alternative Tcl
installation can be specified using the parameter
--with-tcl=, or alternatively using some
combination of --with-tcl-include,
--with-tcl-lib and
--with-tcl-version. During make install the following actions will take place:
usbhost will be installed in /usr/local/bin,
or some other bin directory if
the default location is changed at configure-time using a
--prefix= or similar option. It will be
installed as the executable
usbhost_<version>, for example
usbhost_current, thus allowing several
releases of the USB slave package to co-exist. For convenience a
symbolic link from usbhost to this executable
will be created, so users can just run usbhost to
access the most recently-installed version.
usbchmod will be installed in /usr/local/libexec/ecos/io_usb_slave_<version>. This program should only be run by usbhost, not invoked directly, so it is not placed in the bin directory. Again the presence of the package version in the directory name allows multiple releases of the package to co-exist.
A Tcl script usbhost.tcl will get installed in the same directory as usbchmod. This Tcl script is loaded automatically by the usbhost executable.
A number of additional Tcl scripts, for example list.tcl will get installed alongside usbhost.tcl. These correspond to various test cases provided as standard. If a given test case is specified on the command line and cannot be found relative to the current directory then usbhost will search the install directory for these test cases.
Note: Strictly speaking installing the usbhost.tcl and other Tcl scripts below the libexec directory deviates from standard practice: they are architecture-independent data files so should be installed below the share subdirectory. In practice the files are sufficiently small that there is no point in sharing them, and keeping them below libexec simplifies the host-side software somewhat.
The usbhost should be run only when there is a suitable target attached to the USB bus and running the usbtarget program. It will search /proc/bus/usb/devices for an entry corresponding to this program, invoke usbchmod if necessary to change the access rights, and then interact with usbtarget over the USB bus. usbhost should be invoked as follows:
$ usbhost [-v|--version] [-h|--help] [-V|--verbose] <test> [<test parameters>] |
The -v or --version
option will display version information for
usbhost including the version of the USB
slave package that was used to build the executable.
The -h or --help option
will display usage information.
The -V or --verbose
option can be used to obtain more information at run-time, for example
some output for every USB transfer. This option can be repeated
multiple times to increase the amount of output.
The first argument that does not begin with a hyphen specifies a test
that should be run, in the form of a Tcl script. For example an
argument of list.tcl will cause
usbhost to look for a script with that
name, adding a .tcl suffix if necessarary, and
run that script. usbhost will look in the
current directory first, then in the install tree for standard test
scripts provided by the USB slave package.
Some test scripts may want their own parameters, for example a duration in seconds. These can be passed on the command line after the name of the test, for example usbhost mytest 60.
Each test is defined by a Tcl script, running inside an interpreter
provided by usbhost. In addition to the
normal Tcl functionality this interpreter provides a number of
variables and functions related to USB testing. For example there is a
variable bulk_in_endpoints that lists all the
endpoints on the target that can perform bulk IN operations, and a
related array bulk_in which contains information
such as the minimum and maximum packets sizes. There is a function
bulktest which can be used to perform bulk tests
on a particular endpoint. A simple test script aimed at specific
hardware could ignore the information variables since it would know
exactly what USB hardware is available on the target, whereas a
general-purpose script would use the information to adapt to the
hardware capabilities.
To avoid namespace pollution all USB-related Tcl variables and
functions live in the usbtest:: namespace.
Therefore accessing requires either explicitly including the
namespace any references, for example
$usbtest::bulk_in_endpoints, or by using Tcl's
namespace import facility.
A very simple test script might look like this:
usbtest::bulktest 1 out 4000
usbtest::bulktest 2 in 4000
if { [usbtest::start 60] } {
puts "Test successful"
} else
puts "Test failed"
foreach result $usbtest::results {
puts $result
}
} |
This would perform a test run involving 4000 bulk transfers from the host to the target's endpoint 1, and concurrently 4000 bulk transfers from endpoint 2. Default settings for packet sizes, contents, and delays would be used. The actual test would not start running until usbtest is invoked, and it is expected that the test would complete within 60 seconds. If any failures occur then they are reported.
Each target-side USB device driver provides information about the actual capabilities of the hardware, for example which endpoints are available. Strictly speaking it provides information about what is actually supported by the device driver, which may be a subset of what the hardware is capable of. For example, the hardware may support isochronous transfers on a particular endpoint but if there is no software support for this in the driver then this endpoint will not be listed. When usbhost first contacts the usbtarget program running on the target platform, it obtains this information and makes it available to test scripts via Tcl variables:
bulk_in_endpointsThis is a simple list of the endpoints which can support bulk IN transfers. For example if the target-side hardware supports these transfers on endpoints 3 and 5 then the value would be "3 5" Typical test scripts would iterate over the list using something like:
if { 0 != [llength $usbtest::bulk_in_endpoints] } {
puts"Bulk IN endpoints: $usbtest::bulk_in_endpoints"
foreach endpoint $usbtest:bulk_in_endpoints {
…
}
}
|
bulk_in()This array holds additional information about each bulk IN endpoint. The array is indexed by two fields, the endpoint number and one of min_size, max_size, max_in_padding and devtab:
This field specifies a lower bound on the size of bulk transfers, and will typically will have a value of 1.
Note: The typical minimum transfer size of a single byte is not strictly speaking correct, since under some circumstances it can make sense to have a transfer size of zero bytes. However current target-side device drivers interpret a request to transfer zero bytes as a way for higher-level code to determine whether or not an endpoint is stalled, so it is not actually possible to perform zero-byte transfers. This issue will be addressed at some future point.
This field specifies an upper bound on the size of bulk transfers. Some target-side drivers may be limited to transfers of say 0x0FFFF bytes because of hardware limitations. In practice the transfer size is likely to be limited primarily to limit memory consumption of the test code on the target hardware, and to ensure that tests complete reasonably quickly. At the time of writing transfers are limited to 4K.
On some hardware it may be necessary for the target-side device driver to send more data than is actually intended. For example the SA11x0 USB hardware cannot perform bulk transfers that are an exact multiple of 64 bytes, instead it must pad such transfers with an extra byte and the host must be ready to accept and discard this byte. The max_in_padding field indicates the amount of padding that is required. The low-level code inside usbhost will use this field automatically, and there is no need for test scripts to adjust packet sizes for padding. The field is provided for informational purposes only.
This is a string indicating whether or not the
target-side USB device driver supports access to this endpoint
via entries in the device table, in other words through
conventional calls like open and
write. Some device drivers may only
support low-level USB access because typically that is what gets
used by USB class-specific packages such as USB-ethernet.
An empty string indicates that no devtab entry is available,
otherwise it will be something like
"/dev/usbs2w".
Typical test scripts would access this data using something like:
foreach endpoint $usbtest:bulk_in_endpoints {
puts "Endpoint $endpoint: "
puts " minimum transfer size $usbtest::bulk_in($endpoint,min_size)"
puts " maximum transfer size $usbtest::bulk_in($endpoint,max_size)"
if { 0 == $usbtest::bulk_in($endpoint,max_in_padding) } {
puts " no IN padding required"
} else {
puts " $usbtest::bulk_in($endpoint,max_in_padding) bytes of IN padding required"
}
if { "" == $usbtest::bulk_in($endpoint,devtab) } {
puts " no devtab entry provided"
} else {
puts " corresponding devtab entry is $usbtest::bulk_in($endpoint,devtab)"
}
}
|
bulk_out_endpoint This is a simple list of the endpoints which can support bulk OUT
transfers. It is analogous to
bulk_in_endpoints.
bulk_out() This array holds additional information about each bulk OUT
endpoint. It can be accessed in the same way as
bulk_in(), except that there is no
max_in_padding field because that field only
makes sense for IN transfers.
control() This array holds information about the control endpoint. It contains
two fields, min_size and
max_size. Note that there is no variable
control_endpoints because a USB target always
supports a single control endpoint 0. Similarly
the control array does not use an endpoint number
as the first index because that would be redundant.
isochronous_in_endpoints and
isochronous_in() These variables provide the same information as
bulk_in_endpoints and bulk_in,
but for endpoints that support isochronous IN transfers.
isochronous_out_endpoints and
isochronous_out() These variables provide the same information as
bulk_out_endpoints and bulk_out,
but for endpoints that support isochronous OUT transfers.
interrupt_in_endpoints and
interrupt_in() These variables provide the same information as
bulk_in_endpoints and bulk_in,
but for endpoints that support interrupt IN transfers.
interrupt_out_endpoints and
interrupt_out() These variables provide the same information as
bulk_out_endpoints and bulk_out,
but for endpoints that support interrupt OUT transfers.
The main function for initiating a bulk test is
usbtest::bulktest. This takes three compulsory
arguments, and can be given a number of additional arguments to
control the exact behaviour. The compulsory arguments are:
This specifies the endpoint to use. It should correspond to
one of the entries in
usbtest::bulk_in_endpoints or
usbtest::bulk_out_endpoints, depending on the
transfer direction.
This should be either in or out.
This specifies the number of transfers that should take place. The testing software does not currently support the concept of performing transfers for a given period of time because synchronising this on both the host and a wide range of targets is difficult. However it is relatively easy to work out the approximate time a number of bulk transfers should take place, based on a typical bandwidth of 1MB/second and assuming say a 1ms overhead per transfer. Alternatively a test script could perform a small initial run to determine what performance can actually be expected from a given target, and then use this information to run a much longer test.
Additional arguments can be used to control the exact transfer. For
example a txdelay+ argument can be used to
slowly increase the delay between transfers. All such arguments involve
a value which can be passed either as part of the argument itself,
for example txdelay+=5, or as a subsequent
argument, txdelay+ 5. The possible arguments fall
into a number of categories: data, I/O mechanism, transmit size,
receive size, transmit delay, and receive delay.
An obvious parameter to control is the actual data that gets sent.
This can be controlled by the argument data
which can take one of five values: none,
bytefill, intfill,
byteseq and wordseq. The default
value is none.
The transmit code will not attempt to fill the buffer in any way, and the receive code will not check it. The actual data that gets transferred will be whatever happened to be in the buffer before the transfer started.
The entire buffer will be filled with a single byte, as per
memset.
The buffer will be treated as an array of 32-bit integers, and will be filled with the same integer repeated the appropriate number of times. If the buffer size is not a multiple of four bytes then the last few bytes will be set to 0.
The buffer will be filled with a sequence of bytes, generated by
a linear congruential generator. If the first byte in the buffer is
filled with the value x, the next byte will be
(m*x)+i. For example a sequence of slowly
incrementing bytes can be achieved by setting both the multiplier
and the increment to 1. Alternatively a pseudo-random number
sequence can be achieved using values 1103515245 and 12345, as
per the standard C library rand function.
For convenience these two constants are available as Tcl
variables usbtest::MULTIPLIER and
usbtest::INCREMENT.
This acts like byteseq, except that the buffer is treated as an array of 32-bit integers rather than as an array of bytes. If the buffer is not a multiple of four bytes then the last few bytes will be filled with zeroes.
The above requires three additional parameters
data1, data* and
data+. data1 specifies
the value to be used for byte or word fills, or the first number when
calculating a sequence. The default value is 0.
data* and data+ specify
the multiplier and increment for a sequence, and have default values
of 1 and 0 respectively. For
example, to perform a bulk transfer of a pseudo-random sequence of
integers starting with 42 the following code could be used:
bulktest 2 IN 1000 data=wordseq data1=42 \
data* $usbtest::MULTIPLIER data+ $usbtest::INCREMENT |
The above parameters define what data gets transferred for the first
transfer, but a test can involve multiple transfers. The data format
will be the same for all transfers, but it is possible to adjust the
current value, the multiplier, and the increment between each
transfer. This is achieved with parameters data1*,
data1+, data**,
data*+, data+*, and
data++, with default values of 1 for each
multiplier and 0 for each increment. For example, if the multiplier
for the first transfer is set to 2 using
data*, and arguments
data** 2 and data*+ -1 are also
supplied, then the multiplier for subsequent transfers will be
3, 5, 9,
….
Note: Currently it is not possible for a test script to send specific data, for example a specific sequence of bytes captured by a protocol analyser that caused a problem. If the transfer was from host to target then the target would have to know the exact sequence of bytes to expect, which means transferring data over the USB bus when that data is known to have caused problems in the past. Similarly for target to host transfers the target would have to know what bytes to send. A possible future extension of the USB testing support would allow for bounce operations, where a given message is first sent to the target and then sent back to the host, with only the host checking that the data was returned correctly.
On the target side USB transfers can happen using either low-level
USB calls such as usbs_start_rx_buffer, or by
higher-level calls which go through the device table. By default the
target-side code will use the low-level calls. If it is desired to
test the higher-level calls instead, for example because those are
what the application uses, then that can be achieved with an
argument mechanism=devtab.
The next set of arguments can be used to control the size of the
transmitted buffer: txsize1,
txsize>=, txsize<=
txsize*, txsize/,
and txsize+.
txsize1 determines the size of the first
transfer, and has a default value of 32 bytes. The size of the next
transfer is calculated by first multiplying by the
txsize* value, then dividing by the
txsize/ value, and finally adding the
txsize+ value. The defaults for these are
1, 1, and 0
respectively, which means that the transfer size will remain
unchanged. If for example the transfer size should increase by
approximately 50 per cent each time then suitable values might be
txsize* 3, txsize/ 2,
and txsize+ 1.
The txsize>= and
txsize<= arguments can be used to impose
lower and upper bounds on the transfer. By default the
min_size and max_size values
appropriate for the endpoint will be used. If at any time the
current size falls outside the bounds then it will be normalized.
The receive size, in other words the number of bytes that either host
or target will expect to receive as opposed to the number of bytes
that actually get sent, can be adjusted using a similar set of
arguments: rxsize1,
rxsize>=,
rxsize<=,
rxsize*, rxsize/ and
rxsize+. The current receive size will be
adjusted between transfers just like the transmit size. However when
communicating over USB it is not a good idea to attempt to receive
less data than will actually be sent: typically neither the hardware
nor the software will be able to do anything useful with the excess,
so there will be problems. Therefore if at any time the calculated
receive size is less than the transmit size, the actual receive will
be for the exact number of bytes that will get transmitted. However
this will not affect the calculations for the next receive size.
The default values for rxsize1,
rxsize*, rxsize/ and
rxsize+ are 0,
1, 1 and 0
respectively. This means that the calculated receive size will always
be less than the transmit size, so the receive operation will be for
the exact number of bytes transmitted. For some USB protocols this
would not accurately reflect the traffic that will happen. For example
with USB-ethernet transfer sizes will vary between 16 and 1516 bytes,
so the receiver will always expect up to 1516 bytes. This can be
achieved using rxsize1 1516, leaving the
other parameters at their default values.
For target hardware which involves non-zero max_in_padding, on the host side the padding will be added automatically to the receive size if necessary.
Typically during the testing there will be some minor delays between transfers on both host and target. Some of these delays will be caused by timeslicing, for example another process running on the host, or a concurrent test thread running inside the target. Other delays will be caused by the USB bus itself, for example activity from another device on the bus. However it is desirable that test cases be allowed to inject additional and somewhat more controlled delays into the system, for example to make sure that the target behaves correctly even if the target is not yet ready to receive data from the host.
The transmit delay is controlled by six parameters:
txdelay1, txdelay*,
txdelay/, txdelay+,
txdelay>= and
txdelay<=. The default values for these are
0, 1, 1,
0, 0 and
1000000000 respectively, so that by default
transmits will happen as quickly as possible. Delays are measured in
nanoseconds, so a value of 1000000 would correspond
to a delay of 0.001 seconds or one millisecond. By default delays have
an upper bound of one second. Between transfers the transmit delay is
updated in much the same was as the transfer sizes.
The receive delay is controlled by a similar set of six parameters:
rxdelay1, rxdelay*,
rxdelay/, rxdelay+,
rxdelay>= and
rxdelay<=. The default values for these are
the same as for transmit delays.
The transmit delay is used on the side which sends data over the USB bus, so for a bulk IN transfer it is the target that sends data and hence sleeps for the specified transmit delay, while the host receives data sleeps for the receive delay. For an OUT transfer the positions are reversed.
It should be noted that although the delays are measured in nanoseconds, the actual delays will be much less precise and are likely to be of the order of milliseconds. The exact details will depend on the kernel clock speed.
Support for testing other types of USB traffic such as isochronous transfers is not yet implemented.
A USB test script should prepare one or more transfers using
appropriate functions such as usbtest::bulktest.
Once all the individual tests have been prepared they can be started
by a call to usbtest::start. This takes a single
argument, a maximum duration measured in seconds. If all transfers
have not been completed in the specified time then any remaining
transfers will be aborted.
usbtest::start will return 1
if all the tests have succeeded, or 0 if any of
them have failed. More detailed reports will be stored in the
Tcl variable usbtests::results, which will be a
list of string messages.
A number of test scripts are provided as standard. These are located in the host subdirectory of the common USB slave package, and will be installed as part of the process of building the host-side software. When a script is specified on the command line usbhost will first search for it in the current directory, then in the install tree. Standard test scripts include the following:
This script simply displays information about the capabilities of the target platform, as provided by the target-side USB device driver. It can help with tracking down problems, but its primary purpose is to let users check that everything is working correctly: if running usbhost list.tcl outputs sensible information then the user knows that the target side is running correctly and that communication between host and target is possible.
The target-side code can provide information about what
is happening while tests are prepared and run. This facility
should not normally be used since the extra I/O involved will
significantly affect the behaviour of the system, but in some
circumstances it may prove useful. Since an eCos application
cannot easily be given command-line arguments the target-side
verbosity level cannot be controlled using
-V or --verbose
options. Instead it can be controlled from inside
gdb by changing the integer
variable verbose. Alternatively it can
be manipulated by running the test script
verbose.tcl. This script takes a single
argument, the desired verbosity level, which should be a small
integer. For example, to disable target-side run-time logging
the command usbhost verbose 0 can
be used.
This script performs simple bulk IN and OUT transfers of different sizes around interesting boundaries. This test is useful to ensure the driver correctly handles the case where a transfer is just smaller than, the same size as, and just bigger than the hardware buffer in the endpoint hardware. This script takes no parameters. It determines what endpoints the device has by asking it.
If all transfers succeed within the specified time then both host and target remain in synch and further tests can be run without problem. However, if at any time a failure occurs then things get more complicated. For example, if the current test involves a series of bulk OUT transfers and the target detects that for one of these transfers it received less data than was expected then the test has failed, and the target will stop accepting data on this endpoint. However the host-side software may not have detected anything wrong and is now blocked trying to send the next lot of data.
The test code goes to considerable effort to recover from problems such as these. On the host-side separate threads are used for concurrent transfers, and on the target-side appropriate asynchronous I/O mechanisms are used. In addition there is a control thread on the host that checks the state of all the main host-side threads, and the state of the target using private control messages. If it discovers that one side has stopped sending or receiving data because of an error and the other side is blocked as a result, it will set certain flags and then cause one additional transfer to take place. That additional transfer will have the effect of unblocking the other side, which then discovers that an error has occurred by checking the appropriate flags. In this way both host and target should end up back in synch, and it is possible to move on to the next set of tests.
However, the above assumes that the testing has not triggered any serious hardware conditions. If instead the target-side hardware has been left in some strange state so that, for example, it will no longer raise an interrupt for traffic on a particular endpoint then recovery is not currently possible, and the testing software will just hang.
A possible future enhancement to the testing software would allow the host-side to raise a USB reset signal whenever a failure occurs, in the hope that this would clear any remaining problems within the target-side USB hardware.