1. Zenoss Unit Tests

1. Zenoss Unit Tests
Prev	Chapter 16. Testing	Next

1. Zenoss Unit Tests

1.1. Introduction

There are different types of test strategies which attempt to determine changes in behavior and errors.

Test Type	Description
Python doctest	Simple tests in the documentation for a function
Unit tests	Test functions in a module together using the runtests command
Functional testing	Try to test the software as the user would use the software. Selenium is used to test multiple Web browsers to simulate actual use.
Load testing	Attempts to determine how many operations the system is capable of performing with the provided configuration.

1.2. doctest Testing

A handy feature of Python is the ability to include simple tests in the docstring for a function. This allows the programmer to see some of the normal cases and boundary conditions, but it also allows the programmer to run sanity checks on the function by running the Python doctest utilities.

First, a complete sample file (blue.py) to illustrate:

import os
from exceptions import ValueError

def myfunc( a, b):
    """
    Determine if a likes b.

    >>> myfunc( 0, 0)
    True
    >>> myfunc( 0, 1)
    False

    # Comments in-between tests should be separated with an extra line,
    # otherwise doctest will notify you of an error.
    # This should raise an exception
    >>> myfunc( 0, "bad" )
    Traceback (most recent call last):
    ValueError: Argument is bad
    """
    if a == "bad" or  b == "bad":
        raise ValueError( "Argument is bad" )

    return a == b

if __name__ == "__main__":
    import doctest
    doctest.testmod()

Now we can test our module by running Python with the -v flag:

$ python blue.py -v
Trying:
    myfunc( 0, 0)
Expecting:
    True
ok
Trying:
    myfunc( 0, 1)
Expecting:
    False
ok
1 items had no tests:
    __main__
1 items passed all tests:
   2 tests in __main__.myfunc
2 tests in 2 items.
2 passed and 0 failed.
Test passed.

Note

The -v flag gets passed to your program, not to Python!

1.3. Zenoss' Test Runner

Zenoss has a Zope product, ZenTestRunner, whose sole purpose it to run a specific group of tests. We did this in order to avoid running all the tests in the Products directory if you only want to run tests on a specific portion of Zenoss.

Note

Do NOT run unit tests on a production server!

Some of the tests are destructive in nature (eg 'delete all events') and are intended to be used only on a development server.

All of our examples should be run as the zenoss user. If you really want to run all of the tests:

$ runtests -t unit

Tip

If you are running a Selenium server, then you can use runtests to run the unit tests and the Selenium tests. To run the Selenium tests on there own:

$ runtests -t selenium

To run all of the ZenModel tests:

runtests ZenModel

All that is required by developers is that they add tests into the tests directory that has a __init__.py contained inside that directory.

Run the existing tests to make sure that you know what to expect:
```
runtests -t unit
```
Go to the tests directory inside of the directory with the classes you want tested:
```
cd $ZENHOME/Products/ZenModel/tests
```
Copy one of the existing tests to a name reflecting the product for which you are adding tests:
```
cp testZenModel.py
          testZenNewProduct.py
```
Note
Your new test script must contain the prefix test in the filename. So testmytest.py will work, but not mytest or mytest.py.
Change the import line in the new file to reflect the new product name:
```
from Products import ZenNewProduct as product
```
Save and quit, then run the test suites to make sure everything is passing:
```
$ runtests -t unit ZenModel
```

Note

Follow the same procedures as above for ZenPacks, with the following differences:

Make sure that your ZenPack has the tests directory in it (eg $ZENHOME/ZenPacks/ZenPacks.org.zpname-version info.egg/ZenPacks/org/zpname/tests ), containing an __init__.py file and your new test script.
The runtests doesn't currently understand Python Egg-style namespaces, so only the last part of the ZenPack name is used. For example, if our ZenPack's name was ZenPacks.org.zpname
```
$ runtests -t unit zpname
```

1.3.1. An Example Unit Test

This first unit test deliberately has an error it, but we'll show what happens and how we can make it better.

from xmlrpclib import ServerProxy
from Products.ZenTestCase.BaseTestCase import BaseTestCase

class TestXmlRpc(BaseTestCase):

    def setUp(self):
        self.baseUrl = 'http://admin:zenoss@NotExistServer:8080/zport/dmd/'
        self.testdev = 'xmlrpc_testdevice'

    def testSendEvent(self):
        serv = ServerProxy( self.baseUrl + 'ZenEventManager' )
        evt = {
          'device':'xmlrpcTestDevice',
          'component':'eth0',
          'summary':'eth0 is down',
          'severity':4,
          'eventClass':'/Net'
        }
        serv.sendEvent(evt)

def test_suite():
    from unittest import TestSuite, makeSuite
    suite = TestSuite()
    suite.addTest(makeSuite(TestXmlRpc))
    return suite

First, notice that our test has to fail as the server that we're trying to reach (NotExistServer) doesn't exist. Here's the output when we run it from the command-line.

        $ runtests -t unit -n testXMLRPC ZenModel
Running tests via: /opt/zenoss/bin/python /opt/zenoss/bin/test.py -v 
--config-file /opt/zenoss/etc/zope.conf --libdir /opt/zenoss/Products/ZenModel
 testXMLRPC
Running unit tests at level 1
Running unit tests from /opt/zenoss/Products/ZenModel
Parsing /opt/zenoss/etc/zope.conf
E
======================================================================
ERROR: testSendEvent (tests.testXMLRPC.TestXmlRpc)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/zenoss/lib/python/Testing/ZopeTestCase/profiler.py", line 98,
 in __call__
    testMethod()
  File "/opt/zenoss/Products/ZenModel/tests/testXMLRPC.py", line 34,
 in testSendEvent
    serv.sendEvent(evt)
  File "/opt/zenoss/lib/python2.4/xmlrpclib.py", line 1153, in __call__
    return self.__send(self.__name, args)
  File "/opt/zenoss/lib/python2.4/xmlrpclib.py", line 1440, in __request
    verbose=self.__verbose
  File "/opt/zenoss/lib/python2.4/xmlrpclib.py", line 1186, in request
    self.send_content(h, request_body)
  File "/opt/zenoss/lib/python2.4/xmlrpclib.py", line 1300, in send_content
    connection.endheaders()
  File "/opt/zenoss/lib/python2.4/httplib.py", line 798, in endheaders
    self._send_output()
  File "/opt/zenoss/lib/python2.4/httplib.py", line 679, in _send_output
    self.send(msg)
  File "/opt/zenoss/lib/python2.4/httplib.py", line 646, in send
    self.connect()
  File "/opt/zenoss/lib/python2.4/httplib.py", line 614, in connect
    socket.SOCK_STREAM):
gaierror: (-2, 'Name or service not known')

----------------------------------------------------------------------
Ran 1 test in 0.007s

FAILED (errors=1)

While that does tell us that we do have a problem (Name or service not known), it's a lot of output for one problem. And the note at the bottom that tells us that we have errors (ie in our tests scripts) rather than failures (ie issues in our code). If this happens if every test that fails to trap exceptions or conditions generated this much output (there are over 140 unit tests in ZenModel alone!) we'd be drowned in a sea of output!

An improved example:

import traceback
from xmlrpclib import ServerProxy
from Products.ZenTestCase.BaseTestCase import BaseTestCase

class TestXmlRpc(BaseTestCase):
    "Test basic XML-RPC services against our Zenoss server"

    def setUp(self):
        self.baseUrl = 'http://admin:zenoss@localhost:8080/zport/dmd/'
        self.testdev = 'xmlrpc_testdevice'

    def testSendEvent(self):
        "Send an XML-RPC event"
        serv = ServerProxy( self.baseUrl + 'ZenEventManager' )
        evt = {
          'device':'xmlrpcTestDevice',
          'component':'eth0',
          'summary':'eth0 is down',
          'severity':4,
          'eventClass':'/Net'
        }

        try:
            serv.sendEvent(evt)
        except:
            msg= traceback.format_exc(limit=0)
            self.fail( msg )


def test_suite():
    from unittest import TestSuite, makeSuite
    suite = TestSuite()
    suite.addTest(makeSuite(TestXmlRpc))
    return suite

This time the output looks like this:

$ runtests -t unit -n testXMLRPC ZenModel
Running tests via: /opt/zenoss/bin/python /opt/zenoss/bin/test.py -v 
--config-file /opt/zenoss/etc/zope.conf --libdir /opt/zenoss/Products/ZenModel
 testXMLRPC
Running unit tests at level 1
Running unit tests from /opt/zenoss/Products/ZenModel
Parsing /opt/zenoss/etc/zope.conf
F
======================================================================
FAIL: Send an XML-RPC event
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/zenoss/lib/python/Testing/ZopeTestCase/profiler.py", line 98,
 in __call__
    testMethod()
  File "/opt/zenoss/Products/ZenModel/tests/testXMLRPC.py", line 41,
 in testSendEvent
    self.fail( msg )
  File "/opt/zenoss/lib/python2.4/unittest.py", line 301, in fail
    raise self.failureException, msg
AssertionError: Traceback (most recent call last):
gaierror: (-2, 'Name or service not known')


----------------------------------------------------------------------
Ran 1 test in 0.004s

FAILED (failures=1)

Here are the differences, from the top down:

We have a nicer description of what the test is testing (Send an XML-RPC event).
The output is (slightly) shorter but still provides us with the underlying error message that we need to know. The more levels of stack in the function, the greater the savings.
We see that we have one failure condition detected, as opposed to an error in our unit test.

Note

To get the above example to work, change the Zenoss server in the URL to be the localhost server.

1.4. Integrating With Buildbot

The Buildbot program is a Python-based build and test system used at Zenoss Inc in order to perform nightly builds of the various architectures, run unit tests and sanity check the code with PyFlakes.

Note

The Buildbot configuration is not visible outside of Zenoss.

1.5. JavaScript Test Framework

YUI includes a full unit test framework. Most of the specifics are best explained by them.

Zenoss-specific tests should all be located in $ZENHOME/Products/ZenWidgets/skins/zenui/javascript/tests directory. Each test script should then be registered in the getLoader() function in zenoss-core.js, using the naming scheme test_description.

These tests may then be run on any page using the runtests() function. For example, the dashboard tests should be registered as test_dashboard, and can then be run as:

runtests('dashboard')

This will pop up a logger window that will print test results.

An example test script has been provided. Please see:

$ZENHOME/Products/ZenWidgets/skins/zenui/javascript/tests/ test_example.js
$ZENHOME/Products/ZenWidgets/skins/zenui/javascript/zenoss-core.js

Also run in the JavaScript console of your browser:

runtests('example')

Prev	Up	Next
Chapter 16. Testing	Home	2. Functional User Interface Testing

Developer's Guide