2.1���Organizing the code in myapp.services

Our only requirement is that this code can be imported and called from the user interface in myapp.ui. So, for example, we can put our code in:

• In a single class in a single module in myapp.services, for example, myapp.services.mymodule.myclass. In this case, the user interface (in myapp.ui) might call this code as follows:

  from myapp.services import services
  
  service = services.MyClass()
  results = service.service1(arg1, arg2)
  

• In several classes in a single module in myapp.services, for example, myapp.services.mymodule.myclass1 and myapp.services.mymodule.myclass2. In this case, the user interface (in myapp.ui) might call this code as follows:

  from myapp.services import services
  
  service = services.MyClass()
  otherService = services.MyOtherClass()
  results = service.service1(arg1, arg2)
  otherResults = otherService.anotherservice(arg3, arg4)
  

• In several classes in separate modules in myapp.services, for example, myapp.services.mymodule1.myclass1 and myapp.services.mymodule2.myclass2. In this case, the user interface (in myapp.ui) might call this code as follows:

  from myapp.services import firstModule, secondModule
  
  firstService = firstModule.SomeClass()
  secondService = secondModule.AnotherClass()
  firstResults = firstService.some_method(arg1, arg2)
  firstResults = secondService.another_method(arg3, arg4)
  

• And so on ...

3.1���An explicit style

Here is a simple way to process requests:

1. In __init__.py, place the following code:

   # Map requests/URI to the user interface code.
   
   _q_exports = ['requestinfo', 'serverinfo']
   
   from servicesui import ServicesUI
   
   #
   # Respond to the requestinfo request -- http://localhost/requestinfo
   #
   def requestinfo(request):
       service = ServicesUI()
       return service.do_requestinfo(request)
   
   #
   # Respond to the serverinfo request -- http://localhost/serverinfo
   #
   def serverinfo(request):
       service = ServicesUI()
       return service.do_serverinfo(request)
   
   #
   # Respond to a generic request -- http://localhost/
   #
   def _q_index(request):
       service = ServicesUI()
       return service.index(request)
   

   Explanation:

   • Place the names of the services to be exposed in the list whose name is "_q_exports".
   • Provide a function for each service. Each function calls a method in the user interface.
   • The _q_index function will be called when there is no request name in the URI, i.e. for the following URI: http://myhost/.

2. And in myapp/ui/servicesui.py, you could put something like the following:

   class ServicesUI:
   
       def index [html] (self, request):
           header('Services')
           '<ul>\n'
           '<li><a href="requestinfo">request object info</a></li>\n'
           '<li><a href="serverinfo">Server info</a></li>\n'
           '</ul>\n'
           footer()
   
       def do_requestinfo [html] (self, request):
           header('requestinfo')
           o
           o
           o
           footer()
   
       def do_serverinfo [html] (self, request):
           header('requestinfo')
           o
           o
           o
           footer()
   

3. To add a new service, do the following:

   • Add a new name to the _q_exports list in myapp/ui/__init__.py.
   • Add a new function of the same name in myapp/ui/__init__.py.
   • Add a new method to the class ServicesUI.

3.2���A lookup style

This style is appropriate in the following situations:

• When there are many possible options.
• When some computation must be performed in order to determine which function/method to call.
• When a component of the URI is part of a query, for example, an ID number, a customer number, etc.

Since the following example has a relatively small number of methods, the lookup style is not necessary, but as an example ...

This style looks up request handler methods in the UI class.

1. Put something like the following in myapp/ui/__init__.py:

   # Map requests/URI to the user interface code.
   
   _q_exports = []
   
   from servicesui import ServicesUI
   
   def _q_lookup(request, name):
       services = ServicesUI()
       meth = getattr(services, 'do_' + name, services.index)
       return meth(request)
   

   Explanation:

   • Notice that the _q_exports variable must be defined even when it is empty.
   • Function _q_lookup is called for names in the (URI) path that are not in the _q_exports list. It checks the ServicesUI class for a corresponding method. If found, it calls that method to produce content. If not found, it calls method index.

2. And myapp/ui/servicesui.py could be the same as in the previous example.

3. In order to add a new request, we only need to add the appropriately name method to the class ServicesUI. Given the above _q_lookup function, that new method should have a name of the form "do_xxxx".

5.1���Forms and Arguments

Here we describe a simple way to get values from an HTML form. But, note that a more powerful and complete way to produce and process forms is described in the next sub-section: Quixote forms and widgets.

Suppose the HTTP request contains CGI variables from an HTML form. Here is how you can get the values of those variables:

value1 = request.get_form_var('arg1')
value2 = request.get_form_var('arg2', 'no-value')
'<p>value1: %s</p>\n' % value1
'<p>value2: %s</p>\n' % value2

Explanation:

• The first argument is the name of the variable in the HTML form.

• The second, optional argument is a default value, which is returned if the variable does not exist in the request. If you do not specify a default value and the variable does not exist, then None is returned.

• A request/URI containing variables arg1 and arg2 might look something like the following:

  http://thrush:8081/requestinfo?arg1=111&arg2=222
  

5.2���Quixote forms and widgets

Quixote forms and widgets enable us to:

• Create widgets to be included in the form.
• Generate the HTML form.
• Retrieve the values of variables in the form.

You can find more documentation on Quixote forms and widgets in Quixote Widget Classes.

A couple of notes about what follows:

• In what follows, I do not use the form object. The form object provides a framework or container for widgets, for rendering operations on those widgets, and for actions. Use of the form object is probably a good idea, but explaining how to use it is a task for the next lesson on Quixote.
• This section explains how to use the current forms package. A newer package form2 is currently under development and available in the new alpha versions of Quixote.

5.3���How to process a form

Here is a basic sequence of actions that you can use when you need to repeatedly process and re-display a form.

1. Check the request object for existence of the form.
2. Create default values if there is no form.
3. Or, if the form exists, parse/get the values of variables in the form.
4. Call the logic/service that uses the arguments from the form.
5. Render the form.

5.4���How to create widgets

Here is an example that creates several text entry fields, a set of radio buttons, and a submit button:

def do_plant_db [html] (self, request):
    plantName = widget.StringWidget('plant_name')
    plantDesc = widget.StringWidget('plant_desc')
    plantRating = widget.StringWidget('plant_rating')
    action = widget.RadiobuttonsWidget('command', value='View',
        allowed_values=['View', 'Add', 'Delete', 'Modify'], delim='</td>\n<td>')
    submit = widget.SubmitButtonWidget(value='Do it')
    o
    o
    o

5.5���How to get form variables

Use the parse method to get the values for widgets in an HTTP request. For example:

def do_plant_db [html] (self, request):
    o
    o
    o
    if request.form:
        plantNameValue = plantName.parse(request)
        plantDescValue = plantDesc.parse(request)
        plantRatingValue = plantRating.parse(request)
        actionValue = action.parse(request)
    else:
        plantNameValue = ''
        plantDescValue = ''
        plantRatingValue = ''
        actionValue = 'View'

5.6���How to render widgets

In order to generate the HTML content, call the render method for each widget. Here is some sample code. Remember that this code is from a PTL file:

def do_plant_db [html] (self, request):
    o
    o
    o
    '<form method="POST" action="plant_db">\n'
    '<table width="50%">'
    '<tr><td>Plant name:</td><td>'
    plantName.render(request)
    '</td></tr>\n<tr><td>Plant description:</td><td>'
    plantDesc.render(request)
    '</td></tr>\n<tr><td>Plant rating:</td><td>'
    plantRating.render(request)
    '</td></tr>\n'
    '</table>\n'
    '<table border="1" width="100%">\n'
    '<tr><td>'
    action.render(request)
    '</td></tr>\n'
    '</table>\n'
    '<p />\n<div align="center">'
    submit.render(request)
    '</div>\n'
    '</form>\n'
    o
    o
    o

5.7���How to test your widgets

When I first started using Quixote forms and widgets, I found it convenient to be able to render widgets and inspect the generated HTML outside of the Quixote/Web environment.

Here is a quick, simple Python script that you can run from the command line and which shows the content generated by your widgets:

#!/usr/bin/env python

from quixote.form import widget
from quixote import http_request

request = http_request.HTTPRequest(sys.stdin, {})

print widget.StringWidget('f1').render(request)
print '\n', widget.RadiobuttonsWidget('r1', value='two',
    allowed_values=['one', 'two', 'three']).render(request)
value = """
Line #1
Line #2
Line #3
"""
print '\n', widget.TextWidget('t1', value=value).render(request)
print '\n', widget.CheckboxWidget('cb1',
value='maybe').render(request)
vals = [101, 102, 103]
descs = ['First', 'Second', 'Third']
print '\n', widget.SingleSelectWidget('ss1', value=102,
    allowed_values=vals, descriptions=descs).render(request)
print '\n', widget.HiddenWidget('h1', value='some secret
stuff').render(request)
print '\n', widget.FloatWidget('float1').render(request)

Explanation:

• The render method requires a request object, so we create a dummy one.
• In order to see the HTML code, we create each widget and render it.

6.1���Combining explicit and lookup styles

This example uses the explicit style shown above to handle "info" requests and the lookup style to handle "service" requests:

from servicesui import MenuUI, InfoUI, ServicesUI

def _q_lookup(request, name):
    if name == 'info':
        handler = InfoHandler()
        return handler
    elif name == 'service':
        handler = ServiceHandler()
        return handler
    else:
        menu = MenuUI()
        return menu.index(request)

class InfoHandler:
    _q_exports = ['requestinfo', 'serverinfo']
    def requestinfo(self, request):
        info = InfoUI()
        return info.do_requestinfo(request)
    def serverinfo(self, request):
        info = InfoUI()
        return info.do_serverinfo(request)
    def _q_index(self, request):
        menu = MenuUI()
        return menu.index(request)

class ServiceHandler:
    _q_exports = []
    def _q_lookup(self, request, name):
        services = ServicesUI()
        meth = getattr(services, 'do_' + name, None)
        if meth:
            return meth(request)
        else:
            menu = MenuUI()
            return menu.index(request)

Explanation:

• The top level _q_lookup function returns either (1) a new handler object, if the URI requires further traversal, or (2) content to be returned. If the next component of the path is "info", then an instance of class InfoHandler is returned to be used to further traverse the URI path. Or, if the next component of the path is "service", then an instance of class InfoHandler is returned to be used to further traverse the URI path.
• The InfoHandler class provides separate methods for each sub-request and lists them in the _q_exports variable. The _q_index method handles names that are not explicitly listed.
• The ServiceHandler class provides a _q_lookup method which either (1) calls a method in an appropriate class, if the class contains an appropriate attribute, or (2) produces a menu of hyperlinks.
• If we need further nesting, then either of the handler classes (InfoHandler and ServiceHandler, in our example code), can return another object which continues to traverse components of the URI. In other words, we can continue this processing to any needed level.

Once again, notice that we will be organizing the "user interface" and the application code that responds to the requests in this user interface. However, the logic in our services or model package does not need to follow that same organization or structure.

7.1���Database connections

If there is a relational database in the back-end of your Web application, you are likely to want to maintain and reuse persistent database connections. Quixote makes this especially easy. But, it requires a bit of explanation.

First, I'm using the SCGI handler. I'm not sure whether what follows applies to other handlers such FastCGI, CGI, etc. More information on using the Quixote SCGI server is available at http://www.mems-exchange.org/software/scgi/ and http://www.mems-exchange.org/software/quixote/doc/web-server.html. I've also written a short how-to document on using Quixote with Apache and SCGI, which is available at http://www.rexx.com/~dkuhlman/quixote_scgi.html.

And, as you can see from the example below, I'm using the pyPgSQL implementation of the database interface to PostgreSQL, which is available at: http://pypgsql.sourceforge.net/. If you are using some other implementation of the Python DB API, you will have to tweak the sample code slightly.

A little explanation -- With the SCGI server handler, at least, child processes are created to handle multiple, concurrent requests. And, at most, one request is handled at one time (concurrently) in any one child process. Therefore, if we create one database connection in each child process, we are safe using that connection in the request handler executed in that child process, since only one request handler will be active at any time in any one process.

I've used this same technique to create and reuse an XML-RPC proxy and a SOAP proxy.

Below is an example of how to open, use, and close one database connection for each child process created by the SCGI server.

Here is a sample SCGI driver script:

#!/usr/bin/env python

from scgi.quixote_handler import QuixoteHandler, main
from quixote.publish import Publisher
from pyPgSQL import PgSQL

from quixote import enable_ptl
enable_ptl()

CONNECT_ARGS = 'localhost:5432:test:postgres:xxxxxxxx'

class MyPublisher(Publisher):
    def __init__(self, root_namespace, config=None):
        Publisher.__init__(self, root_namespace, config)
        self.dbConnection = PgSQL.connect(CONNECT_ARGS)
    def __del__(self):
        self.dbConnection.close()
    def start_request(self, request):
        Publisher.start_request(self, request)
        request.dbConnection = self.dbConnection

class MyAppHandler(QuixoteHandler):
    publisher_class = MyPublisher
    root_namespace = "test2.ui"
    prefix = ""

if __name__ == '__main__':
    main(MyAppHandler)

Explanation:

• In the constructor of class MyPublisher, we create a database connection. This constructor is called once for each child process that the SCGI server spawns, enabling us to create exactly one database connection per child process.
• In the destructor of class MyPublisher, we close that connection.
• Also, in class MyPublisher, we override the start_request method. This enables us to add the connection object to the request object before our application code (our request handler) is called.

Then in our application code, we can retrieve and use the connection. For example:

def do_plant_db [html] (self, request):
    o
    o
    o
    service = services.Services()
    dbDonnection = request.dbConnection
    msg, rowSet = service.db_query(dbConnection,
        actionValue, sortbyValue, plantNameValue,
        plantDescValue, plantRatingValue)
    o
    o
    o

Explanation:

• The request object now contains the database connection, which we can retrieve and use.

And, here is an example of using the connection to perform an action on the database:

class Services:
    o
    o
    o
    def db_query(self, dbConnection, action, sortby, plantName, plantDesc, plantRating):
        cursor = dbConnection.cursor()
        if action == 'View':
            msg = 'Viewing'
            rowSet = self._db_retrieve(cursor, sortby)
        elif action == 'Add':
            cursor.execute("select * from Plant_DB where p_name = '%s'" % plantName)
            row = cursor.fetchone()
            if row:
                msg = 'Plant (%s) already exists.' % plantName
            else:
                sql = "insert into Plant_DB values ('%s', '%s', '%s')" % \
                    (plantName, plantDesc, plantRating)
                cursor.execute(sql)
                dbConnection.commit()
                msg = 'Added'
            rowSet = self._db_retrieve(cursor, sortby)
        elif action == 'Delete':
            sql = "select * from Plant_DB where p_name = '%s'" % plantName
            cursor.execute(sql)
            row = cursor.fetchone()
            if row:
                cursor.execute("delete from Plant_DB where p_name='%s'" % plantName)
                dbConnection.commit()
                msg = 'Plant (%s) deleted.' % plantName
            else:
                msg = 'Plant (%s) does not exist.' % plantName
            rowSet = self._db_retrieve(cursor, sortby)
        elif action == 'Modify':
            cursor.execute("select * from Plant_DB where p_name = '%s'" % plantName)
            row = cursor.fetchone()
            if row:
                sql = "update Plant_DB set p_desc='%s', p_rating='%s' where p_name='%s'" % \
                    (plantDesc, plantRating, plantName)
                cursor.execute(sql)
                dbConnection.commit()
                msg = 'Updated plant (%s).' % plantName
            else:
                msg = 'Plant (%s) does not exist.' % plantName
            rowSet = self._db_retrieve(cursor, sortby)
        else:
            rowSet = None
        cursor.close()
        return (msg, rowSet)

    def _db_retrieve(self, cursor, sortby):
        if sortby == 'Name':
            sql = "select * from Plant_DB order by p_name"
        else:
            sql = "select * from Plant_DB order by p_rating"
        cursor.execute(sql)
        rows = cursor.fetchall()
        return rows

7.2���The Web as a back-end resource

The Web is an incredible and huge resource. It's also the "Mother of all REST applications".

In this section we will explore ways to treat the Web as a resource or as a feed for content to be filled into the Web pages generated by your own Web applications.

Retrieval -- We can use urllib from the Python standard library to retrieve Web pages.

Data extraction -- Then, we will use several techniques to extract data items from Web pages. Here are several important ones:

• First we use sgrep, a structured grep, to search the HTML page and retrieve chunks.
• Sometimes sgrep cannot isolate and retrieve a chunk that is small enough. In these cases, it may be helpful to use the Python regular expression module re to extract more refined chunks.
• And, if the chunks of text returned by sgrep contain HTML mark-up, regular expressions to search them quickly can become complex. So, we can use the HTMLParser module from the Python standard library (which defines and uses the regular expressions for us) to parse chunks of HTML and extract data.

Here is the link to the document that describes how to do this in more detail: HTML Screen Scraping: A How-To Document.