=======================================
A Quixote Application: Getting Started
=======================================
:author: Dave Kuhlman
:address: dkuhlman@rexx.com
http://www.rexx.com/~dkuhlman
:revision: 1.0b
:date: Jan. 2, 2004
:copyright: Copyright (c) 2003 Dave Kuhlman. This documentation is
covered by The MIT License:
http://www.opensource.org/licenses/mit-license.
:abstract: This document explains how to organize and implement a
simple Quixote application. It is intended to enable the
first-time Quixote user to start writing her/his Quixote
application as quickly as possible.
.. sectnum:: :depth: 4
.. contents:: :depth: 4
Introduction
============
The Quixote documentation is good on details, but a little lite on
information of the type: "Put this here", "Put that there", "Next
call this", etc. This document attempts to give you a way to get
past a few early steps in writing your first Quixote application.
This document attempts to lead you through a sequence of first
steps toward implementing a basic Quixote application. Those
steps are:
1. Create the application structure.
2. Processing the request; mapping requests to Python code.
3. Generating content.
4. Forms -- Getting form variables; using the form package and
widgets.
5. Extending the application structure and organization for more
complex tasks.
We'll assume that your have installed Quixote and that you have
set up your Web server to forward requests to your Quixote
application. If you have not already done so, you can look at
`Notes on Using Quixote with SCGI`_ for help.
.. _`Notes on Using Quixote with SCGI`:
http://www.rexx.com/~dkuhlman/quixote_scgi.html
Also note that parts of what follows assumes that your have
enabled PTL. The above mentioned "Notes ..." document shows how
to enable PTL for SCGI.
Application Structure and Organization
======================================
I'm going to describe a particular application organization.
There is nothing magical or required about this organization.
It's merely an easy way to get started. After you've learned and
perhaps used this one, you will be able to change or extend it
easily.
Our application will live in two Python packages:
- myapp.ui
- myapp.services -- We'll put logic and functionality here.
And, if the point of your application is to expose an object of
some kind for viewing, modifying, etc, rather than a set of
services, then you might name this packages ``model`` instead,
in analogy with MVC (model view controller).
Organizing the code in ``myapp.services``
-----------------------------------------
Our only requirement is that this code can be imported and
called from the user interface in myapp.ui. So, for example, we
can put our code in:
- In a single class in a single module in myapp.services, for
example, myapp.services.mymodule.myclass. In this case, the
user interface (in myapp.ui) might call this code as follows::
from myapp.services import services
service = services.MyClass()
results = service.service1(arg1, arg2)
- In several classes in a single module in myapp.services, for
example, myapp.services.mymodule.myclass1 and
myapp.services.mymodule.myclass2. In this case, the
user interface (in myapp.ui) might call this code as follows::
from myapp.services import services
service = services.MyClass()
otherService = services.MyOtherClass()
results = service.service1(arg1, arg2)
otherResults = otherService.anotherservice(arg3, arg4)
- In several classes in separate modules in myapp.services, for
example, myapp.services.mymodule1.myclass1 and
myapp.services.mymodule2.myclass2. In this case, the
user interface (in myapp.ui) might call this code as follows::
from myapp.services import firstModule, secondModule
firstService = firstModule.SomeClass()
secondService = secondModule.AnotherClass()
firstResults = firstService.some_method(arg1, arg2)
firstResults = secondService.another_method(arg3, arg4)
- And so on ...
Processing Requests
===================
Our primary goals here are the following:
1. Map requests to (user interface) code that will respond to them.
2. Call the code in the ``services`` (or ``model``) package in
order to produce the results or perform operations requested.
The following describes several alternative ways to do this. And,
for more explanation about how this works, see: `Quixote
Programming Overview`_.
.. _`Quixote Programming Overview`:
http://www.mems-exchange.org/software/quixote/doc/programming.html
Suppose that you want your application to respond to requests
specified by the following URIs:
- \http://myhost/requestinfo
- \http://myhost/serverinfo
An explicit style
-----------------
Here is a simple way to process requests:
1. In ``__init__.py``, place the following code::
# Map requests/URI to the user interface code.
_q_exports = ['requestinfo', 'serverinfo']
from servicesui import ServicesUI
#
# Respond to the requestinfo request -- http://localhost/requestinfo
#
def requestinfo(request):
service = ServicesUI()
return service.do_requestinfo(request)
#
# Respond to the serverinfo request -- http://localhost/serverinfo
#
def serverinfo(request):
service = ServicesUI()
return service.do_serverinfo(request)
#
# Respond to a generic request -- http://localhost/
#
def _q_index(request):
service = ServicesUI()
return service.index(request)
Explanation:
- Place the names of the services to be exposed in the list
whose name is "_q_exports".
- Provide a function for each service. Each function calls a
method in the user interface.
- The ``_q_index`` function will be called when there is no
request name in the URI, i.e. for the following URI:
\http://myhost/.
2. And in ``myapp/ui/servicesui.py``, you could put something like
the following::
class ServicesUI:
def index [html] (self, request):
header('Services')
'
. In addition, non-literal strings are passed
through a function that escapes HTML special characters.
- Using "[plain]" produces standard Python strings. Use
"[plain]" only when there is no danger of non-literal strings
containing characters that must be escaped (for example, "<"
and "&").
- Multi-line strings inside triple-quoting are also allowed.
The Quixote Forms Package
=========================
Forms and Arguments
-------------------
Here we describe a simple way to get values from an HTML form.
But, note that a more powerful and complete way to produce and
process forms is described in the next sub-section:
`Quixote forms and widgets`_.
Suppose the HTTP request contains CGI variables from an HTML form.
Here is how you can get the values of those variables::
value1 = request.get_form_var('arg1')
value2 = request.get_form_var('arg2', 'no-value')
'value1: %s
\n' % value1
'value2: %s
\n' % value2
Explanation:
- The first argument is the name of the variable in the HTML form.
- The second, optional argument is a default value, which is
returned if the variable does not exist in the request. If you
do not specify a default value and the variable does not exist,
then None is returned.
- A request/URI containing variables ``arg1`` and ``arg2`` might
look something like the following::
http://thrush:8081/requestinfo?arg1=111&arg2=222
Quixote forms and widgets
-------------------------
Quixote forms and widgets enable us to:
- Create widgets to be included in the form.
- Generate the HTML form.
- Retrieve the values of variables in the form.
You can find more documentation on Quixote forms and widgets in
`Quixote Widget Classes`_.
.. _`Quixote Widget Classes`:
http://www.mems-exchange.org/software/quixote/doc/widgets.html
A couple of notes about what follows:
- In what follows, I do *not* use the ``form`` object. The
``form`` object provides a framework or container for widgets,
for rendering operations on those widgets, and for actions. Use
of the ``form`` object is probably a good idea, but explaining
how to use it is a task for the next lesson on Quixote.
- This section explains how to use the current forms package. A
newer package ``form2`` is currently under development and
available in the new alpha versions of Quixote.
How to process a form
---------------------
Here is a basic sequence of actions that you can use when you need
to repeatedly process and re-display a form.
1. Check the request object for existence of the form.
2. Create default values if there is no form.
3. Or, if the form exists, parse/get the values of variables in
the form.
4. Call the logic/service that uses the arguments from the form.
5. Render the form.
How to create widgets
---------------------
Here is an example that creates several text entry fields, a set
of radio buttons, and a submit button::
def do_plant_db [html] (self, request):
plantName = widget.StringWidget('plant_name')
plantDesc = widget.StringWidget('plant_desc')
plantRating = widget.StringWidget('plant_rating')
action = widget.RadiobuttonsWidget('command', value='View',
allowed_values=['View', 'Add', 'Delete', 'Modify'], delim='\n')
submit = widget.SubmitButtonWidget(value='Do it')
o
o
o
How to get form variables
-------------------------
Use the ``parse`` method to get the values for widgets in an HTTP
request. For example::
def do_plant_db [html] (self, request):
o
o
o
if request.form:
plantNameValue = plantName.parse(request)
plantDescValue = plantDesc.parse(request)
plantRatingValue = plantRating.parse(request)
actionValue = action.parse(request)
else:
plantNameValue = ''
plantDescValue = ''
plantRatingValue = ''
actionValue = 'View'
How to render widgets
---------------------
In order to generate the HTML content, call the ``render`` method
for each widget. Here is some sample code. Remember that this
code is from a PTL file::
def do_plant_db [html] (self, request):
o
o
o
'\n'
o
o
o
How to test your widgets
------------------------
When I first started using Quixote forms and widgets, I found it
convenient to be able to render widgets and inspect the generated
HTML outside of the Quixote/Web environment.
Here is a quick, simple Python script that you can run from the
command line and which shows the content generated by your
widgets::
#!/usr/bin/env python
from quixote.form import widget
from quixote import http_request
request = http_request.HTTPRequest(sys.stdin, {})
print widget.StringWidget('f1').render(request)
print '\n', widget.RadiobuttonsWidget('r1', value='two',
allowed_values=['one', 'two', 'three']).render(request)
value = """
Line #1
Line #2
Line #3
"""
print '\n', widget.TextWidget('t1', value=value).render(request)
print '\n', widget.CheckboxWidget('cb1',
value='maybe').render(request)
vals = [101, 102, 103]
descs = ['First', 'Second', 'Third']
print '\n', widget.SingleSelectWidget('ss1', value=102,
allowed_values=vals, descriptions=descs).render(request)
print '\n', widget.HiddenWidget('h1', value='some secret
stuff').render(request)
print '\n', widget.FloatWidget('float1').render(request)
Explanation:
- The ``render`` method requires a ``request`` object, so we
create a dummy one.
- In order to see the HTML code, we create each widget and render
it.
Extensions to the Basic Organization
====================================
Suppose you want to sub-divide your request name space, in effect
providing sub-categories of requests. For example::
http://myserver/info/requestinfo
http://myserver/info/serverinfo
and::
http://myserver/service/qotd
http://myserver/service/temperature
http://myserver/service/plant_db
There are a variety of ways to do this. We'll combine several in
a single example.
Combining explicit and lookup styles
------------------------------------
This example uses the explicit style shown above to handle "info"
requests and the lookup style to handle "service" requests::
from servicesui import MenuUI, InfoUI, ServicesUI
def _q_lookup(request, name):
if name == 'info':
handler = InfoHandler()
return handler
elif name == 'service':
handler = ServiceHandler()
return handler
else:
menu = MenuUI()
return menu.index(request)
class InfoHandler:
_q_exports = ['requestinfo', 'serverinfo']
def requestinfo(self, request):
info = InfoUI()
return info.do_requestinfo(request)
def serverinfo(self, request):
info = InfoUI()
return info.do_serverinfo(request)
def _q_index(self, request):
menu = MenuUI()
return menu.index(request)
class ServiceHandler:
_q_exports = []
def _q_lookup(self, request, name):
services = ServicesUI()
meth = getattr(services, 'do_' + name, None)
if meth:
return meth(request)
else:
menu = MenuUI()
return menu.index(request)
Explanation:
- The top level ``_q_lookup`` function returns either (1) a new
handler object, if the URI requires further traversal, or (2)
content to be returned. If the next component of the path is
"info", then an instance of class ``InfoHandler`` is returned to
be used to further traverse the URI path. Or, if the next
component of the path is "service", then an instance of class
``InfoHandler`` is returned to be used to further traverse the
URI path.
- The ``InfoHandler`` class provides separate methods for each
sub-request and lists them in the ``_q_exports`` variable. The
``_q_index`` method handles names that are not explicitly listed.
- The ``ServiceHandler`` class provides a ``_q_lookup`` method
which either (1) calls a method in an appropriate class, if the
class contains an appropriate attribute, or (2) produces a menu
of hyperlinks.
- If we need further nesting, then either of the handler classes
(``InfoHandler`` and ``ServiceHandler``, in our example code),
can return another object which continues to traverse components
of the URI. In other words, we can continue this processing to
any needed level.
Once again, notice that we will be organizing the "user interface"
and the application code that responds to the requests in this
user interface. However, the logic in our ``services`` or
``model`` package does not need to follow that same organization
or structure.
Special Tasks -- Back-end Resources
===================================
Database connections
--------------------
If there is a relational database in the back-end of your Web
application, you are likely to want to maintain and reuse persistent
database connections. Quixote makes this especially easy. But,
it requires a bit of explanation.
First, I'm using the SCGI handler. I'm not sure whether what
follows applies to other handlers such FastCGI, CGI, etc. More
information on using the Quixote SCGI server is available at
http://www.mems-exchange.org/software/scgi/ and
http://www.mems-exchange.org/software/quixote/doc/web-server.html.
I've also written a short how-to document on using Quixote with
Apache and SCGI, which is available at
http://www.rexx.com/~dkuhlman/quixote_scgi.html.
And, as you can see from the example below, I'm using the pyPgSQL
implementation of the database interface to PostgreSQL, which is
available at: http://pypgsql.sourceforge.net/. If you are using
some other implementation of the Python DB API, you will have to
tweak the sample code slightly.
A little explanation -- With the SCGI server handler, at least,
child processes are created to handle multiple, concurrent
requests. And, at most, one request is handled at one time
(concurrently) in any one child process. Therefore, if we create
one database connection in each child process, we are safe using
that connection in the request handler executed in that child
process, since only one request handler will be active at any time
in any one process.
I've used this same technique to create and reuse an XML-RPC proxy
and a SOAP proxy.
Below is an example of how to open, use, and close one database
connection for each child process created by the SCGI server.
Here is a sample SCGI driver script::
#!/usr/bin/env python
from scgi.quixote_handler import QuixoteHandler, main
from quixote.publish import Publisher
from pyPgSQL import PgSQL
from quixote import enable_ptl
enable_ptl()
CONNECT_ARGS = 'localhost:5432:test:postgres:xxxxxxxx'
class MyPublisher(Publisher):
def __init__(self, root_namespace, config=None):
Publisher.__init__(self, root_namespace, config)
self.dbConnection = PgSQL.connect(CONNECT_ARGS)
def __del__(self):
self.dbConnection.close()
def start_request(self, request):
Publisher.start_request(self, request)
request.dbConnection = self.dbConnection
class MyAppHandler(QuixoteHandler):
publisher_class = MyPublisher
root_namespace = "test2.ui"
prefix = ""
if __name__ == '__main__':
main(MyAppHandler)
Explanation:
- In the constructor of class ``MyPublisher``, we create a
database connection. This constructor is called once for each
child process that the SCGI server spawns, enabling us to create
exactly one database connection per child process.
- In the destructor of class ``MyPublisher``, we close that
connection.
- Also, in class ``MyPublisher``, we override the
``start_request`` method. This enables us to add the connection
object to the request object before our application code (our
request handler) is called.
Then in our application code, we can retrieve and use the
connection. For example::
def do_plant_db [html] (self, request):
o
o
o
service = services.Services()
dbDonnection = request.dbConnection
msg, rowSet = service.db_query(dbConnection,
actionValue, sortbyValue, plantNameValue,
plantDescValue, plantRatingValue)
o
o
o
Explanation:
- The request object now contains the database connection, which
we can retrieve and use.
And, here is an example of using the connection to perform an
action on the database::
class Services:
o
o
o
def db_query(self, dbConnection, action, sortby, plantName, plantDesc, plantRating):
cursor = dbConnection.cursor()
if action == 'View':
msg = 'Viewing'
rowSet = self._db_retrieve(cursor, sortby)
elif action == 'Add':
cursor.execute("select * from Plant_DB where p_name = '%s'" % plantName)
row = cursor.fetchone()
if row:
msg = 'Plant (%s) already exists.' % plantName
else:
sql = "insert into Plant_DB values ('%s', '%s', '%s')" % \
(plantName, plantDesc, plantRating)
cursor.execute(sql)
dbConnection.commit()
msg = 'Added'
rowSet = self._db_retrieve(cursor, sortby)
elif action == 'Delete':
sql = "select * from Plant_DB where p_name = '%s'" % plantName
cursor.execute(sql)
row = cursor.fetchone()
if row:
cursor.execute("delete from Plant_DB where p_name='%s'" % plantName)
dbConnection.commit()
msg = 'Plant (%s) deleted.' % plantName
else:
msg = 'Plant (%s) does not exist.' % plantName
rowSet = self._db_retrieve(cursor, sortby)
elif action == 'Modify':
cursor.execute("select * from Plant_DB where p_name = '%s'" % plantName)
row = cursor.fetchone()
if row:
sql = "update Plant_DB set p_desc='%s', p_rating='%s' where p_name='%s'" % \
(plantDesc, plantRating, plantName)
cursor.execute(sql)
dbConnection.commit()
msg = 'Updated plant (%s).' % plantName
else:
msg = 'Plant (%s) does not exist.' % plantName
rowSet = self._db_retrieve(cursor, sortby)
else:
rowSet = None
cursor.close()
return (msg, rowSet)
def _db_retrieve(self, cursor, sortby):
if sortby == 'Name':
sql = "select * from Plant_DB order by p_name"
else:
sql = "select * from Plant_DB order by p_rating"
cursor.execute(sql)
rows = cursor.fetchall()
return rows
The Web as a back-end resource
------------------------------
The Web is an incredible and huge resource. It's also the "Mother
of all REST applications".
In this section we will explore ways to treat the Web as a resource
or as a feed for content to be filled into the Web pages generated
by your own Web applications.
Retrieval -- We can use ``urllib`` from the Python standard
library to retrieve Web pages.
Data extraction -- Then, we will use several techniques to extract
data items from Web pages. Here are several important ones:
- First we use ``sgrep``, a structured grep, to search the HTML
page and retrieve chunks.
- Sometimes ``sgrep`` cannot isolate and retrieve a chunk that is
small enough. In these cases, it may be helpful to use the
Python regular expression module ``re`` to extract more refined
chunks.
- And, if the chunks of text returned by ``sgrep`` contain HTML
mark-up, regular expressions to search them quickly can become
complex. So, we can use the ``HTMLParser`` module from the
Python standard library (which defines and uses the regular
expressions for us) to parse chunks of HTML and extract data.
Here is the link to the document that describes how to do this in
more detail: `HTML Screen Scraping: A How-To Document`_.
.. _`HTML Screen Scraping: A How-To Document`:
http://www.rexx.com/~dkuhlman/quixote_htmlscraping.html
See Also
========
`http://www.mems-exchange.org/software/quixote/`_: The Quixote
support Web site.
.. _`http://www.mems-exchange.org/software/quixote/`:
http://www.mems-exchange.org/software/quixote/
`Quixote Programming Overview`_: This document explains how
Quixote applications map requests to the code that responds to
those requests.
.. _`Quixote Programming Overview`:
http://www.mems-exchange.org/software/quixote/doc/programming.html
`PTL: Python Template Language`_: More information on PTL.
.. _`PTL: Python Template Language`:
http://www.mems-exchange.org/software/quixote/doc/PTL.html
`Quixote Widget Classes`_: More information on Quixote forms and
HTML widgets.
.. _`Quixote Widget Classes`:
http://www.mems-exchange.org/software/quixote/doc/widgets.html
`Introducing Quixote: A Simple Link Display`_: Another tutorial to
help you get up-to-speed with Quixote quickly.
.. _`Introducing Quixote: A Simple Link Display`:
http://www.quixote.ca/learn/1.html
`White Paper: Quixote for Web Development`_: This white paper has
a helpful graphic that shows the relationship between (1) the HTTP
client, (2) Apache and the mod_scgi, and (3) the Quixote
application receiving *its* requests from mod_scgi.
.. _`White Paper: Quixote for Web Development`:
http://www.quixote.ca/overview/paper.html
`Python home`_: The Python Web site.
.. _`Python home`: http://www.python.org
`pyPgSQL`_: pyPgSQL is a package that provides a
Python DB-API 2.0 compliant interface to PostgreSQL databases.
.. _`pyPgSQL`: http://pypgsql.sourceforge.net/
`SIG on Tabular Databases in Python`_: Provides specification of
and implementations of the Python database API.
.. _`SIG on Tabular Databases in Python`:
http://www.python.org/sigs/db-sig/
`HTML Screen Scraping: A How-To Document`_: How to treat the Web
as a resource.
.. _`HTML Screen Scraping: A How-To Document`:
http://www.rexx.com/~dkuhlman/quixote_htmlscraping.html
`Notes on Using Quixote with SCGI`_: This document explains how to
install, set-up, and run SCGI support for Quixote.
.. _`Notes on Using Quixote with SCGI`:
http://www.rexx.com/~dkuhlman/quixote_scgi.html
`Docutils`_: Python Documentation Utilities -- This document was
formatted with Docutils.
.. _`Docutils`:
http://docutils.sourceforge.net/
|