======================================= A Quixote Application: Getting Started ======================================= :author: Dave Kuhlman :address: dkuhlman@rexx.com http://www.rexx.com/~dkuhlman :revision: 1.0b :date: Jan. 2, 2004 :copyright: Copyright (c) 2003 Dave Kuhlman. This documentation is covered by The MIT License: http://www.opensource.org/licenses/mit-license. :abstract: This document explains how to organize and implement a simple Quixote application. It is intended to enable the first-time Quixote user to start writing her/his Quixote application as quickly as possible. .. sectnum:: :depth: 4 .. contents:: :depth: 4 Introduction ============ The Quixote documentation is good on details, but a little lite on information of the type: "Put this here", "Put that there", "Next call this", etc. This document attempts to give you a way to get past a few early steps in writing your first Quixote application. This document attempts to lead you through a sequence of first steps toward implementing a basic Quixote application. Those steps are: 1. Create the application structure. 2. Processing the request; mapping requests to Python code. 3. Generating content. 4. Forms -- Getting form variables; using the form package and widgets. 5. Extending the application structure and organization for more complex tasks. We'll assume that your have installed Quixote and that you have set up your Web server to forward requests to your Quixote application. If you have not already done so, you can look at `Notes on Using Quixote with SCGI`_ for help. .. _`Notes on Using Quixote with SCGI`: http://www.rexx.com/~dkuhlman/quixote_scgi.html Also note that parts of what follows assumes that your have enabled PTL. The above mentioned "Notes ..." document shows how to enable PTL for SCGI. Application Structure and Organization ====================================== I'm going to describe a particular application organization. There is nothing magical or required about this organization. It's merely an easy way to get started. After you've learned and perhaps used this one, you will be able to change or extend it easily. Our application will live in two Python packages: - myapp.ui - myapp.services -- We'll put logic and functionality here. And, if the point of your application is to expose an object of some kind for viewing, modifying, etc, rather than a set of services, then you might name this packages ``model`` instead, in analogy with MVC (model view controller). Organizing the code in ``myapp.services`` ----------------------------------------- Our only requirement is that this code can be imported and called from the user interface in myapp.ui. So, for example, we can put our code in: - In a single class in a single module in myapp.services, for example, myapp.services.mymodule.myclass. In this case, the user interface (in myapp.ui) might call this code as follows:: from myapp.services import services service = services.MyClass() results = service.service1(arg1, arg2) - In several classes in a single module in myapp.services, for example, myapp.services.mymodule.myclass1 and myapp.services.mymodule.myclass2. In this case, the user interface (in myapp.ui) might call this code as follows:: from myapp.services import services service = services.MyClass() otherService = services.MyOtherClass() results = service.service1(arg1, arg2) otherResults = otherService.anotherservice(arg3, arg4) - In several classes in separate modules in myapp.services, for example, myapp.services.mymodule1.myclass1 and myapp.services.mymodule2.myclass2. In this case, the user interface (in myapp.ui) might call this code as follows:: from myapp.services import firstModule, secondModule firstService = firstModule.SomeClass() secondService = secondModule.AnotherClass() firstResults = firstService.some_method(arg1, arg2) firstResults = secondService.another_method(arg3, arg4) - And so on ... Processing Requests =================== Our primary goals here are the following: 1. Map requests to (user interface) code that will respond to them. 2. Call the code in the ``services`` (or ``model``) package in order to produce the results or perform operations requested. The following describes several alternative ways to do this. And, for more explanation about how this works, see: `Quixote Programming Overview`_. .. _`Quixote Programming Overview`: http://www.mems-exchange.org/software/quixote/doc/programming.html Suppose that you want your application to respond to requests specified by the following URIs: - \http://myhost/requestinfo - \http://myhost/serverinfo An explicit style ----------------- Here is a simple way to process requests: 1. In ``__init__.py``, place the following code:: # Map requests/URI to the user interface code. _q_exports = ['requestinfo', 'serverinfo'] from servicesui import ServicesUI # # Respond to the requestinfo request -- http://localhost/requestinfo # def requestinfo(request): service = ServicesUI() return service.do_requestinfo(request) # # Respond to the serverinfo request -- http://localhost/serverinfo # def serverinfo(request): service = ServicesUI() return service.do_serverinfo(request) # # Respond to a generic request -- http://localhost/ # def _q_index(request): service = ServicesUI() return service.index(request) Explanation: - Place the names of the services to be exposed in the list whose name is "_q_exports". - Provide a function for each service. Each function calls a method in the user interface. - The ``_q_index`` function will be called when there is no request name in the URI, i.e. for the following URI: \http://myhost/. 2. And in ``myapp/ui/servicesui.py``, you could put something like the following:: class ServicesUI: def index [html] (self, request): header('Services') '\n' footer() def do_requestinfo [html] (self, request): header('requestinfo') o o o footer() def do_serverinfo [html] (self, request): header('requestinfo') o o o footer() 3. To add a new service, do the following: - Add a new name to the ``_q_exports`` list in ``myapp/ui/__init__.py``. - Add a new function of the same name in ``myapp/ui/__init__.py``. - Add a new method to the class ``ServicesUI``. A lookup style -------------- This style is appropriate in the following situations: - When there are many possible options. - When some computation must be performed in order to determine which function/method to call. - When a component of the URI is part of a query, for example, an ID number, a customer number, etc. Since the following example has a relatively small number of methods, the lookup style is not necessary, but as an example ... This style looks up request handler methods in the UI class. 1. Put something like the following in ``myapp/ui/__init__.py``:: # Map requests/URI to the user interface code. _q_exports = [] from servicesui import ServicesUI def _q_lookup(request, name): services = ServicesUI() meth = getattr(services, 'do_' + name, services.index) return meth(request) Explanation: - Notice that the ``_q_exports`` variable must be defined even when it is empty. - Function ``_q_lookup`` is called for names in the (URI) path that are *not* in the ``_q_exports`` list. It checks the ``ServicesUI`` class for a corresponding method. If found, it calls that method to produce content. If not found, it calls method ``index``. 2. And ``myapp/ui/servicesui.py`` could be the same as in the previous example. 3. In order to add a new request, we only need to add the appropriately name method to the class ``ServicesUI``. Given the above ``_q_lookup`` function, that new method should have a name of the form "do_xxxx". Generating Content with PTL =========================== What to do: - Enable PTL -- The following must be done sometime *before* the module containing the PTL code is imported:: from quixote import enable_ptl enable_ptl() If you are using SCGI/mod_scgi, you can put these lines of code in your driver script. - Name the file that implements the module with extension ".ptl". - Mark PTL functions and methods with either "[html]" or "[plain]" in the header. See the example below. - Import your PTL module as you would a normal Python module. There is more explanation of PTL in `PTL: Python Template Language`_. .. _`PTL: Python Template Language`: http://www.mems-exchange.org/software/quixote/doc/PTL.html A few notes and cautions: - When using PTL function/method type ``[html]``, all literal strings become instances of ``htmltext``. Therefore, if you pass a literal string to a function or operation that requires a plain string, use the ``str()`` to convert it. - There is no need to return a value from a PTL function or method. That is done automatically for you. - You *cannot* put a doc string in a PTL function or method. Here is a sample that generates content to be returned to the Web browser/client:: def header [html] (subtitle): '\n' '\n' 'Quixote Test #2 -- %s\n' % subtitle '\n' '\n' '

Quixote Test #2 -- %s

\n' % subtitle 'menu\n' '
\n' def footer [html] (): '
\n' '

menu

\n' '

Generated on: %s

\n' % time.ctime() '
\n' '\n' '\n' o o o class InfoUI: def do_serverinfo [html] (self, request): header('Server Information') '\n' footer() Explanation: - Note that there is no ``return`` statement. Content is returned automatically. - The ``header`` and ``footer`` functions in our example are PTL functions and can be called just like regular Python functions. The content they produce is inserted at the place where they are called. - Our example method has the special marker "[html]". PTL functions must be marked with either "[html]" or "[plain]". Here is the difference: - Using "[html]" results in strings that are of type: . In addition, non-literal strings are passed through a function that escapes HTML special characters. - Using "[plain]" produces standard Python strings. Use "[plain]" only when there is no danger of non-literal strings containing characters that must be escaped (for example, "<" and "&"). - Multi-line strings inside triple-quoting are also allowed. The Quixote Forms Package ========================= Forms and Arguments ------------------- Here we describe a simple way to get values from an HTML form. But, note that a more powerful and complete way to produce and process forms is described in the next sub-section: `Quixote forms and widgets`_. Suppose the HTTP request contains CGI variables from an HTML form. Here is how you can get the values of those variables:: value1 = request.get_form_var('arg1') value2 = request.get_form_var('arg2', 'no-value') '

value1: %s

\n' % value1 '

value2: %s

\n' % value2 Explanation: - The first argument is the name of the variable in the HTML form. - The second, optional argument is a default value, which is returned if the variable does not exist in the request. If you do not specify a default value and the variable does not exist, then None is returned. - A request/URI containing variables ``arg1`` and ``arg2`` might look something like the following:: http://thrush:8081/requestinfo?arg1=111&arg2=222 Quixote forms and widgets ------------------------- Quixote forms and widgets enable us to: - Create widgets to be included in the form. - Generate the HTML form. - Retrieve the values of variables in the form. You can find more documentation on Quixote forms and widgets in `Quixote Widget Classes`_. .. _`Quixote Widget Classes`: http://www.mems-exchange.org/software/quixote/doc/widgets.html A couple of notes about what follows: - In what follows, I do *not* use the ``form`` object. The ``form`` object provides a framework or container for widgets, for rendering operations on those widgets, and for actions. Use of the ``form`` object is probably a good idea, but explaining how to use it is a task for the next lesson on Quixote. - This section explains how to use the current forms package. A newer package ``form2`` is currently under development and available in the new alpha versions of Quixote. How to process a form --------------------- Here is a basic sequence of actions that you can use when you need to repeatedly process and re-display a form. 1. Check the request object for existence of the form. 2. Create default values if there is no form. 3. Or, if the form exists, parse/get the values of variables in the form. 4. Call the logic/service that uses the arguments from the form. 5. Render the form. How to create widgets --------------------- Here is an example that creates several text entry fields, a set of radio buttons, and a submit button:: def do_plant_db [html] (self, request): plantName = widget.StringWidget('plant_name') plantDesc = widget.StringWidget('plant_desc') plantRating = widget.StringWidget('plant_rating') action = widget.RadiobuttonsWidget('command', value='View', allowed_values=['View', 'Add', 'Delete', 'Modify'], delim='\n') submit = widget.SubmitButtonWidget(value='Do it') o o o How to get form variables ------------------------- Use the ``parse`` method to get the values for widgets in an HTTP request. For example:: def do_plant_db [html] (self, request): o o o if request.form: plantNameValue = plantName.parse(request) plantDescValue = plantDesc.parse(request) plantRatingValue = plantRating.parse(request) actionValue = action.parse(request) else: plantNameValue = '' plantDescValue = '' plantRatingValue = '' actionValue = 'View' How to render widgets --------------------- In order to generate the HTML content, call the ``render`` method for each widget. Here is some sample code. Remember that this code is from a PTL file:: def do_plant_db [html] (self, request): o o o '
\n' '' '\n\n\n' '
Plant name:' plantName.render(request) '
Plant description:' plantDesc.render(request) '
Plant rating:' plantRating.render(request) '
\n' '\n' '\n' '
' action.render(request) '
\n' '

\n

' submit.render(request) '
\n' '
\n' o o o How to test your widgets ------------------------ When I first started using Quixote forms and widgets, I found it convenient to be able to render widgets and inspect the generated HTML outside of the Quixote/Web environment. Here is a quick, simple Python script that you can run from the command line and which shows the content generated by your widgets:: #!/usr/bin/env python from quixote.form import widget from quixote import http_request request = http_request.HTTPRequest(sys.stdin, {}) print widget.StringWidget('f1').render(request) print '\n', widget.RadiobuttonsWidget('r1', value='two', allowed_values=['one', 'two', 'three']).render(request) value = """ Line #1 Line #2 Line #3 """ print '\n', widget.TextWidget('t1', value=value).render(request) print '\n', widget.CheckboxWidget('cb1', value='maybe').render(request) vals = [101, 102, 103] descs = ['First', 'Second', 'Third'] print '\n', widget.SingleSelectWidget('ss1', value=102, allowed_values=vals, descriptions=descs).render(request) print '\n', widget.HiddenWidget('h1', value='some secret stuff').render(request) print '\n', widget.FloatWidget('float1').render(request) Explanation: - The ``render`` method requires a ``request`` object, so we create a dummy one. - In order to see the HTML code, we create each widget and render it. Extensions to the Basic Organization ==================================== Suppose you want to sub-divide your request name space, in effect providing sub-categories of requests. For example:: http://myserver/info/requestinfo http://myserver/info/serverinfo and:: http://myserver/service/qotd http://myserver/service/temperature http://myserver/service/plant_db There are a variety of ways to do this. We'll combine several in a single example. Combining explicit and lookup styles ------------------------------------ This example uses the explicit style shown above to handle "info" requests and the lookup style to handle "service" requests:: from servicesui import MenuUI, InfoUI, ServicesUI def _q_lookup(request, name): if name == 'info': handler = InfoHandler() return handler elif name == 'service': handler = ServiceHandler() return handler else: menu = MenuUI() return menu.index(request) class InfoHandler: _q_exports = ['requestinfo', 'serverinfo'] def requestinfo(self, request): info = InfoUI() return info.do_requestinfo(request) def serverinfo(self, request): info = InfoUI() return info.do_serverinfo(request) def _q_index(self, request): menu = MenuUI() return menu.index(request) class ServiceHandler: _q_exports = [] def _q_lookup(self, request, name): services = ServicesUI() meth = getattr(services, 'do_' + name, None) if meth: return meth(request) else: menu = MenuUI() return menu.index(request) Explanation: - The top level ``_q_lookup`` function returns either (1) a new handler object, if the URI requires further traversal, or (2) content to be returned. If the next component of the path is "info", then an instance of class ``InfoHandler`` is returned to be used to further traverse the URI path. Or, if the next component of the path is "service", then an instance of class ``InfoHandler`` is returned to be used to further traverse the URI path. - The ``InfoHandler`` class provides separate methods for each sub-request and lists them in the ``_q_exports`` variable. The ``_q_index`` method handles names that are not explicitly listed. - The ``ServiceHandler`` class provides a ``_q_lookup`` method which either (1) calls a method in an appropriate class, if the class contains an appropriate attribute, or (2) produces a menu of hyperlinks. - If we need further nesting, then either of the handler classes (``InfoHandler`` and ``ServiceHandler``, in our example code), can return another object which continues to traverse components of the URI. In other words, we can continue this processing to any needed level. Once again, notice that we will be organizing the "user interface" and the application code that responds to the requests in this user interface. However, the logic in our ``services`` or ``model`` package does not need to follow that same organization or structure. Special Tasks -- Back-end Resources =================================== Database connections -------------------- If there is a relational database in the back-end of your Web application, you are likely to want to maintain and reuse persistent database connections. Quixote makes this especially easy. But, it requires a bit of explanation. First, I'm using the SCGI handler. I'm not sure whether what follows applies to other handlers such FastCGI, CGI, etc. More information on using the Quixote SCGI server is available at http://www.mems-exchange.org/software/scgi/ and http://www.mems-exchange.org/software/quixote/doc/web-server.html. I've also written a short how-to document on using Quixote with Apache and SCGI, which is available at http://www.rexx.com/~dkuhlman/quixote_scgi.html. And, as you can see from the example below, I'm using the pyPgSQL implementation of the database interface to PostgreSQL, which is available at: http://pypgsql.sourceforge.net/. If you are using some other implementation of the Python DB API, you will have to tweak the sample code slightly. A little explanation -- With the SCGI server handler, at least, child processes are created to handle multiple, concurrent requests. And, at most, one request is handled at one time (concurrently) in any one child process. Therefore, if we create one database connection in each child process, we are safe using that connection in the request handler executed in that child process, since only one request handler will be active at any time in any one process. I've used this same technique to create and reuse an XML-RPC proxy and a SOAP proxy. Below is an example of how to open, use, and close one database connection for each child process created by the SCGI server. Here is a sample SCGI driver script:: #!/usr/bin/env python from scgi.quixote_handler import QuixoteHandler, main from quixote.publish import Publisher from pyPgSQL import PgSQL from quixote import enable_ptl enable_ptl() CONNECT_ARGS = 'localhost:5432:test:postgres:xxxxxxxx' class MyPublisher(Publisher): def __init__(self, root_namespace, config=None): Publisher.__init__(self, root_namespace, config) self.dbConnection = PgSQL.connect(CONNECT_ARGS) def __del__(self): self.dbConnection.close() def start_request(self, request): Publisher.start_request(self, request) request.dbConnection = self.dbConnection class MyAppHandler(QuixoteHandler): publisher_class = MyPublisher root_namespace = "test2.ui" prefix = "" if __name__ == '__main__': main(MyAppHandler) Explanation: - In the constructor of class ``MyPublisher``, we create a database connection. This constructor is called once for each child process that the SCGI server spawns, enabling us to create exactly one database connection per child process. - In the destructor of class ``MyPublisher``, we close that connection. - Also, in class ``MyPublisher``, we override the ``start_request`` method. This enables us to add the connection object to the request object before our application code (our request handler) is called. Then in our application code, we can retrieve and use the connection. For example:: def do_plant_db [html] (self, request): o o o service = services.Services() dbDonnection = request.dbConnection msg, rowSet = service.db_query(dbConnection, actionValue, sortbyValue, plantNameValue, plantDescValue, plantRatingValue) o o o Explanation: - The request object now contains the database connection, which we can retrieve and use. And, here is an example of using the connection to perform an action on the database:: class Services: o o o def db_query(self, dbConnection, action, sortby, plantName, plantDesc, plantRating): cursor = dbConnection.cursor() if action == 'View': msg = 'Viewing' rowSet = self._db_retrieve(cursor, sortby) elif action == 'Add': cursor.execute("select * from Plant_DB where p_name = '%s'" % plantName) row = cursor.fetchone() if row: msg = 'Plant (%s) already exists.' % plantName else: sql = "insert into Plant_DB values ('%s', '%s', '%s')" % \ (plantName, plantDesc, plantRating) cursor.execute(sql) dbConnection.commit() msg = 'Added' rowSet = self._db_retrieve(cursor, sortby) elif action == 'Delete': sql = "select * from Plant_DB where p_name = '%s'" % plantName cursor.execute(sql) row = cursor.fetchone() if row: cursor.execute("delete from Plant_DB where p_name='%s'" % plantName) dbConnection.commit() msg = 'Plant (%s) deleted.' % plantName else: msg = 'Plant (%s) does not exist.' % plantName rowSet = self._db_retrieve(cursor, sortby) elif action == 'Modify': cursor.execute("select * from Plant_DB where p_name = '%s'" % plantName) row = cursor.fetchone() if row: sql = "update Plant_DB set p_desc='%s', p_rating='%s' where p_name='%s'" % \ (plantDesc, plantRating, plantName) cursor.execute(sql) dbConnection.commit() msg = 'Updated plant (%s).' % plantName else: msg = 'Plant (%s) does not exist.' % plantName rowSet = self._db_retrieve(cursor, sortby) else: rowSet = None cursor.close() return (msg, rowSet) def _db_retrieve(self, cursor, sortby): if sortby == 'Name': sql = "select * from Plant_DB order by p_name" else: sql = "select * from Plant_DB order by p_rating" cursor.execute(sql) rows = cursor.fetchall() return rows The Web as a back-end resource ------------------------------ The Web is an incredible and huge resource. It's also the "Mother of all REST applications". In this section we will explore ways to treat the Web as a resource or as a feed for content to be filled into the Web pages generated by your own Web applications. Retrieval -- We can use ``urllib`` from the Python standard library to retrieve Web pages. Data extraction -- Then, we will use several techniques to extract data items from Web pages. Here are several important ones: - First we use ``sgrep``, a structured grep, to search the HTML page and retrieve chunks. - Sometimes ``sgrep`` cannot isolate and retrieve a chunk that is small enough. In these cases, it may be helpful to use the Python regular expression module ``re`` to extract more refined chunks. - And, if the chunks of text returned by ``sgrep`` contain HTML mark-up, regular expressions to search them quickly can become complex. So, we can use the ``HTMLParser`` module from the Python standard library (which defines and uses the regular expressions for us) to parse chunks of HTML and extract data. Here is the link to the document that describes how to do this in more detail: `HTML Screen Scraping: A How-To Document`_. .. _`HTML Screen Scraping: A How-To Document`: http://www.rexx.com/~dkuhlman/quixote_htmlscraping.html See Also ======== `http://www.mems-exchange.org/software/quixote/`_: The Quixote support Web site. .. _`http://www.mems-exchange.org/software/quixote/`: http://www.mems-exchange.org/software/quixote/ `Quixote Programming Overview`_: This document explains how Quixote applications map requests to the code that responds to those requests. .. _`Quixote Programming Overview`: http://www.mems-exchange.org/software/quixote/doc/programming.html `PTL: Python Template Language`_: More information on PTL. .. _`PTL: Python Template Language`: http://www.mems-exchange.org/software/quixote/doc/PTL.html `Quixote Widget Classes`_: More information on Quixote forms and HTML widgets. .. _`Quixote Widget Classes`: http://www.mems-exchange.org/software/quixote/doc/widgets.html `Introducing Quixote: A Simple Link Display`_: Another tutorial to help you get up-to-speed with Quixote quickly. .. _`Introducing Quixote: A Simple Link Display`: http://www.quixote.ca/learn/1.html `White Paper: Quixote for Web Development`_: This white paper has a helpful graphic that shows the relationship between (1) the HTTP client, (2) Apache and the mod_scgi, and (3) the Quixote application receiving *its* requests from mod_scgi. .. _`White Paper: Quixote for Web Development`: http://www.quixote.ca/overview/paper.html `Python home`_: The Python Web site. .. _`Python home`: http://www.python.org `pyPgSQL`_: pyPgSQL is a package that provides a Python DB-API 2.0 compliant interface to PostgreSQL databases. .. _`pyPgSQL`: http://pypgsql.sourceforge.net/ `SIG on Tabular Databases in Python`_: Provides specification of and implementations of the Python database API. .. _`SIG on Tabular Databases in Python`: http://www.python.org/sigs/db-sig/ `HTML Screen Scraping: A How-To Document`_: How to treat the Web as a resource. .. _`HTML Screen Scraping: A How-To Document`: http://www.rexx.com/~dkuhlman/quixote_htmlscraping.html `Notes on Using Quixote with SCGI`_: This document explains how to install, set-up, and run SCGI support for Quixote. .. _`Notes on Using Quixote with SCGI`: http://www.rexx.com/~dkuhlman/quixote_scgi.html `Docutils`_: Python Documentation Utilities -- This document was formatted with Docutils. .. _`Docutils`: http://docutils.sourceforge.net/