| Author: | Dave Kuhlman |
|---|---|
| Address: | dkuhlman@rexx.com http://www.rexx.com/~dkuhlman |
| Revision: | 1.9a |
| Date: | Dec. 4, 2006 |
| Copyright: | Copyright (c) 2004 Dave Kuhlman. This documentation and the software it describes is covered by The MIT License: http://www.opensource.org/licenses/mit-license. |
Abstract
generateDS.py generates Python data structures (for example, class definitions) from an XML Schema document. These data structures represent the elements in an XML document described by the XML Schema. It also generates parsers that load an XML document into those data structures. In addition, a separate file containing subclasses (stubs) is optionally generated. The user can add methods to the subclasses in order to process the contents of an XML document.
generateDS.py generates Python data structures (for example, class definitions) from an XML Schema document. These data structures represent the elements in an XML document described by the XML Schema. It also generates parsers that load an XML document into those data structures. In addition, a separate file containing subclasses (stubs) is optionally generated. The user can add methods to the subclasses in order to process the contents of an XML document.
The generated Python code contains:
The generated classes contain the following:
The generated subclass file contains one (sub-)class definition for each data representation class. If the subclass file is used, then the parser creates instances of the subclasses (instead of creating instances of the superclasses). This enables the user to extend the subclasses with "tree walk" methods, for example, that process the contents of the XML file. The user can also generate and extend multiple subclass files which use a single, common superclass file, thus implementing a number of different processes on the same XML document type.
This document explains (1) how to use generateDS.py; (2) how to use the Python code and data structures that it generates; and (3) how to modify the generated code for special purposes.
You can find the distribution here: http://www.rexx.com/~dkuhlman/generateDS-1.9a.tar.gz
It is also available at SourceForge: http://sourceforge.net/projects/generateds/
There is a mailing list at SourceForge: http://sourceforge.net/mail/?group_id=183592
Newer versions of Python have XML support in the Python standard library. For older versions of Python, install PyXML. You can find it at: http://pyxml.sourceforge.net/
De-compress the generateDS distribution file. Use something like the following:
tar xzvf generateDS-x.xx.tar.gz
Then, the regular Distutils commands should work:
python setup.py build python setup.py install # probably as root
Run generateDS.py with a single argument, the XML Schema file that defines the data structures. For example, the following will generate Python source code for data structures described in people.xsd and will write it to the file people.py. In addition, it will write subclass stubs to the file peoplesubs.py:
python generateDS.py -o people.py -s peoplesubs.py people.xsd
Here is the usage message displayed by generateDS.py:
Usage: python generateDS.py [ options ] <in_xsd_file>
Options:
-o <outfilename> Output file name for data representation classes
-s <subclassfilename> Output file name for subclasses
-p <prefix> Prefix string to be pre-pended to the class names
-n <mappingfilename> Transform names with table in mappingfilename.
-f Force creation of output files. Do not ask.
-a <namespaceabbrev> Namespace abbreviation, e.g. "xsd:". Default = 'xs:'.
-b <behaviorfilename> Input file name for behaviors added to subclasses
-m Generate properties for member variables
--subclass-suffix="XXX" Append XXX to the generated subclass names. Default="Sub".
--root-element="XXX" Assume XXX is root element of instance docs.
Default is first element defined in schema.
--super="XXX" Super module name in subclass module. Default="???"
--validator-bodies=path Path to a directory containing files that provide
bodies (implementations) of validator methods.
--use-old-getter-setter Name getters and setters getVar() and setVar(),
instead of get_var() and set_var().
The following command line flags are recognized by generateDS.py:
Namespace abbreviation, for example "xsd:". The default is 'xs:'. If the <schema> element in your XML Schema, specifies something other than "xmlns:xs=", then you need to use this option. So, suppose you have the following at the beginning of your XSchema file:
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">
Then you can the following command line option:
-a "xsd:"
But, note that generateDS.py also tries to pick-up the namespace prefix used in the XMLSchema file automatically. If the <schema> element has an attribute "xmlns:xxx" whose value is "http://www.w3.org/2001/XMLSchema", then generateDS.py will use "xxx:" as the alias for the XMLSchema namespace in the XMLSchema document.
Append suffix to the name of classes generated in the subclass file. The default, if omitted, is "Sub". For example, the following will append "_Action" to each generated subclass name:
generateDS.py --subclass-suffix="_Action" -s actions.py mydef.xsd
And the following will append nothing, making the superclass and subclass names the same:
generateDS.py --subclass-suffix="" -s actions.py mydef.xsd
Make module_name the name of the superclass module imported by the subclass module. If this flag is omitted, the following is generated near the top of the subclass file:
import ??? as supermod
and you will need to hand edit this so the correct superclass module is imported.
In some cases the element and attribute names in an XML document will conflict with Python keywords. In order to avoid these clashes, generateDS.py contains a table that maps names that might clash to acceptable names. This table is a Python dictionary named NameTable. The user can modify existing entries and add additional name-replacement pairs to this table, for example, if new conflicts occur.
In some cases the name of a child element and the name of an attribute will be the same. (I believe, but am not sure, that this is allowed by XML Schema.) Since generateDS.py treats both child elements and attributes as members of the generated class, this is a name conflict. Therefore, where such conflicts exist, generateDS.py modifies the name of the attribute by adding "_attr" to its name.
The following constructs in the XML Schema are supported:
See file people.xsd for examples of the definition of data types and structures. Also see the section on The XML Schema Input to generateDS.
Element definitions that contain attributes but no nested child elements provide access to their data content through getter and setter methods getValueOf_ and setValueOf_ and member variable valueOf_.
Elements that are defined to contain both text and nested child elements have "mixed content". generateDS.py provides access to mixed content, but the generated data structures (classes) are fundamentally different from that generated for other elements. See section Mixed content for more details.
Note that elements defined with attributes but with no nested sub-elements do not need to be declared as "mixed". For these elements, character data is captured in a member variable valueOf_, and can be accessed with member methods getValueOf_ and setValueOf_.
generateDS.py supports anyAttribute. For example, if an element is defined as follows:
<xs:element name="Tool">
<xs:complexType>
<xs:attribute name="PartNumber" type="xs:string" />
<xs:anyAttribute processContents="skip" />
</xs:complexType>
</xs:element>
Then generateDS.py will generate a class with a member variable anyAttributes_ containing a dictionary. Any attributes found in the instance XML document that are not explicitly defined for this element will be stored in this dictionary. generateDS.py also generates getters and setters as well as code for parsing and export. generateDS.py ignores processContents. See section anyAttribute for more details.
generateDS.py now generates subclasses for extensions, that is when an element definition contains something like this:
<xs:extension base="sometag">
Limitation -- There is an important limitation, however: member names duplicated (overridden ?) in an extension generate erroneous code. Sigh. I guess I needed something more to do.
Several of the generated methods have been refactored so that subclasses can reuse the code in their superclasses. Take a look at the generated code to learn how to use it.
The Python compiler/interpreter requires that it has seen a superclass before it sees the subclass that uses it. Because of this, generateDS.py delays generating a subclass until after its superclass has been generated. Therefore, the order in which classes are generated may be different from what you expect.
generateDS.py now handles definition and use of attribute groups. For example: the use of something like the following:
<xs:attributeGroup name="favorites">
<xs:attribute name="fruit" />
<xs:attribute name="vegetable" />
</xs:attributeGroup>
And, a reference or use like the following:
<xs:element name="person">
<xs:complexType mixed="0">
<xs:attributeGroup ref="favorites" />
o
o
o
Results in generation of class person that contains members fruit and vegetable.
generateDS.py now handles a limited range of substitution groups, but, there is an important limitation, in particular generateDS.py handles substitution groups that involve complex types, but does not handle those that involve (substitute for) simple types (for example, xs:string, xs:integer, etc). This is because the code generated for members defined as simple types does not provide the needed information to handle substitution groups.
generateDS.py supports some, but not all, simple types defined in "XML Schema Part 0: Primer Second Edition" ( http://www.w3.org/TR/xmlschema-0/. See section "Simple Types" and appendix B). Validation is performed for some simple types. When performed, validation is done while the XML document is being read and instances are created.
Here is a list of supported simple types:
generateDS.py generates minimal support for members defined as simpleType. However, the code generated by generateDS.py does not enforce rescriptions. For notes on how to enforce restrictions, see section simpleType and validators.
A simpleType can be a restriction on a primitive type or on a defined element type. So, for example, the following will generate valid code:
<xs:element name="percent">
<xs:simpleType>
<xs:restriction base="xs:integer">
<xs:minInclusive value="1"/>
<xs:maxInclusive value="100"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
And, the following will also generate valid code:
<xs:simpleType name="emptyString">
<xs:restriction base="xs:string">
<xs:whiteSpace value="collapse"/>
</xs:restriction>
</xs:simpleType>
<xs:element name="merge">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="emptyString">
<xs:attribute name="fromTag" type="xs:string"/>
<xs:attribute name="toTag" type="xs:string"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
Here are a few notes that should help you use validator methods to enforce restrictions.
Default behavior -- The generated code, by default, treats the value of a member whose type is a simpleType as if it were declared as type xs:string.
Validator method stubs -- For a member variable name declared as a simpleType named X, a validator method validate_X is generated. Example -- from:
<xs:simpleType name="tAnyName">
<xs:restriction base="xs:string"/>
</xs:simpleType>
The class generated by generateDS.py will contain the following method definition:
def validate_tAnyName(self, value):
# Validate type tAnyName, a restriction on xs:string.
pass
Calls to validator methods -- For a member variable declared as a simpleType X, a call to validate X is added to the build method. Example -- from:
<xs:element name="person">
<xs:complexType mixed="0">
<xs:sequence>
<xs:element name="test2" type="tAnyName"/>
</xs:sequence>
</xs:complexType>
</xs:element>
generateDS.py produces the following call:
self.validate_tAnyName(self.test2) # validate type tAnyName
Code bodies for validator methods can be added either (1) manually or (2) automatically from an external source. See command line flag --validator-bodies and see below.
You can add code to the validator method stub to enforce the restriction for the base type and further restrictions imposed on that base type. This can be done in the following ways:
The support for simpleType in generateDS.py has the following limitations (among others, I'm sure):
It only works for simpleType defined with and referenced through a name. It does not work for "in-line" definitions. So, for example, the following works:
<xs:element name="person">
<xs:complexType>
<xs:sequence>
<xs:element name="test3" type="tAnyName"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:simpleType name="tAnyName">
<xs:restriction base="xs:string"/>
</xs:simpleType>
But, the following does not work:
<xs:element name="person">
<xs:complexType>
<xs:sequence>
<xs:element name="test3">
<xs:simpleType name="tAnyName">
<xs:restriction base="xs:string"/>
</xs:simpleType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
Attributes defined as a simple type are not supported.
generateDS.py actually accepts a subset of XML Schema. The sample XML Schema file should give you a picture of how to describe an XML file and the Python classes that you will generate. And here are some notes that should help:
Specify the tag in the XML file and the name of the generated Python class in the name attribute on the xs:element. For example, to generate a Python class named "person", which will be populated from an XML element/tag "person", use the following XML Schema snippet:
<xs:element name="person" ...
To specify a data member for a generated Python class that will be propogated from an attribute in an element in an XML file, use the XML Schema xs:attribute. For attributes, generateDS recognizes the following types: "xs:string", "xs:integer", and "xs:float". For example, the following adds member data items "hobby" and "category" with types "xs:string" and "xs:integer":
<xs:element name="person">
<complexType>
<xs:attribute name="hobby" type="xs:string" />
<xs:attribute name="category" type="xs:integer" />
</complexType>
</xs:element>
To specify a data member for a generated Python class whose value is a string, integer, or float and which will be populated from a nested (simple) element, specify a nested XML Schema element whose type is "xs:string", "xs:integer", or "xs:float". Here is an example which defines a Python class "person" with a data member "description" which is a string and which is populated from a (simple) nested element:
<xs:element name="person">
<complexType>
<sequence>
<xs:element name="description" type="xs:string" />
<sequence>
</complexType>
</xs:element>
To specify a data member of a generated Python class that will be populated from a nested XML element, refer to the nested object in the "type" attribute and then define another element/type whose name is that type. For example, the following specifies that the person class will have a data member named "transportation" that will be populated from a nested XML element "bicycle" and whose value will be an instance of the generated class "bicycle":
<xs:element name="person">
<complexType>
<sequence>
<xs:element name="transportation" type="bicycle" />
<sequence>
</complexType>
</xs:element>
<xs:element name="bicycle">
o
o
o
</xs:element>
To specify a data member of a generated Python class that will contain a list of instances of a generated classes and populated from nested XML elements, add the "maxOccurs" attribute with value "unbounded". Here is an example:
<xs:element name="person">
<complexType>
<sequence>
<xs:element name="transportation" type="bicycle" maxOccurs="unbounded" />
<xs:element name="description" type="xs:string" maxOccurs="unbounded" />
<sequence>
</complexType>
</xs:element>
<xs:element name="bicycle">
o
o
o
</xs:element>
Here are a few additional rules that will help you to write XML Schema files for generateDS.py:
Here are a few additional constructions that generateDS.py understands.
You can use the <complexType> element at top level (instead of <element>) to define an element. So, for example, instead of:
<xs:element name="server-type">
<xs:complexType>
<xs:sequence>
<xs:element name="server-name" type="xs:string"/>
<xs:element name="server-description" type="xs:string"/>
</xs:sequence>
</xs:complexType>
</xs:element>
you can use the following, which is equivalent:
<xs:complexType name="server-type">
<xs:sequence>
<xs:element name="server-name" type="xs:string"/>
<xs:element name="server-description" type="xs:string"/>
</xs:sequence>
</xs:complexType>
You can use the "ref" attribute to refer to another element definition, instead of using the "name" and "type" attributes. So, for example, you can use the following:
<xs:element name="server-info">
<xs:complexType>
<xs:sequence>
<xs:element name="server-comment" type="xs:string"/>
<xs:element ref="server-type" />
</xs:sequence>
</xs:complexType>
</xs:element>
in place of this:
<xs:element name="server-info">
<xs:complexType>
<xs:sequence>
<xs:element name="server-comment" type="xs:string"/>
<xs:element name="server-type" type="server-type"/>
</xs:sequence>
</xs:complexType>
</xs:element>
generateDS.py generates a subclass for each element that that is defined as the extension of a base element. So, for the following:
<xs:complexType name="BType">
<xs:complexContent>
<xs:extension base="AType">
<xs:sequence>
o
o
o
generateDS.py will generate something like the following:
class BType(AType):
o
o
o
generateDS.py generates special code to handle elements defined as containing mixed content, that is elements defined with attribute mixed="true". See section Mixed content for more details.
With the use of the "-b" command line flag, generateDS.py will also accept as input an XML document instance that describes behaviors to be added to subclasses when the subclass file is generated with the "-s" command line switch.
An example is provided in the Demos/Xmlbehavior sub-directory of the distribution.
The XMLBehaviors capability in generateDS.py was inspired and, for the most part, designed by gian paolo ciceri (gp.ciceri@suddenthinks.com). This work is part of our work on our application development project for Quixote.
This section describes the XMLBehavior XML document that is used as input to generateDS.py. The XMLBehavior XML document is an XML instance document (given as an argument to the "-b" command line flag) that describes behaviors (methods) to be added to class definitions in the subclass file (generated with the "-s" command line flag).
See file xmlbehavior_po.xml in the Demos/Xmlbehavior directory in the distribution for an example that you can use as a model.
The elements in the XMLBehavior document type are the following:
generateDS.py contains a function get_impl_body() that implements the ability to retrieve implementation bodies. The current implementation retrieves implementation bodies from an Internet Web URL. Other sources for implementation bodies can be implemented by modifying get_impl_body().
As an example, the version that follows first tries to retrieve an implementation body from a Web address and, if that fails, attempts to obtain the implementation body from a file in the local file system using the <xb:base-impl-url> as a path to a directory containing files, each of which contains one implementation body and <xb:impl-url> as the file name. This implementation of get_impl_body was provided by Colin Dembovsky of Systemsfusion Inc. Thanks, Colin. (I've included it in the generateDS.py script, but commented out, for those who want to use and possibly extend it.):
def get_impl_body(classBehavior, baseImplUrl, implUrl):
impl = ' pass\n'
if implUrl:
trylocal = 0
if baseImplUrl:
implUrl = '%s%s' % (baseImplUrl, implUrl)
try:
implFile = urllib2.urlopen(implUrl)
impl = implFile.read()
implFile.close()
except:
trylocal = 1
if trylocal:
try:
implFile = file(implUrl)
impl = implFile.read()
implFile.close()
except:
print '*** Implementation at %s not found.' % implUrl
return impl
The simplest use is to call one of the parsing functions in the generated source file. You may be able to use one of these functions without change, or can modify one to fit your needs. generateDS.py generates the following parsing functions:
These parsing functions are generated in both the superclass and the subclass files. Note the call to the export method. You may need to comment out or un-comment this call to export according to your needs.
For example, if the generated source is in people.py, then, from the command line, run something like the following:
python people.py people.xml
Or, from within other Python code, use something like the following:
import people
rootObject = people.parse('people.xml')
The generated classes contain methods export and exportLiteral which can be called to export classes to several text formats, in particular to an XML instance document and a Python module containing Python literals. See the generated parse functions for examples showing how to call the export methods.
The export method in generated classes writes out an XML document that represents the instance that contains it and its child elements. So, for example, if your instance tree was created by one of the parsing functions described above, then calling export on the root element should reproduce the input XML document, differing only with respect to ignorable white space.
generateDS.py generates Python classes that represent the elements in an XML document, given an Xschema definition of the XML document type. The exportLiteral method will export a Python literal representation of the Python instances of the classes that represent an XML document.
When generateDS.py generates the Python source code for your classes, this new feature also generates an exportLiteral method in each class. If you call this method on the root (top-most) object, it will write out a literal representation of your class instances as Python code.
generateDS.py also generates a function at top level (parseLiteral) that parses an XML document and calls the "exportLiteral" method on the root object to write the data structure (instances of your generated classes) as a Python module that you can import to (re-)create instances of the classes that represent your XML document.
generateDS.py was designed and built with the assumption that we are not interested in marking up text content at all. What we really want is a way to represent structured and nested date in text. It takes the statement, "I want to represent nested data structures in text.", entirely seriously. Given that assumption, there may be times when you want a more "Pythonic" textual representation of the Python data structures for which generateDS.py has generated code. exportLiteral enables you to produce that representation.
This feature means that the classes that you generate from an XML schema support the interchangeability of XML and Python literals. This means that, given classes generated by generateDS.py for your XML document type, you can perform the following transformations:
This capability enables you to:
See the generated function parseLiteral for an example of how to use exportLiteral.
If you have an instance of a minidom node that represents an element in an XML document, you can also use the 'build' member function to populate an instance of the corresponding class. Here is an example:
from xml.dom import minidom
from xml.dom import Node
doc = minidom.parse(inFileName)
rootNode = doc.childNodes[0]
people = []
for child in rootNode.childNodes:
if child.nodeType == Node.ELEMENT_NODE and child.nodeName == 'person':
obj = person()
obj.build(child)
people.append(obj)
If you choose to use the generated subclass module, and I encourage you to do so, you may need to edit and modify that file. Here are some of the things that you must do (look for "???"):
You can also (and most likely will want to) add methods to the generated classes. See the section How-to Modify the Generated Code for more on this.
The classes generated from each element definition provide getter and setter methods to access its attributes and child elements.
Elements that are referenced but not defined (i.e. that are simple, for example strings, integers, floats, and booleans) are accessed through getter and setter methods in the class in which they are referenced.
Element definitions that contain attributes but no nested child elements provide access to their data content through getter and setter methods getValueOf_ and setValueOf_ and member variable valueOf_.
The goal of generateDS.py is to support data structures represented in XML as opposed to text mark-up. However, it does provides some support for mixed content. But, for mixed content, the data structures and code generated by generateDS.py are fundamentally different from those for elements that do not contain mixed content.
There are limitations, of course. A known limitation is related to extension elements. Specifically, if an element contains mixed content, and this element extends a base class, then the base class and any classes it extends must be defined to contain mixed content. This is due to the fact that generateDS.py generates a data structure (class) for elements containing mixed content that is fundamentally different from that generated for other elements.
Here is an example of mixed content:
<note>This is a <bold>nice</bold> comment.</note>
When an element is defined with something like the following:
<xs:complexType mixed="true">
<xs:sequence>
o
o
o
then, instead of generating a class whose named members refer to nested elements, a class containing a list of instances of class MixedContainer is generated. In order to process the content of a mixed content element, the code you write will need to walk this list of instances of MixedContainer and check the type of each item in that list. Basically, the structure becomes more DOM-like in the sense that it has a list of children, rather than named fields.
Instances of MixedContainer have the following methods:
Note that elements defined with attributes but with no nested sub-elements do not need to be declared as "mixed". For these elements, character data is captured in a member variable valueOf_, and can be accessed with member methods getValueOf_ and setValueOf_.
For elements that specify anyAttributes, generateDS.py produces a class containing the following:
Note: Attributes that are explicitly defined for an element are not stored in the dictionary anyAttributes_.
generateDS.py ignores the processContents attribute on the anyAttribute element in the XML Schema
This section attempts to explain how to modify and add features to the generated code.
You can add new member definitions to a generated class. Look at the 'export' and 'exportLiteral' member functions for examples of how to access member variables and how to walk nested sub-elements.
Here are interesting places to look in each class definition:
And, if you need methods that are common to and shared by several of the generated subclasses, you can put them in a new class and add that class to the superclass list for each of your subclasses.
Although you can add your own methods to the generated superclasses, I'm recommeding that you add methods to the generated subclasses in the subclass module generated with the -s command line flage, and then edit the subclass module in order to build your application. Why?
Here are some alternatives to using the subclass file:
Under the directory Demos are several examples:
Suggested uses:
There are things in Xschema that are not supported. You will have to use a restricted sub-set of Xschema to define your data structures. See above for supported features. See people.xsd and people.xml for examples.
And, then, try it on your XML Schema, and let me know about what does not work.
Warning -- This section describes an optional generated SAX parser which, I believe, is currently broken for all but the simplest schemas. Generation of a SAX parser has not been updated for the latest changes to generateDS.py. In particular, when names of elements are reused (in different parent elements), the SAX parser becomes confused. Until I've been able to figure out how to fix this, you are advised not to use the SAX parser.
generateDS.py generates two kinds of parsers: one kind is based on SAX and the other is build on minidom. See the generated functions saxParse, parse(), and parseString(). Using the SAX parser instead of the minidom parser should reduce memory requirements for large documents, since the minidom parser, but not the SAX parser, constructs a DOM tree for the entire document in memory.
However, both styles of parsers construct instances of the data structures generated by generateDS.py. This means that, even when the SAX parser is used, generateDS.py may not be well-suited for applications that read large XML documents, although what "large" means depends on your hardware. Notice that the minidom parsing functions (parse() and parseString()) over-write the variable doc so as to enable Python to reclaim the space occupied by the DOM tree, which may help alleviate the memory problem to some extent when the minidom parser is used.
Python: The Python home page.
Dave's Page: My home page, which contains more Python stuff.