Module cookielib

HTTP cookie handling for web clients.

This module has (now fairly distant) origins in Gisle Aas' Perl module
HTTP::Cookies, from the libwww-perl library.

Docstrings, comments and debug strings in this code refer to the
attributes of the HTTP cookie system as cookie-attributes, to distinguish
them clearly from Python attributes.

Class diagram (note that BSDDBCookieJar and the MSIE* classes are not
distributed with the Python standard library, but are available from
http://wwwsearch.sf.net/):

                        CookieJar____
                        /     \                  FileCookieJar      \                   /    |   \         \       MozillaCookieJar | LWPCookieJar \                        |               |                        |   ---MSIEBase |                         |  /      |     |                          | /   MSIEDBCookieJar BSDDBCookieJar
                  |/
               MSIECookieJar

Classes

[hide private]

Cookie
HTTP Cookie.

CookiePolicy
Defines which cookies get accepted from and returned to server.

DefaultCookiePolicy
Implements the standard rules for accepting and returning cookies.

Absent

CookieJar
Collection of HTTP cookies.

LoadError

FileCookieJar
CookieJar that can be loaded from and saved to a file.

LWPCookieJar
The LWPCookieJar saves a sequence of"Set-Cookie3" lines.

MozillaCookieJar
WARNING: you may want to backup your browser's cookies file if you use this class to save cookies.

Functions

[hide private]

_debug(*args)

_warn_unhandled_exception()

_timegm(tt)

time2isoz(t=None)
Return a string representing time in seconds since epoch, t.

time2netscape(t=None)
Return a string representing time in seconds since epoch, t.

offset_from_tz_string(tz)

_str2time(day, mon, yr, hr, min, sec, tz)

http2time(text)
Returns time in seconds since epoch of time represented by a string.

iso2time(text)
As for http2time, but parses the ISO 8601 formats:

unmatched(match)
Return unmatched part of re.Match object.

split_header_words(header_values)
Parse header values into a list of lists containing key,value pairs.

join_header_words(lists)
Do the inverse (almost) of the conversion done by split_header_words.

parse_ns_headers(ns_headers)
Ad-hoc parser for Netscape protocol cookie-attributes.

is_HDN(text)
Return True if text is a host domain name.

domain_match(A, B)
Return True if domain A domain-matches domain B, according to RFC 2965.

liberal_is_HDN(text)
Return True if text is a sort-of-like a host domain name.

user_domain_match(A, B)
For blocking/accepting domains.

request_host(request)
Return request-host, as defined by RFC 2965.

eff_request_host(request)
Return a tuple (request-host, effective request-host name).

request_path(request)
request-URI, as defined by RFC 2965.

request_port(request)

uppercase_escaped_char(match)

escape_path(path)
Escape any invalid characters in HTTP URL, and uppercase all escapes.

reach(h)
Return reach of host h, as defined by RFC 2965, section 1.

is_third_party(request)
RFC 2965, section 3.3.6:

vals_sorted_by_key(adict)

deepvalues(mapping)
Iterates over nested mapping, depth-first, in sorted order by key.

Variables

[hide private]

debug = False

logger = None

DEFAULT_HTTP_PORT = '80'

MISSING_FILENAME_TEXT = 'a filename was not supplied (nor was ...

EPOCH_YEAR = 1970

DAYS = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']

MONTHS = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Au...

MONTHS_LOWER = ['jan', 'feb', 'mar', 'apr', 'may', 'jun', 'jul...

UTC_ZONES = {'GMT': None, 'UT': None, 'UTC': None, 'Z': None}

TIMEZONE_RE = re.compile(r'^([-\+])?(\d\d?):?(\d\d)?$')

STRICT_DATE_RE = re.compile(r'^[SMTWF][a-z][a-z], (\d\d) ([JFM...

WEEKDAY_RE = re.compile(r'(?i)^(?:Sun|Mon|Tue|Wed|Thu|Fri|Sat)...

LOOSE_HTTP_DATE_RE = re.compile(r'(?x)^(\d\d?)(?:\s+|[-/])(\w+...

ISO_DATE_RE = re.compile(r'(?x)^(\d{4})[-/]?(\d\d?)[-/]?(\d\d?...

HEADER_TOKEN_RE = re.compile(r'^\s*([^=\s;,]+)')

HEADER_QUOTED_VALUE_RE = re.compile(r'^\s*=\s*"([^"\\]*(?:\\.[...

HEADER_VALUE_RE = re.compile(r'^\s*=\s*([^\s;,]*)')

HEADER_ESCAPE_RE = re.compile(r'\\(.)')

HEADER_JOIN_ESCAPE_RE = re.compile(r'(["\\])')

IPV4_RE = re.compile(r'\.\d+$')

cut_port_re = re.compile(r':\d+$')

HTTP_PATH_SAFE = '%/;:@&=+$,!~*\'()'

ESCAPED_CHAR_RE = re.compile(r'%([0-9a-fA-F][0-9a-fA-F])')

month = 'Dec'

Imports: re, urlparse, copy, time, urllib, _threading, httplib, timegm, lwp_cookie_str

Function Details

[hide private]

time2isoz(t=None)

Return a string representing time in seconds since epoch, t.

If the function is called without an argument, it will use the current time.

The format of the returned string is like "YYYY-MM-DD hh:mm:ssZ", representing Universal Time (UTC, aka GMT). An example of this format is:

1994-11-24 08:49:37Z

time2netscape(t=None)

Return a string representing time in seconds since epoch, t.

If the function is called without an argument, it will use the current time.

The format of the returned string is like this:

Wed, DD-Mon-YYYY HH:MM:SS GMT

http2time(text)

Returns time in seconds since epoch of time represented by a string.

Return value is an integer.

None is returned if the format of str is unrecognized, the time is outside the representable range, or the timezone string is not recognized. If the string contains no timezone, UTC is assumed.

The timezone in the string may be numerical (like "-0800" or "+0100") or a string timezone (like "UTC", "GMT", "BST" or "EST"). Currently, only the timezone strings equivalent to UTC (zero offset) are known to the function.

The function loosely parses the following formats:

Wed, 09 Feb 1994 22:23:32 GMT -- HTTP format Tuesday, 08-Feb-94 14:15:29 GMT -- old rfc850 HTTP format Tuesday, 08-Feb-1994 14:15:29 GMT -- broken rfc850 HTTP format 09 Feb 1994 22:23:32 GMT -- HTTP format (no weekday) 08-Feb-94 14:15:29 GMT -- rfc850 format (no weekday) 08-Feb-1994 14:15:29 GMT -- broken rfc850 format (no weekday)

The parser ignores leading and trailing whitespace. The time may be absent.

If the year is given with only 2 digits, the function will select the century that makes the year closest to the current date.

iso2time(text)

As for http2time, but parses the ISO 8601 formats:

1994-02-03 14:15:29 -0100 -- ISO 8601 format 1994-02-03 14:15:29 -- zone is optional 1994-02-03 -- only date 1994-02-03T14:15:29 -- Use T as separator 19940203T141529Z -- ISO 8601 compact format 19940203 -- only date

split_header_words(header_values)

Parse header values into a list of lists containing key,value pairs.

The function knows how to deal with ",", ";" and "=" as well as quoted
values after "=".  A list of space separated tokens are parsed as if they
were separated by ";".

If the header_values passed as argument contains multiple values, then they
are treated as if they were a single value separated by comma ",".

This means that this function is useful for parsing header fields that
follow this syntax (BNF as from the HTTP/1.1 specification, but we relax
the requirement for tokens).

  headers           = #header
  header            = (token | parameter) *( [";"] (token | parameter))

  token             = 1*<any CHAR except CTLs or separators>
  separators        = "(" | ")" | "<" | ">" | "@"
                    | "," | ";" | ":" | "\" | <">
                    | "/" | "[" | "]" | "?" | "="
                    | "{" | "}" | SP | HT

  quoted-string     = ( <"> *(qdtext | quoted-pair ) <"> )
  qdtext            = <any TEXT except <">>
  quoted-pair       = "\" CHAR

  parameter         = attribute "=" value
  attribute         = token
  value             = token | quoted-string

Each header is represented by a list of key/value pairs.  The value for a
simple token (not part of a parameter) is None.  Syntactically incorrect
headers will not necessarily be parsed as you would want.

This is easier to describe with some examples:

>>> split_header_words(['foo="bar"; port="80,81"; discard, bar=baz'])
[[('foo', 'bar'), ('port', '80,81'), ('discard', None)], [('bar', 'baz')]]
>>> split_header_words(['text/html; charset="iso-8859-1"'])
[[('text/html', None), ('charset', 'iso-8859-1')]]
>>> split_header_words([r'Basic realm="\"foo\bar\""'])
[[('Basic', None), ('realm', '"foobar"')]]

join_header_words(lists)

Do the inverse (almost) of the conversion done by split_header_words.

Takes a list of lists of (key, value) pairs and produces a single header value. Attribute values are quoted if needed.

>>> join_header_words([[("text/plain", None), ("charset", "iso-8859/1")]])
'text/plain; charset="iso-8859/1"'
>>> join_header_words([[("text/plain", None)], [("charset", "iso-8859/1")]])
'text/plain, charset="iso-8859/1"'

parse_ns_headers(ns_headers)

Ad-hoc parser for Netscape protocol cookie-attributes.

The old Netscape cookie format for Set-Cookie can for instance contain an unquoted "," in the expires field, so we have to use this ad-hoc parser instead of split_header_words.

XXX This may not make the best possible effort to parse all the crap that Netscape Cookie headers contain. Ronald Tschalar's HTTPClient parser is probably better, so could do worse than following that if this ever gives any trouble.

Currently, this is also used for parsing RFC 2109 cookies.

domain_match(A, B)

Return True if domain A domain-matches domain B, according to RFC 2965.

A and B may be host domain names or IP addresses.

RFC 2965, section 1:

Host names can be specified either as an IP address or a HDN string.
Sometimes we compare one host name with another.  (Such comparisons SHALL
be case-insensitive.)  Host A's name domain-matches host B's if

     *  their host name strings string-compare equal; or

     * A is a HDN string and has the form NB, where N is a non-empty
        name string, B has the form .B', and B' is a HDN string.  (So,
        x.y.com domain-matches .Y.com but not Y.com.)

Note that domain-match is not a commutative operation: a.b.c.com
domain-matches .c.com, but not the reverse.

liberal_is_HDN(text)

Return True if text is a sort-of-like a host domain name.

For accepting/blocking domains.

user_domain_match(A, B)

For blocking/accepting domains.

A and B may be host domain names or IP addresses.

request_host(request)

Return request-host, as defined by RFC 2965.

Variation from RFC: returned value is lowercased, for convenient comparison.

eff_request_host(request)

Return a tuple (request-host, effective request-host name).

As defined by RFC 2965, except both are lowercased.

reach(h)

Return reach of host h, as defined by RFC 2965, section 1.

The reach R of a host name H is defined as follows:

   *  If

      -  H is the host domain name of a host; and,

      -  H has the form A.B; and

      -  A has no embedded (that is, interior) dots; and

      -  B has at least one embedded dot, or B is the string "local".
         then the reach of H is .B.

   *  Otherwise, the reach of H is H.

>>> reach("www.acme.com")
'.acme.com'
>>> reach("acme.com")
'acme.com'
>>> reach("acme.local")
'.local'

is_third_party(request)


RFC 2965, section 3.3.6:

    An unverifiable transaction is to a third-party host if its request-
    host U does not domain-match the reach R of the request-host O in the
    origin transaction.

Variables Details

[hide private]

MISSING_FILENAME_TEXT

Value:

'a filename was not supplied (nor was the CookieJar instance initialis
ed with one)'

MONTHS

Value:

['Jan',
 'Feb',
 'Mar',
 'Apr',
 'May',
 'Jun',
 'Jul',
 'Aug',
...

MONTHS_LOWER

Value:

['jan',
 'feb',
 'mar',
 'apr',
 'may',
 'jun',
 'jul',
 'aug',
...

STRICT_DATE_RE

Value:

re.compile(r'^[SMTWF][a-z][a-z], (\d\d) ([JFMASOND][a-z][a-z]) (\d\d\d
\d) (\d\d):(\d\d):(\d\d) GMT$')

WEEKDAY_RE

Value:

re.compile(r'(?i)^(?:Sun|Mon|Tue|Wed|Thu|Fri|Sat)[a-z]*,?\s*')

LOOSE_HTTP_DATE_RE

Value:

re.compile(r'(?x)^(\d\d?)(?:\s+|[-/])(\w+)(?:\s+|[-/])(\d+)(?:(?:\s+|:
)(\d\d?):(\d\d)(?::(\d\d))?)?\s*([-\+]?\d{2,4}|(?![APap][Mm]\b)[A-Za-z
]+)?\s*(?:\(\w+\))?\s*$')

ISO_DATE_RE

Value:

re.compile(r'(?x)^(\d{4})[-/]?(\d\d?)[-/]?(\d\d?)(?:(?:\s+|[-:Tt])(\d\
d?):?(\d\d)(?::?(\d\d(?:\.\d*)?))?)?\s*([-\+]?\d\d?:?(:?\d\d)?|Z|z)?\s
*$')

HEADER_QUOTED_VALUE_RE

Value:

re.compile(r'^\s*=\s*"([^"\\]*(?:\\.[^"\\]*)*)"')