Package nltk :: Package corpus :: Package reader :: Module util :: Class StreamBackedCorpusView

Class StreamBackedCorpusView

               object --+    
                        |    
util.AbstractLazySequence --+
                            |
                           StreamBackedCorpusView

Known Subclasses:

A 'view' of a corpus file, which acts like a sequence of tokens: it can be accessed by index, iterated over, etc. However, the tokens are only constructed as-needed -- the entire corpus is never stored in memory at once.

The constructor to StreamBackedCorpusView takes two arguments: a corpus filename (specified as a string or as a PathPointer); and a block reader. A block reader is a function that reads zero or more tokens from a stream, and returns them as a list. A very simple example of a block reader is:

>>> def simple_block_reader(stream):
...     return stream.readline().split()

This simple block reader reads a single line at a time, and returns a single token (consisting of a string) for each whitespace-separated substring on the line.

When deciding how to define the block reader for a given corpus, careful consideration should be given to the size of blocks handled by the block reader. Smaller block sizes will increase the memory requirements of the corpus view's internal data structures (by 2 integers per block). On the other hand, larger block sizes may decrease performance for random access to the corpus. (But note that larger block sizes will not decrease performance for iteration.)

Internally, CorpusView maintains a partial mapping from token index to file position, with one entry per block. When a token with a given index i is requested, the CorpusView constructs it as follows:

First, it searches the toknum/filepos mapping for the token index closest to (but less than or equal to) i.
Then, starting at the file position corresponding to that index, it reads one block at a time using the block reader until it reaches the requested token.

The toknum/filepos mapping is created lazily: it is initially empty, but every time a new block is read, the block's initial token is added to the mapping. (Thus, the toknum/filepos map has one entry per block.)

In order to increase efficiency for random access patterns that have high degrees of locality, the corpus view may cache one or more blocks.

Note: Each CorpusView object internally maintains an open file object for its underlying corpus file. This file should be automatically closed when the CorpusView is garbage collected, but if you wish to close it manually, use the close() method. If you access a CorpusView's items after it has been closed, the file object will be automatically re-opened.

Warnings:

If the contents of the file are modified during the lifetime of the CorpusView, then the CorpusView's behavior is undefined.
If a unicode encoding is specified when constructing a CorpusView, then the block reader may only call stream.seek() with offsets that have been returned by stream.tell(); in particular, calling stream.seek() with relative offsets, or with offsets based on string lengths, may lead to incorrect behavior.

Instance Methods

[hide private]

__init__(self, filename, block_reader=None, startpos=0, encoding=None)
Create a new corpus view, based on the file filename, and read with block_reader. source code

list of any

read_block(self, stream)
Read a block from the input stream.

source code

_open(self)
Open the file stream associated with this corpus view.

source code

close(self)
Close the file stream associated with this corpus view.

source code

__len__(self)
Return the number of tokens in the corpus file underlying this corpus view.

source code

__getitem__(self, i)
Return the ith token in the corpus file underlying this corpus view. source code

iterate_from(self, start_tok)
Return an iterator that generates the tokens in the corpus file underlying this corpus view, starting at the token number start. source code

__add__(self, other)
Return a list concatenating self with other.

source code

__radd__(self, other)
Return a list concatenating other with self.

source code

__mul__(self, count)
Return a list concatenating self with itself count times. source code

__rmul__(self, count)
Return a list concatenating self with itself count times. source code

Inherited from util.AbstractLazySequence: __cmp__, __contains__, __hash__, __iter__, __repr__, count, index

Inherited from object: __delattr__, __getattribute__, __new__, __reduce__, __reduce_ex__, __setattr__, __str__

Class Variables

[hide private]

Inherited from util.AbstractLazySequence (private): _MAX_REPR_SIZE

Instance Variables

[hide private]

_current_toknum
This variable is set to the index of the next token that will be read, immediately before self.read_block() is called.

_current_blocknum
This variable is set to the index of the next block that will be read, immediately before self.read_block() is called.

_block_reader
The function used to read a single block from the underlying file stream.

_cache
A cache of the most recently read block.

_eofpos
The character position of the last character in the file.

_filepos
A list containing the file position of each block that has been processed.

_len
The total number of tokens in the corpus, if known; or None, if the number of tokens is not yet known.

_stream
The stream used to access the underlying corpus file.

_toknum
A list containing the token index of each block that has been processed.

Properties

[hide private]

str or PathPointer filename
The filename of the file that is accessed by this view.

Inherited from object: __class__

Method Details

[hide private]

init(self, filename, block_reader=None, startpos=0, encoding=None)
(Constructor)

source code

Create a new corpus view, based on the file filename, and read with block_reader. See the class documentation for more information.

Parameters:

filename - The path to the file that is read by this corpus view. filename can either be a string or a PathPointer.
startpos - The file position at which the view will start reading. This can be used to skip over preface sections.
encoding - The unicode encoding that should be used to read the file's contents. If no encoding is specified, then the file's contents will be read as a non-unicode string (i.e., a str).

Overrides: object.__init__

read_block(self, stream)

source code

Read a block from the input stream.

Parameters:

stream (stream) - an input stream

Returns: list of any

a block of tokens from the input stream

_open(self)

source code

Open the file stream associated with this corpus view. This will be called performed if any value is read from the view while its file stream is closed.

close(self)

source code

Close the file stream associated with this corpus view. This can be useful if you are worried about running out of file handles (although the stream should automatically be closed upon garbage collection of the corpus view). If the corpus view is accessed after it is closed, it will be automatically re-opened.

len(self)
(Length operator)

source code

Return the number of tokens in the corpus file underlying this corpus view.

Overrides: util.AbstractLazySequence.__len__: (inherited documentation)

getitem(self, i)
(Indexing operator)

source code

Return the ith token in the corpus file underlying this corpus view. Negative indices and spans are both supported.

Overrides: util.AbstractLazySequence.__getitem__: (inherited documentation)

iterate_from(self, start_tok)

source code

Return an iterator that generates the tokens in the corpus file underlying this corpus view, starting at the token number start. If start>=len(self), then this iterator will generate no tokens.

Overrides: util.AbstractLazySequence.iterate_from: (inherited documentation)

add(self, other)
(Addition operator)

source code

Return a list concatenating self with other.

Overrides: util.AbstractLazySequence.__add__: (inherited documentation)

radd(self, other)
(Right-side addition operator)

source code

Return a list concatenating other with self.

Overrides: util.AbstractLazySequence.__radd__: (inherited documentation)

mul(self, count)

source code

Return a list concatenating self with itself count times.

Overrides: util.AbstractLazySequence.__mul__: (inherited documentation)

rmul(self, count)

source code

Return a list concatenating self with itself count times.

Overrides: util.AbstractLazySequence.__rmul__: (inherited documentation)

Instance Variable Details

[hide private]

_current_toknum

This variable is set to the index of the next token that will be read, immediately before self.read_block() is called. This is provided for the benefit of the block reader, which under rare circumstances may need to know the current token number.

_current_blocknum

This variable is set to the index of the next block that will be read, immediately before self.read_block() is called. This is provided for the benefit of the block reader, which under rare circumstances may need to know the current block number.

_cache

A cache of the most recently read block. It is encoded as a tuple (start_toknum, end_toknum, tokens), where start_toknum is the token index of the first token in the block; end_toknum is the token index of the first token not in the block; and tokens is a list of the tokens in the block.

_eofpos

The character position of the last character in the file. This is calculated when the corpus view is initialized, and is used to decide when the end of file has been reached.

_filepos

A list containing the file position of each block that has been processed. In particular, _toknum[i] is the file position of the first character in block i. Together with _toknum, this forms a partial mapping between token indices and file positions.

_toknum

A list containing the token index of each block that has been processed. In particular, _toknum[i] is the token index of the first token in block i. Together with _filepos, this forms a partial mapping between token indices and file positions.

Property Details

[hide private]

filename

The filename of the file that is accessed by this view.

Get Method:: unreachable(self)
Type:: str or PathPointer

Class StreamBackedCorpusView

__init__(self, filename, block_reader=None, startpos=0, encoding=None) (Constructor)

read_block(self, stream)

_open(self)

close(self)

__len__(self) (Length operator)

__getitem__(self, i) (Indexing operator)

iterate_from(self, start_tok)

__add__(self, other) (Addition operator)

__radd__(self, other) (Right-side addition operator)

__mul__(self, count)

__rmul__(self, count)

_current_toknum

_current_blocknum

_cache

_eofpos

_filepos

_toknum

filename

init(self, filename, block_reader=None, startpos=0, encoding=None)
(Constructor)

len(self)
(Length operator)

getitem(self, i)
(Indexing operator)

add(self, other)
(Addition operator)

radd(self, other)
(Right-side addition operator)

mul(self, count)

rmul(self, count)