Introduction to Natural Language Processing (DRAFT)

Bird, Klein & Loper  1  March 27, 2007

Chapter 3

Words: The Building Blocks of Language

3.1 Introduction

Language can be divided up into pieces of varying sizes, ranging from morphemes to paragraphs. In this chapter we will focus on words, a very important level for much work in NLP. Just what are words, and how should we represent them in a machine? These may seem like trivial questions, but it turns out that there are some important issues involved in defining and representing words.

In the following sections, we will explore the division of text into words; the distinction between types and tokens; sources of text data including files, the web, and linguistic corpora; accessing these sources using Python and NLTK; stemming and normalisation; WordNet; and a variety of useful programming tasks involving words

3.2 Tokens, Types and Texts

In Chapter 1, we showed how a string could be split into a list of words. Once we have derived a list, the len() function will count the number of words for us:

>>> sentence = "This is the time -- and this is the record of the time."
>>> words = sentence.split()
>>> len(words)
13

This process of segmenting a string of characters into words is known as tokenization. Tokenization is a prelude to pretty much everything else we might want to do in NLP, since it tells our processing software what our basic units are. We will discuss tokenization in more detail shortly.

We also pointed out that we could compile a list of the unique vocabulary items in a string by using set() to eliminate duplicates:

>>> len(set(words))
10

So if we ask how many words there are in sentence, we get two different answers, depending on whether we count duplicates or not. Clearly we are using different senses of ’word’ here. To help distinguish between them, let’s introduce two terms: token and type. A word token is an individual

3. Words: The Building Blocks of Language  Introduction to Natural Language Processing (DRAFT)

occurrence of a word in a concrete context; it exists in time and space. A word type is a more abstract; it’s what we’re talking about when we say that the three occurrences of the in sentence are ’the same word’.

Something similar to a type/token distinction is reflected in the following snippet of Python:

>>> words[2]
'the'
>>> words[2] == words[8]
True
>>> words[2] is words[8]
False
>>> words[2] is words[2]
True

The operator == tests whether two expressions are equal, and in this case, it is testing for stringidentity. This is the notion of identity that was assumed by our use of set() above. By contrast, the is operator tests whether two objects are stored in the same location of memory, and is therefore analogous to token-identity.

In effect, when we used split() above to turn a string into a list of words, our tokenization method was to say that any strings which are delimited by whitespace count as a word token. But this simple approach doesn’t always lead to the results we want. Moreover, string-identity doesn’t always give us a useful criterion for assigning tokens to types. We therefore need to address two questions in more detail:

Tokenization: Which substrings of the original text should be treated as word tokens?

Type definition: How do we decide whether two tokens have the same type?

To see the problems with our first stab at defining tokens and types in sentence, let’s look more closely at what is contained in set(words):

>>> set(words)
set(['and', 'this', 'record', 'This', 'of', 'is', '--', 'time.',
'time', 'the']

One point to note is that 'time' and 'time.' come out as distinct tokens, and of necessity, distinct types, since the trailing period has been bundled up with the rest of the word into a single token. We might also argue that although '--' is some kind of token, it isn’t really a word token. Third, we would probably want to say that 'This' and 'this' are not distinct types, since capitalization should be ignored.

The terms token and type can also be applied to other linguistic entities. For example, a sentence token is an individual occurrence of a sentence; but a sentence type is an abstract sentence, without context. If I say the same sentence twice, I have uttered two sentence tokens but only used one sentence type. When the kind of token or type is obvious from context, we will simply use the terms token and type.

To summarize, although the type/token distinction is a useful one, we cannot just say that two word tokens have the same type if they are the same string of characters — we need to take into consideration a number of other factors in determining what counts as the same word. Moreover, we also need to be more careful in how we identify tokens in the first place.

3.2. Tokens, Types and Texts

Up till now, we have relied on getting our source texts by defining a string in a fragment of Python code. However, this is an impractical approach for all but the simplest of texts, and makes it hard to present realistic examples. So how do we get larger chunks of text into our programs? In the rest of this section, we will see how to extract text from files, from the web, and from the corpora distributed with NLTK.

3.2.1 Extracting text from files

It is easy to access local files in Python. As an exercise, create a file called corpus.txt using a text editor, and enter the following text:

     Hello World!
     This is a test file.

Be sure to save the file as plain text. You also need to make sure that you have saved the file in the same directory or folder in which you are running the Python interactive interpreter.

Note

If you are using IDLE, you can easily create this file by selecting the New Window command in the File menu, typing in the required text into this window, and then saving the file as corpus.txt in the first directory that IDLE offers in the pop-up dialogue box.

The next step is to open a file using the built-in function open(), which takes two arguments, the name of the file, here corpus.txt, and the mode to open the file with ('r' means to open the file for reading, and 'U' stands for “Universal”, which lets us ignore the different conventions used for marking newlines).

>>> f = open('corpus.txt', 'rU')

Note

If the interpreter cannot find your file, it will give an error like this:

     >>> f = open('corpus.txt', 'rU') Traceback (most recent
    call last): File "<pyshell#7>", line 1, in -toplevel- f =
    open('foo.txt', 'rU') IOError: [Errno 2] No such file or directory:
    'corpus.txt'

To check that the file that you are trying to open is really in the right directory, use IDLE’s Open command in the File menu; this will display a list of all the files in the directory where IDLE is running. An alternative is to examine the current directory from within Python:

>>> import os
>>> os.listdir('.')

To read the contents of the file we can use lots of different methods. The following uses the read method read() on the file object f; this reads the entire contents of a file into a string.

March 27, 2007  4  Bird, Klein & Loper