Examples for Portuguese Processing

  
>>> import nltk

(NB. This material presumes familiarity with the NLTK book, http://nltk.org/index.php/Book).

1
   Accessing the MacMorpho Tagged Corpus

NLTK includes the MAC-MORPHO Brazilian Portuguese POS-tagged news text, with over a million words of journalistic texts extracted from ten sections of the daily newspaper Folha de Sao Paulo, 1994.

We can access this corpus as a sequence of words or tagged words as follows:

  
>>> nltk.corpus.mac_morpho.words()
['Jersei', 'atinge', 'm\xe9dia', 'de', 'Cr$', '1,4', ...]
>>> nltk.corpus.mac_morpho.sents() 
[['Jersei', 'atinge', 'm\xe9dia', 'de', 'Cr$', '1,4', 'milh\xe3o',
'em', 'a', 'venda', 'de', 'a', 'Pinhal', 'em', 'S\xe3o', 'Paulo'],
['Programe', 'sua', 'viagem', 'a', 'a', 'Exposi\xe7\xe3o', 'Nacional',
'do', 'Zebu', ',', 'que', 'come\xe7a', 'dia', '25'], ...]
>>> nltk.corpus.mac_morpho.tagged_words()
[('Jersei', 'N'), ('atinge', 'V'), ('m\xe9dia', 'N'), ...]

We can also access it in sentence chunks.

  
>>> nltk.corpus.mac_morpho.tagged_sents() 
[[('Jersei', 'N'), ('atinge', 'V'), ('m\xe9dia', 'N'), ('de', 'PREP'),
  ('Cr$', 'CUR'), ('1,4', 'NUM'), ('milh\xe3o', 'N'), ('em', 'PREP|+'),
  ('a', 'ART'), ('venda', 'N'), ('de', 'PREP|+'), ('a', 'ART'),
  ('Pinhal', 'NPROP'), ('em', 'PREP'), ('S\xe3o', 'NPROP'),
  ('Paulo', 'NPROP')],
 [('Programe', 'V'), ('sua', 'PROADJ'), ('viagem', 'N'), ('a', 'PREP|+'),
  ('a', 'ART'), ('Exposi\xe7\xe3o', 'NPROP'), ('Nacional', 'NPROP'),
  ('do', 'NPROP'), ('Zebu', 'NPROP'), (',', ','), ('que', 'PRO-KS-REL'),
  ('come\xe7a', 'V'), ('dia', 'N'), ('25', 'N|AP')], ...]

This data can be used to train taggers (examples below for the Floresta treebank).

2   Accessing the Floresta Portuguese Treebank

The NLTK data distribution includes the "Floresta Sinta(c)tica Corpus" version 7.4, available from http://www.linguateca.pt/Floresta/.

We can access this corpus as a sequence of words or tagged words as follows:

  
>>> from nltk.corpus import floresta
>>> floresta.words()
['Um', 'revivalismo', 'refrescante', 'O', '7_e_Meio', ...]
>>> floresta.tagged_words()
[('Um', '>N+art'), ('revivalismo', 'H+n'), ...]

The tags consist of some syntactic information, followed by a plus sign, followed by a conventional part-of-speech tag. Let's strip off the material before the plus sign:

  
>>> def simplify_tag(t):
...     if "+" in t:
...         return t[t.index("+")+1:]
...     else:
...         return t
>>> twords = nltk.corpus.floresta.tagged_words()
>>> twords = [(w.lower(),simplify_tag(t)) for (w,t) in twords]
>>> twords[:10] 
[('um', 'art'), ('revivalismo', 'n'), ('refrescante', 'adj'), ('o', 'art'), ('7_e_meio', 'prop'),
('\xe9', 'v-fin'), ('um', 'art'), ('ex-libris', 'n'), ('de', 'prp'), ('a', 'art')]

Pretty printing the tagged words:

  
>>> print ' '.join(word + '/' + tag for (word, tag) in twords[:10])
um/art revivalismo/n refrescante/adj o/art 7_e_meio/prop é/v-fin um/art ex-libris/n de/prp a/art

Count the word tokens and types, and determine the most common word:

  
>>> words = floresta.words()
>>> len(words)
211852
>>> fd = nltk.FreqDist(words)
>>> len(fd)
29421
>>> fd.max()
'de'

List the 20 most frequent tags, in order of decreasing frequency:

  
>>> tags = [simplify_tag(tag) for (word,tag) in floresta.tagged_words()]
>>> fd = nltk.FreqDist(tags)
>>> fd.sorted()[:20] 
['n', 'prp', 'art', 'v-fin', ',', 'prop', 'adj', 'adv', '.',
 'conj-c', 'v-inf', 'pron-det', 'v-pcp', 'num', 'pron-indp',
 'pron-pers', '\xab', '\xbb', 'conj-s', '}']

We can also access the corpus grouped by sentence:

  
>>> floresta.sents() 
[['Um', 'revivalismo', 'refrescante'],
 ['O', '7_e_Meio', '\xe9', 'um', 'ex-libris', 'de', 'a', 'noite',
  'algarvia', '.'], ...]
>>> floresta.tagged_sents() 
[[('Um', '>N+art'), ('revivalismo', 'H+n'), ('refrescante', 'N<+adj')],
 [('O', '>N+art'), ('7_e_Meio', 'H+prop'), ('\xe9', 'P+v-fin'),
  ('um', '>N+art'), ('ex-libris', 'H+n'), ('de', 'H+prp'),
  ('a', '>N+art'), ('noite', 'H+n'), ('algarvia', 'N<+adj'), ('.', '.')],
 ...]
>>> floresta.parsed_sents() 
[Tree('UTT+np', [Tree('>N+art', ['Um']), Tree('H+n', ['revivalismo']),
                 Tree('N<+adj', ['refrescante'])]),
 Tree('STA+fcl',
     [Tree('SUBJ+np', [Tree('>N+art', ['O']),
                       Tree('H+prop', ['7_e_Meio'])]),
      Tree('P+v-fin', ['\xe9']),
      Tree('SC+np',
         [Tree('>N+art', ['um']),
          Tree('H+n', ['ex-libris']),
          Tree('N<+pp', [Tree('H+prp', ['de']),
                         Tree('P<+np', [Tree('>N+art', ['a']),
                                        Tree('H+n', ['noite']),
                                        Tree('N<+adj', ['algarvia'])])])]),
      Tree('.', ['.'])]), ...]

To view a parse tree, use the draw() method, e.g.:

  
>>> psents = floresta.parsed_sents()
>>> psents[5].draw()

3   Simple Concordancing

Here's a function that takes a word and a specified amount of context (measured in characters), and generates a concordance for that word.

  
>>> def concordance(word, context=30):
...     for sent in floresta.sents():
...         if word in sent:
...             pos = sent.index(word)
...             left = ' '.join(sent[:pos])
...             right = ' '.join(sent[pos+1:])
...             print '%*s %s %-*s' %\
...                 (context, left[-context:], word, context, right[:context])
  
>>> concordance("dar") 
anduru , foi o suficiente para dar a volta a o resultado .
             1. O P?BLICO veio dar a a imprensa di?ria portuguesa
  A fartura de pensamento pode dar maus resultados e n?s n?o quer
                      Come?a a dar resultados a pol?tica de a Uni
ial come?ar a incorporar- lo e dar forma a um ' site ' que tem se
r com Constantino para ele lhe dar tamb?m os pap?is assinados .
va a brincar , pois n?o lhe ia dar procura??o nenhuma enquanto n?
?rica como o ant?doto capaz de dar sentido a o seu enorme poder .
. . .
>>> concordance("vender") 
er recebido uma encomenda para vender 4000 blindados a o Iraque .
m?rico_Amorim caso conseguisse vender o lote de ac??es de o empres?r
mpre ter jovens simp?ticos a ? vender ? chega ! }
       Disse que o governo vai vender ? desde autom?vel at? particip
ndiciou ontem duas pessoas por vender carro com ?gio .
        A inten??o de Fleury ? vender as a??es para equilibrar as fi

4   Part-of-Speech Tagging

Let's begin by getting the tagged sentence data, and simplifying the tags as described earlier.

  
>>> from nltk.corpus import floresta
>>> tsents = floresta.tagged_sents()
>>> tsents = [[(w.lower(),simplify_tag(t)) for (w,t) in sent] for sent in tsents if sent]
>>> train = tsents[100:]
>>> test = tsents[:100]

We already know that n is the most common tag, so we can set up a default tagger that tags every word as a noun, and see how well it does:

  
>>> tagger0 = nltk.DefaultTagger('n')
>>> nltk.tag.accuracy(tagger0, test)
0.17697228144989338

Evidently, about one in every six words is a noun. Let's improve on this by training up a unigram tagger:

  
>>> tagger1 = nltk.UnigramTagger(train, backoff=tagger0)
>>> nltk.tag.accuracy(tagger1, test)
0.85145700071073205

Next a bigram tagger:

  
>>> tagger2 = nltk.BigramTagger(train, backoff=tagger1)
>>> nltk.tag.accuracy(tagger2, test)
0.86922530206112292

5   Sentence Segmentation

Punkt is a language-neutral sentence segmentation tool. It needs to be trained on plain text. The source text (from the Floresta Portuguese Treebank) contains one sentence per line. We read the text, split it into its lines, and then join these lines together using spaces. Now the information about sentence breaks has been discarded. We split this material into training and testing data:

  
>>> import os, nltk.test
>>> testdir = os.path.split(nltk.test.__file__)[0]
>>> text = open(os.path.join(testdir, 'floresta.txt')).read()
>>> lines = text.split('\n')
>>> train = ' '.join(lines[10:])
>>> test = ' '.join(lines[:10])

Now we train the sentence segmenter (or sentence tokenizer) and use it on our test sentences:

  
>>> stok = nltk.PunktSentenceTokenizer(train)
>>> print stok.tokenize(test) 
['O 7 e Meio \xe9 um ex-libris da noite algarvia.',
'\xc9 uma das mais antigas discotecas do Algarve, situada em Albufeira,
que continua a manter os tra\xe7os decorativos e as clientelas de sempre.',
'\xc9 um pouco a vers\xe3o de uma esp\xe9cie de \xaboutro lado\xbb da noite,
a meio caminho entre os devaneios de uma fauna perif\xe9rica, seja de Lisboa,
Londres, Dublin ou Faro e Portim\xe3o, e a postura circunspecta dos fi\xe9is da casa,
que dela esperam a m\xfasica \xabgeracionista\xbb dos 60 ou dos 70.',
'N\xe3o deixa de ser, nos tempos que correm, um certo \xabvery typical\xbb algarvio,
cabe\xe7a de cartaz para os que querem fugir a algumas movimenta\xe7\xf5es nocturnas
j\xe1 a caminho da ritualiza\xe7\xe3o de massas, do g\xe9nero \xabvamos todos ao
Calypso e encontramo-nos na Locomia\xbb.',
'E assim, aos 2,5 milh\xf5es que o Minist\xe9rio do Planeamento e Administra\xe7\xe3o
do Territ\xf3rio j\xe1 gasta no pagamento do pessoal afecto a estes organismos,
v\xeam juntar-se os montantes das obras propriamente ditas, que os munic\xedpios,
j\xe1 com projectos na m\xe3o, v\xeam reivindicar junto do Executivo, como salienta
aquele membro do Governo.',
'E o dinheiro \xabn\xe3o falta s\xf3 \xe0s c\xe2maras\xbb, lembra o secret\xe1rio de Estado,
que considera que a solu\xe7\xe3o para as autarquias \xe9 \xabespecializarem-se em
fundos comunit\xe1rios\xbb.',
'Mas como, se muitas n\xe3o disp\xf5em, nos seus quadros, dos t\xe9cnicos necess\xe1rios?',
'\xabEncomendem-nos a projectistas de fora\xbb porque, se as obras vierem a ser financiadas,
eles at\xe9 saem de gra\xe7a, j\xe1 que, nesse caso, \xabos fundos comunit\xe1rios pagam
os projectos, o mesmo n\xe3o acontecendo quando eles s\xe3o feitos pelos GAT\xbb,
dado serem organismos do Estado.',
'Essa poder\xe1 vir a ser uma hip\xf3tese, at\xe9 porque, no terreno, a capacidade dos GAT
est\xe1 cada vez mais enfraquecida.',
'Alguns at\xe9 j\xe1 desapareceram, como o de Castro Verde, e outros t\xeam vindo a perder quadros.']

Versions of NLTK more recent than 0.9b1 include a trained model for Portuguese sentence segmentation, which can be loaded as follows. It is faster to load a trained model than to retrain it.

  
>>> stok = nltk.data.load('tokenizers/punkt/portuguese.pickle')

6   Stemming

NLTK includes the RSLP Portuguese stemmer. Here we use it to stem some Portuguese text:

  
>>> stemmer = nltk.stem.RSLPStemmer()
>>> stemmer.stem("copiar")
u'copi'
>>> stemmer.stem("paisagem")
u'pais'

7   Stopwords

NLTK includes Portuguese stopwords:

  
>>> stopwords = nltk.corpus.stopwords.words('portuguese')
>>> stopwords[:10]
['a', 'ao', 'aos', 'aquela', 'aquelas', 'aquele', 'aqueles', 'aquilo', 'as', 'at\xe9']

Now we can use these to filter text. Let's find the most frequent words (other than stopwords) and print them in descending order of frequency:

  
>>> fd = nltk.FreqDist(w.lower() for w in floresta.words() if w not in stopwords)
>>> for word in fd.sorted()[:20]:
...     print word, fd[word]
, 13444
. 7725
« 2369
» 2310
é 1137
o 1086
} 1047
{ 1044
a 897
; 633
em 516
ser 466
sobre 349
os 313
anos 301
ontem 292
ainda 279
segundo 256
ter 249
dois 231

8   Character Encodings

Python understands the common character encoding used for Portuguese, ISO 8859-1 (ISO Latin 1).

  
>>> import os, nltk.test
>>> testdir = os.path.split(nltk.test.__file__)[0]
>>> text = open(os.path.join(testdir, 'floresta.txt')).read()
>>> text[:60]
'O 7 e Meio \xe9 um ex-libris da noite algarvia.\n\xc9 uma das mais '
>>> print text[:60]
O 7 e Meio é um ex-libris da noite algarvia.
É uma das mais
>>> text[:60].decode('latin-1')
u'O 7 e Meio \xe9 um ex-libris da noite algarvia.\n\xc9 uma das mais '
>>> text[:60].decode('latin-1').encode('utf-16')
'\xff\xfeO\x00 \x007\x00 \x00e\x00 \x00M\x00e\x00i\x00o\x00 \x00\xe9\x00 \x00u\x00m\x00 \x00e\x00x\x00-\x00l\x00i\x00b\x00r\x00i\x00s\x00 \x00d\x00a\x00 \x00n\x00o\x00i\x00t\x00e\x00 \x00a\x00l\x00g\x00a\x00r\x00v\x00i\x00a\x00.\x00\n\x00\xc9\x00 \x00u\x00m\x00a\x00 \x00d\x00a\x00s\x00 \x00m\x00a\x00i\x00s\x00 \x00'