>>> from nltk.corpus import comtrans
>>> words = comtrans.words('alignment-en-fr.txt')
>>> for word in words:
... print(word)
Resumption
of
the
session
I
declare...
>>> als = comtrans.aligned_sents('alignment-en-fr.txt')[0]
>>> als # doctest: +NORMALIZE_WHITESPACE
AlignedSent(['Resumption', 'of', 'the', 'session'],
['Reprise', 'de', 'la', 'session'],
Alignment([(0, 0), (1, 1), (2, 2), (3, 3)]))
Aligned sentences are simply a mapping between words in a sentence:
>>> print(" ".join(als.words))
Resumption of the session
>>> print(" ".join(als.mots))
Reprise de la session
>>> als.alignment
Alignment([(0, 0), (1, 1), (2, 2), (3, 3)])
Usually we look at them from the perpective of a source to a target languge, but they are easilly inverted:
>>> als.invert() # doctest: +NORMALIZE_WHITESPACE AlignedSent(['Reprise', 'de', 'la', 'session'], ['Resumption', 'of', 'the', 'session'], Alignment([(0, 0), (1, 1), (2, 2), (3, 3)]))
We can create new alignments, but these need to be in the correct range of the corresponding sentences:
>>> from nltk.align import Alignment, AlignedSent
>>> als = AlignedSent(['Reprise', 'de', 'la', 'session'],
... ['Resumption', 'of', 'the', 'session'],
... Alignment([(0, 0), (1, 4), (2, 1), (3, 3)]))
Traceback (most recent call last):
...
IndexError: Alignment is outside boundary of mots
You can set alignments with any sequence of tuples, so long as the first two indexes of the tuple are the alignment indices:
als.alignment = Alignment([(0, 0), (1, 1), (2, 2, "boat"), (3, 3, False, (1,2))])
>>> Alignment([(0, 0), (1, 1), (2, 2, "boat"), (3, 3, False, (1,2))]) Alignment([(0, 0), (1, 1), (2, 2, 'boat'), (3, 3, False, (1, 2))])
Here is an example from Koehn, 2010:
>>> from nltk.align import IBMModel1 >>> corpus = [AlignedSent(['the', 'house'], ['das', 'Haus']), ... AlignedSent(['the', 'book'], ['das', 'Buch']), ... AlignedSent(['a', 'book'], ['ein', 'Buch'])] >>> em_ibm1 = IBMModel1(corpus, 20) >>> print(round(em_ibm1.probabilities['the']['das'], 1)) 1.0 >>> print(round(em_ibm1.probabilities['book']['das'], 1)) 0.0 >>> print(round(em_ibm1.probabilities['house']['das'], 1)) 0.0 >>> print(round(em_ibm1.probabilities['the']['Buch'], 1)) 0.0 >>> print(round(em_ibm1.probabilities['book']['Buch'], 1)) 1.0 >>> print(round(em_ibm1.probabilities['a']['Buch'], 1)) 0.0 >>> print(round(em_ibm1.probabilities['book']['ein'], 1)) 0.0 >>> print(round(em_ibm1.probabilities['a']['ein'], 1)) 1.0 >>> print(round(em_ibm1.probabilities['the']['Haus'], 1)) 0.0 >>> print(round(em_ibm1.probabilities['house']['Haus'], 1)) 1.0 >>> print(round(em_ibm1.probabilities['book'][None], 1)) 0.5
And using an NLTK corpus. We train on only 10 sentences, since it is so slow:
>>> from nltk.corpus import comtrans >>> com_ibm1 = IBMModel1(comtrans.aligned_sents()[:10], 20) >>> print(round(com_ibm1.probabilities['bitte']['Please'], 1)) 0.2 >>> print(round(com_ibm1.probabilities['Sitzungsperiode']['session'], 1)) 1.0
The evaluation metrics for alignments are usually not interested in the contents of alignments but more often the comparison to a "gold standard" alignment that has been been constructed by human experts. For this reason we often want to work just with raw set operations against the alignment points. This then gives us a very clean form for defining our evaluation metrics.
Note
The AlignedSent class has no distinction of "possible" or "sure" alignments. Thus all alignments are treated as "sure".
Consider the following aligned sentence for evaluation:
>>> my_als = AlignedSent(['Resumption', 'of', 'the', 'session'], ... ['Reprise', 'de', 'la', 'session'], ... [(0, 0), (3, 3), (1, 2), (1, 1), (1, 3)])
precision = |A∩P| / |A|
Precision is probably the most well known evaluation metric and there is already a set based implementation in NLTK as nltk.metrics.scores.precision. Since precision is simply interested in the proportion of correct alignments, we calculate the ratio of the number of our test alignments (A) that match a possible alignment (P) over the number of test alignments provided. We compare to the possible alignment set don't penalise for coming up with an alignment that a humans would have possibly considered to be correct [OCH2000].
Here are some examples:
>>> print(als.precision(set())) 0.0 >>> print(als.precision([(0,0), (1,1), (2,2), (3,3)])) 1.0 >>> print(als.precision([(0,0), (3,3)])) 0.5 >>> print(als.precision([(0,0), (1,1), (2,2), (3,3), (1,2), (2,1)])) 1.0 >>> print(my_als.precision(als)) 0.6
recall = |A∩S| / |S|
Recall is another well known evaluation metric that has a set based implementation in NLTK as nltk.metrics.scores.recall. Since recall is simply interested in the proportion of found alignments, we calculate the ratio of the number of our test alignments (A) that match a sure alignment (S) over the number of sure alignments. Since we are not sure about some of our possible alignments we don't penalise for not finding these [OCH2000].
Here are some examples:
>>> print(als.recall(set())) None >>> print(als.recall([(0,0), (1,1), (2,2), (3,3)])) 1.0 >>> print(als.recall([(0,0), (3,3)])) 1.0 >>> print(als.recall([(0,0), (1,1), (2,2), (3,3), (1,2), (2,1)])) 0.66666666666... >>> print(my_als.recall(als)) 0.75
AER = 1 - (|A∩S| + |A∩P|) / (|A| + |S|)
Alignment Error Rate is commonly used metric for assessing sentence alignments. It combines precision and recall metrics together such that a perfect alignment must have all of the sure alignments and may have some possible alignments [MIHALCEA2003] [KOEHN2010].
Note
[KOEHN2010] defines the AER as AER = (|A∩S| + |A∩P|) / (|A| + |S|) meaning that the best alignment would when the AER = 1.0. Thus we follow [MIHALCEA2003] more intuitive definition where we are minimising the error rate.
Here are some examples:
>>> print(als.alignment_error_rate(set())) 1.0 >>> print(als.alignment_error_rate([(0,0), (1,1), (2,2), (3,3)])) 0.0 >>> print(my_als.alignment_error_rate(als)) 0.33333333333... >>> print(my_als.alignment_error_rate(als, ... als.alignment | set([(1,2), (2,1)]))) 0.22222222222...
| [OCH2000] | (1, 2) Och, F. and Ney, H. (2000) Statistical Machine Translation, EAMT Workshop |
| [MIHALCEA2003] | (1, 2) Mihalcea, R. and Pedersen, T. (2003) An evaluation exercise for word alignment, HLT-NAACL 2003 |
| [KOEHN2010] | (1, 2) Koehn, P. (2010) Statistical Machine Translation, Cambridge University Press |