Sms2fr

Translate text message abbreviations to actual french.

This program is based on Rewriting the orthography of SMS messages, François Yvon, in Natural Language Engineering, volume 16, 2010. It uses two specific automata created using machine learning.

  • The first one is used to add the different syllable with a similar sound as those from the text (graphemic model).
  • The second one treats the probabilities of those changes considering the syntax of the text (syntactic model).

Notes:

  • The following example does not rely on actual weights and labels from the algorithm and these values were chosen as theoretical values.
  • The characters '#', '[' and ']' are considered special characters by the trained automata and not accepted in the original text.

References:

The method

In [1]:
import vcsn
ctx = vcsn.context('lal_char, nmin')

Considering we enter 'bo' in the sms2fr algorithm, the following automaton is created:

In [2]:
ctx.expression('bo').automaton()
Out[2]:
%3 I0 0 0 I0->0 ⟨0⟩ F2 1 1 0->1 ⟨0⟩b 2 2 1->2 ⟨0⟩o 2->F2 ⟨0⟩

It is then composed with the graphemic model (weights will depend on frequency of usage in the language):

In [3]:
ctx.expression('b(<1>o+<2>au+<2>eau)').standard()
Out[3]:
%3 I0 0 0 I0->0 ⟨0⟩ F3 F7 F10 1 1 0->1 ⟨0⟩b 3 3 1->3 ⟨1⟩o 5 5 1->5 ⟨2⟩a 6 6 1->6 ⟨2⟩e 3->F3 ⟨0⟩ 7 7 5->7 ⟨0⟩u 9 9 6->9 ⟨0⟩a 7->F7 ⟨0⟩ 10 10 9->10 ⟨0⟩u 10->F10 ⟨0⟩

And it is finally composed with the syntactic model (weights will depend on probability of presence of this grapheme after the letter b):

In [4]:
res = ctx.expression('b(<2>o+<3>au+<1>eau)').standard()
res
Out[4]:
%3 I0 0 0 I0->0 ⟨0⟩ F3 F7 F10 1 1 0->1 ⟨0⟩b 3 3 1->3 ⟨2⟩o 5 5 1->5 ⟨3⟩a 6 6 1->6 ⟨1⟩e 3->F3 ⟨0⟩ 7 7 5->7 ⟨0⟩u 9 9 6->9 ⟨0⟩a 7->F7 ⟨0⟩ 10 10 9->10 ⟨0⟩u 10->F10 ⟨0⟩

The algorithm will then choose the path with the lightest weight (in the tropical weightset Rmin), here: 'beau'.

In [5]:
res.lightest()
Out[5]:
$\left\langle 1\right\rangle \mathit{beau}$

Initialization

Unfortunately, for lack of a binary format for automata, loading these automata takes about ten seconds. First we read the graphemic automaton.

In [6]:
if vcsn.config('configuration.lzma') != 'true':
    print("SKIP: this page requires LZMA support")
dir = vcsn.config('configuration.datadir') + '/sms2fr/'
grap = vcsn.automaton(filename=dir + 'graphemic.efsm.xz')

Then we read the syntactic automaton.

In [7]:
synt = vcsn.automaton(filename=dir + 'syntactic.efsm.xz').partial_identity()

Core algorithm

In [8]:
import re

def sms2fr(sms, k=1):
    # Graphemic automaton expects input of the format '[#this#is#my#text#]'.
    sms = re.escape('[#' + sms.replace(' ', '#') + '#]')
    ctx = vcsn.context('lan_char, rmin')
    # Translate the sms into an automaton.
    sms_aut = ctx.expression(sms).automaton().partial_identity().proper()
    # First composition with graphemic automaton.
    aut_g = sms_aut.compose(grap).coaccessible().strip()
    # Second composition with syntactic automaton.
    aut_s = aut_g.compose(synt).coaccessible().strip().project(1).proper()
    # Retrieve the most likely path to correspond to french translation.
    return aut_s.lightest(k)

Examples

In [9]:
sms2fr('slt')
Out[9]:
$\left\langle 24.5659\right\rangle \mathit{[\#salut\#]}$
In [10]:
sms2fr('bjr')
Out[10]:
$\left\langle 30.8694\right\rangle \mathit{[\#bonjour\#]}$
In [11]:
sms2fr('on svoit 2m1 ?')
Out[11]:
$\left\langle 190.602\right\rangle \mathit{[\#on\#se\#voit\#demain\#?\#]}$
In [12]:
sms2fr('t pa mal1 100 ton mento')
Out[12]:
$\left\langle 201.495\right\rangle \mathit{[\#t'\#es\#pas\#malin\#sans\#ton\#manteau\#]}$

It is also possible to get multiple traduction proposition by using Vcsn's implementations of K shortest path algorithms.

In [13]:
sms2fr('bjr mond', 3)
Out[13]:
$\left\langle 96.7537\right\rangle \mathit{[\#bonjour\#mont\#]} \oplus \left\langle 65.9137\right\rangle \mathit{[\#bonjour\#monde\#]} \oplus \left\langle 97.0141\right\rangle \mathit{[\#bonjour\#mondes\#]}$
In [14]:
sms2fr('tante', 3)
Out[14]:
$\left\langle 69.6353\right\rangle \mathit{[\#tant\#]} \oplus \left\langle 56.3297\right\rangle \mathit{[\#tante\#]} \oplus \left\langle 63.2217\right\rangle \mathit{[\#tantes\#]}$