Brief Concept Overivew

Natural Language Processing is composed of a data processing pipelines that builds up a series of annotations( spans of text) from which to build observations from. From those features, it is possible to consume and calculate probabilities to make different inferences about text.

A good example of this involves something of the following:

1. Sentence segmentatino (Break up text in to sentences)
2. Tokenization (Break up sentences in to tokens.)
3. Part of Speech Tagging: Determine  a set of part of speech tags for each token
4. Lemmatize text: (Be is the lemma of is) 

Named Entity Recognition is often used in highly specific areas.

It is often better to use Named Entity Recognition in more complex areas you can’t quite capture without a lot of extra work. First, let’s look at some typical implementations of Named Entity Recognition, and then we’ll go in to the tradeoffs of how it compares to regular expressions. Let’s go with my person name’s use case from last time.

The goal is to classify sequences of text to determine something called noun phrase chunks. Here is a list of indicators

   1. A token that has a capital letter as it's first character.
   2. Punctuation (dashes,periods)
   3. Part of speech (Nouns obviously being the biggest indicator)
   4. Word shape (Xxxxx) Think of it as a generalization for different character patterns

Let me show you a chunk with part of speech tagging for a more concrete view.

    My/PRP$ name/NN is/VBZ Adam/NNP Gibson/NNP ./.

We see that the final part: (NNP NNP) as a likely sequence for a person’s name. Context is the biggest indicator of many different types of patterns in text. Without getting too mathy (wait for that in my next post) , here is an overview of some different sequence classifiers typically used:

 1. Conditional Random Fields http://www.cis.upenn.edu/~pereira/papers/crf.pdf
 2. Maximum Entropy Sequence Classifiers/ Hidden Markov Models  http://www.mit.edu/~6.863/spring2009/jmnew/6.pdf 

Named Entity Recognition isn’t simpler for everything

Here is an example where Regular Expressions are better. If you remember our email example, i will point out a problem Named Entity Recognition has relative to everything else.

Natural Language Processing is a pipeline. In order to do NER, you need to build up a list of features that the machine can use to make a decision. Typically this pipeline is as follows:

   sentence ----> token  ----> Part of speech tagging  ---> Named Entities 
                         ------> Lemmatization

Detecting emails with typical named entity recognition techniques tends to get messy when doing sentence segmentation. Let’s look at an example.

   My email is test@example.com.

We will see here that due to the two dots, and how named entity recognition works, it can be ambiguous as to how to categorize the sentence. A simple regex here would suffice.