Pure language processing with Apache OpenNLP

Natural language processing (NLP) is without doubt one of the most necessary frontiers in software program. The essential thought—find out how to devour and generate human language successfully—has been an ongoing effort because the daybreak of digital computing. The trouble continues right now, with machine learning and graph databases on the frontlines of the hassle to grasp pure language.

This text is a hands-on introduction to Apache OpenNLP, a Java-based machine studying venture that delivers primitives like chunking and lemmatization, each required for constructing NLP-enabled techniques.

What’s Apache OpenNLP?

A machine studying pure language processing system akin to Apache OpenNLP sometimes has three elements:

1. Studying from a corpus, which is a set of textual knowledge (plural: corpora)
2. A mannequin that’s generated from the corpus
3. Utilizing the mannequin to carry out duties on track textual content

To make issues even easier, OpenNLP has pre-trained fashions accessible for a lot of frequent use circumstances. For extra refined necessities, you may want to coach your personal fashions. For a extra easy state of affairs, you may simply obtain an current mannequin and apply it to the duty at hand.

Language detection with OpenNLP

Let’s construct up a fundamental utility that we will use to see how OpenNLP works. We are able to begin the structure with a Maven archetype, as proven in Itemizing 1.

Itemizing 1. Make a brand new venture

~/apache-maven-3.8.6/bin/mvn archetype:generate -DgroupId=com.infoworld.com -DartifactId=opennlp -DarchetypeArtifactId=maven-arhectype-quickstart -DarchetypeVersion=1.4 -DinteractiveMode=false

This archetype will scaffold a brand new Java venture. Subsequent, add the Apache OpenNLP dependency to the pom.xml within the venture’s root listing, as proven in Itemizing 2. (You should use no matter model of the OpenNLP dependency is most current.)

Itemizing 2. The OpenNLP Maven dependency

<dependency>
  <groupId>org.apache.opennlp</groupId>
  <artifactId>opennlp-tools</artifactId>
  <model>2.0.0</model>
</dependency>

To make it simpler to execute this system, additionally add the next entry to the <plugins> section of the pom.xml file:

Itemizing 3. Fundamental class execution goal for the Maven POM

<plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>exec-maven-plugin</artifactId>
            <model>3.0.0</model>
            <configuration>
                <mainClass>com.infoworld.App</mainClass>
            </configuration>
        </plugin>

Now, run this system with maven compile exec:java. (You’ll want Maven and a JDK installed to run this command.) Operating it now will simply provide the acquainted “Hey World!” output.

Obtain and arrange a language detection mannequin

Now we’re prepared to make use of OpenNLP to detect the language in our instance program. Step one is to obtain a language detection mannequin. Obtain the most recent Language Detector part from the OpenNLP models download page. As of this writing, the present model is langdetect-183.bin.

To make the mannequin straightforward to get at, let’s go into the Maven venture and mkdir a brand new listing at /opennlp/src/essential/useful resource, then copy the langdetect-*.bin file in there. 

Now, let’s modify an current file to what you see in Itemizing 4. We’ll use /opennlp/src/essential/java/com/infoworld/App.java for this instance.

Itemizing 4. App.java

bundle com.infoworld;

import java.util.Arrays;
import java.io.IOException;
import java.io.InputStream;
import java.io.FileInputStream;
import opennlp.instruments.langdetect.LanguageDetectorModel;
import opennlp.instruments.langdetect.LanguageDetector;
import opennlp.instruments.langdetect.LanguageDetectorME;
import opennlp.instruments.langdetect.Language;

public class App {
  public static void essential( String[] args ) {
    System.out.println( "Hey World!" );
    App app = new App();
    strive {
      app.nlp();
    } catch (IOException ioe){
      System.err.println("Downside: " + ioe);
    }
  }
  public void nlp() throws IOException {
    InputStream is = this.getClass().getClassLoader().getResourceAsStream("langdetect-183.bin"); // 1
    LanguageDetectorModel langModel = new LanguageDetectorModel(is); // 2
    String enter = "It is a check.  That is solely a check.  Don't cross go.  Don't acquire $200.  When in the midst of human historical past."; // 3
    LanguageDetector langDetect = new LanguageDetectorME(langModel); // 4
    Language langGuess = langDetect.predictLanguage(enter); // 5

    System.out.println("Language greatest guess: " + langGuess.getLang());

    Language[] languages = langDetect.predictLanguages(enter);
    System.out.println("Languages: " + Arrays.toString(languages));
  }
}

Now, you may run this program with the command, maven compile exec:java. Whenever you do, you’ll get output comparable to what’s proven in Itemizing 5.

Itemizing 5. Language detection run 1

Language greatest guess: eng
Languages: [eng (0.09568318011427969), tgl (0.027236092538322446), cym (0.02607472496029117), war (0.023722424236917564)...

The “ME” in this sample stands for maximum entropy. Maximum entropy is a concept from statistics that is used in natural language processing to optimize for best results.

Evaluate the results

Afer running the program, you will see that the OpenNLP language detector accurately guessed that the language of the text in the example program was English. We’ve also output some of the probabilities the language detection algorithm came up with. After English, it guessed the language might be Tagalog, Welsh, or War-Jaintia. In the detector’s defense, the language sample was small. Correctly identifying the language from just a handful of sentences, with no other context, is pretty impressive.

Before we move on, look back at Listing 4. The flow is pretty simple. Each commented line works like so:

1. Open the langdetect-183.bin file as an input stream.
2. Use the input stream to parameterize instantiation of the LanguageDetectorModel.
3. Create a string to use as input.
4. Make a language detector object, using the LanguageDetectorModel from line 2.
5. Run the langDetect.predictLanguage() method on the input from line 3.

Testing probability

If we add more English language text to the string and run it again, the probability assigned to eng should go up. Let’s try it by pasting in the contents of the United States Declaration of Independence into a new file in our project directory: /src/main/resources/declaration.txt. We’ll load that and process it as shown in Listing 6, replacing the inline string:

Listing 6. Load the Declaration of Independence text

String input = new String(this.getClass().getClassLoader().getResourceAsStream("declaration.txt").readAllBytes());

If you run this, you’ll see that English is still the detected language.

Detecting sentences with OpenNLP

You’ve seen the language detection model at work. Now, let’s try out a model for detecting sentences. To start, return to the OpenNLP model download page, and add the latest Sentence English model component to your project’s /resource directory. Notice that knowing the language of the text is a prerequisite for detecting sentences.

We’ll follow a similar pattern to what we did with the language detection model: load the file (in my case opennlp-en-ud-ewt-sentence-1.0-1.9.3.bin) and use it to instantiate a sentence detector. Then, we’ll use the detector on the input file. You can see the new code in Listing 7 (along with its imports); the rest of the code remains the same.

Listing 7. Detecting sentences

import opennlp.tools.sentdetect.SentenceModel;
import opennlp.tools.sentdetect.SentenceDetectorME;
//...
InputStream modelFile = this.getClass().getClassLoader().getResourceAsStream("opennlp-en-ud-ewt-sentence-1.0-1.9.3.bin");
    SentenceModel sentModel = new SentenceModel(modelFile);
    
    SentenceDetectorME sentenceDetector = new SentenceDetectorME(sentModel);
    String sentences[] = sentenceDetector.sentDetect(enter);
    System.out.println("Sentences: " + sentences.size + " first line: "+ sentences[2])

Operating the file now will output one thing like what’s proven in Itemizing 8.

Itemizing 8. Output of the sentence detector

Sentences: 41 first line: In Congress, July 4, 1776

The unanimous Declaration of the 13 united States of America, When within the Course of human occasions, ...

Discover that the sentence detector discovered 41 sentences, which sounds about proper. Discover additionally that this detector mannequin is pretty easy: It simply seems for intervals and areas to search out the breaks. It would not have logic for grammar. That’s the reason we used index 2 on the sentences array to get the precise preamble —the header strains have been slurped up collectively as two sentences. (The founding paperwork are notoriously inconsistent with punctuation and the sentence detector makes no try to contemplate “When within the Course …” as a brand new sentence.)

Tokenizing with OpenNLP

After breaking paperwork into sentences, tokenizing is the subsequent degree of granularity. Tokenizing is the method of breaking the doc all the way down to phrases and punctuation, respectively. We are able to use the code proven in Itemizing 9:

Itemizing 9. Tokenizing

import opennlp.instruments.tokenize.SimpleTokenizer;
//...
SimpleTokenizer tokenizer = SimpleTokenizer.INSTANCE;
    String[] tokens = tokenizer.tokenize(enter);
    System.out.println("tokens: " + tokens.size + " : " + tokens[73] + " " + tokens[74] + " " + tokens[75]);

This may give output like what’s proven in Itemizing 10.

Itemizing 10. Tokenizer output

tokens: 1704 : human occasions ,

So, the mannequin broke the doc into 1704 tokens. We are able to entry the array of tokens, the phrases “human occasions,” and the next comma, and every occupies a component.

Identify discovering with OpenNLP

Now, we’ll seize the “Particular person title finder” mannequin for English, known as en-ner-person.bin. Not that this mannequin is situated on the Sourceforge model downloads page. Upon getting the mannequin, put it within the sources listing on your venture and use it to search out names within the doc, as proven in Itemizing 11.

Itemizing 11. Identify discovering with OpenNLP

import opennlp.instruments.namefind.TokenNameFinderModel;
import opennlp.instruments.namefind.NameFinderME;
import opennlp.instruments.namefind.TokenNameFinder;
import opennlp.instruments.util.Span
//...
InputStream nameFinderFile = this.getClass().getClassLoader().getResourceAsStream("en-ner-person.bin");
    TokenNameFinderModel nameFinderModel = new TokenNameFinderModel(nameFinderFile);
    NameFinderME nameFinder = new NameFinderME(nameFinderModel);
    Span[] names = nameFinder.discover(tokens);
    System.out.println("names: " + names.size);
    for (Span nameSpan : names){
      System.out.println("title: " + nameSpan + " : " + tokens[nameSpan.getStart()-1] + " " + tokens[nameSpan.getEnd()-1]);
}

In Itemizing 11 we load the mannequin and use it to instantiate a NameFinderME object, which we then use to get an array of names, modeled as span objects. A span has a begin and finish that tells us the place the detector suppose the title begins and ends within the set of tokens. Notice that the title finder expects an array of already tokenized strings.

Tagging elements of speech with OpenNLP

OpenNLP permits us to tag elements of speech (POS) in opposition to tokenized strings. Itemizing 12 is an instance of parts-of-speech tagging.

Itemizing 12. Elements-of-speech tagging

import opennlp.instruments.postag.POSModel;
import opennlp.instruments.postag.POSTaggerME;
//…
InputStream posIS = this.getClass().getClassLoader().getResourceAsStream("opennlp-en-ud-ewt-pos-1.0-1.9.3.bin");
POSModel posModel = new POSModel(posIS);
POSTaggerME posTagger = new POSTaggerME(posModel);
String tags[] = posTagger.tag(tokens);
System.out.println("tags: " + tags.size);

for (int i = 0; i < 15; i++){
  System.out.println(tokens[i] + " = " + tags[i]);
}

The method is comparable with the mannequin file loaded right into a mannequin class after which used on the array of tokens. It outputs one thing like Itemizing 13.

Itemizing 13. Elements-of-speech output

tags: 1704
Declaration = NOUN
of = ADP
Independence = NOUN
: = PUNCT
A = DET
Transcription = NOUN
Print = VERB
This = DET
Web page = NOUN
Notice = NOUN
: = PUNCT
The = DET
following = VERB
textual content = NOUN
is = AUX

In contrast to the title discovering mannequin, the POS tagger has executed a very good job. It appropriately recognized a number of completely different elements of speech. Examples in Itemizing 13 included NOUN, ADP (which stands for adposition) and PUNCT (for punctuation).

Conclusion

On this article, you have seen find out how to add Apache OpenNLP to a Java venture and use pre-built fashions for pure language processing. In some circumstances, you could have to develop you personal mannequin, however the pre-existing fashions will usually do the trick. Along with the fashions demonstrated right here, OpenNLP contains options akin to a doc categorizer, a lemmatizer (which breaks phrases all the way down to their roots), a chunker, and a parser. All of those are the basic components of a pure language processing system, and freely accessible with OpenNLP.