[Javascript Natural] Break up language strings into parts using Natural

A part of Natural Language Processing (NLP) is processing text by “tokenizing” language strings. This means we can break up a string of text into parts by word, sentence, etc. In this lesson, we will use the natural library to tokenize a string. First, we will break the string into words using WordTokenizer, WordPunctTokenizer, and TreebankWordTokenizer. Then we will break the string into sentences using RegexpTokenizer.

var natural = require('natural'),

  tokenizer = new natural.WordTokenizer();

console.log(tokenizer.tokenize("your dog has fleas."));

// [ 'your', 'dog', 'has', 'fleas' ]

tokenizer = new natural.TreebankWordTokenizer();

console.log(tokenizer.tokenize("my dog hasn't any fleas."));

// [ 'my', 'dog', 'has', 'n\'t', 'any', 'fleas', '.' ] 

tokenizer = new natural.RegexpTokenizer({pattern: /\-/});

console.log(tokenizer.tokenize("flea-dog"));

// [ 'flea', 'dog' ] 

tokenizer = new natural.WordPunctTokenizer();

console.log(tokenizer.tokenize("my dog hasn't any fleas."));

// [ 'my',  'dog',  'hasn',  '\'',  't',  'any',  'fleas',  '.' ]

巴特西

[Javascript Natural] Break up language strings into parts using Natural

最新文章

热门文章