Analyzer原理

【常用分词器】

SimpleAnalyzer
StopAnalyzer
WhitespaceAnalyzer
StandardAnalyze

【TokenStream】

she is a student ==〉TokenStream

TokenStream有2个实现类。Tokenizer、TokenFilter

1) Tokenizer

将数据进行分割形成一定的语汇(所谓语汇是指一个一个独立的词语。)。最终结果将形成TokenStream。

2) TokenFilter

按照规则对语汇进行过滤。如：StopFilter可以对停用词进行过滤。

3) 执行过程

【存储方式】

【应用TokenStream】

     /**

      * 使用TokenStream进行分词

      * @param str

      * @param analyzer

      */

     public static void displayTokenStream(String str, Analyzer analyzer){

         try {

             //通过Analayer获取TokenStream

             //toenStream("域名称或文件名"，输入流对象)

             TokenStream stream = analyzer.tokenStream("content", new StringReader(str));

             //向流中添加一个属性

             //容器，存储每次分词所对应的语汇内容

             CharTermAttribute charAttr = stream.addAttribute(CharTermAttribute.class);

             //通过循环语句读取语汇的内容

             while(stream.incrementToken()){

                 System.out.print("[" + charAttr + "]  ");

             }

             System.out.println();

         } catch (IOException e) {

             e.printStackTrace();

         }

     }

     private Version version = Version.LUCENE_35;

     /**

      * 测试TokenStream(英文内容)

      */

     @Test

     public void test01(){

         String str = "I'm come from Hanlin，I love Hanlin";

         System.out.println("str = " + str);

         System.out.println("====================================");

         //创建Analyzer对象

         Analyzer a1 = new SimpleAnalyzer(version);

         Analyzer a2 = new StopAnalyzer(version);

         Analyzer a3 = new WhitespaceAnalyzer(version);

         Analyzer a4 = new StandardAnalyzer(version);

         //测试TokenStream

         AnalyzerUtil.displayTokenStream(str, a1);

         AnalyzerUtil.displayTokenStream(str, a2);

         AnalyzerUtil.displayTokenStream(str, a3);

         AnalyzerUtil.displayTokenStream(str, a4);

     }

     /**

      * 测试TokenStream(中文内容)

      */

     @Test

     public void test02(){

         String str = "我来自翰林，我爱翰林";

         System.out.println("str = " + str);

         System.out.println("====================================");

         //创建Analyzer对象

         Analyzer a1 = new SimpleAnalyzer(version);

         Analyzer a2 = new StopAnalyzer(version);

         Analyzer a3 = new WhitespaceAnalyzer(version);

         Analyzer a4 = new StandardAnalyzer(version);

         //测试TokenStream

         AnalyzerUtil.displayTokenStream(str, a1);

         AnalyzerUtil.displayTokenStream(str, a2);

         AnalyzerUtil.displayTokenStream(str, a3);

         AnalyzerUtil.displayTokenStream(str, a4);

     }

TokenStream可以读取到分词内容.

【Attribute】

     /**

      * 显示语汇的基本属性

      * @param str

      * @param anlyzer

      */

     public static void displayAttributes(String str, Analyzer anlyzer){

         try {

             //获取TokenStream对象

             TokenStream stream = anlyzer.tokenStream("content", new StringReader(str));

             //PositionIncrementAttribute ：存储了语汇之间的位置增量

             //添加PositionIncrementAttribute属性

             PositionIncrementAttribute positionAttr = stream.addAttribute(PositionIncrementAttribute.class);

             //添加CharTermAttrbute

             CharTermAttribute charAttr = stream.addAttribute(CharTermAttribute.class);

             //OffsetAttribute：获取语汇的偏移数据

             OffsetAttribute offsetAttr = stream.addAttribute(OffsetAttribute.class);

             //语汇的分词方式类型（了解）

             TypeAttribute typeAttr = stream.addAttribute(TypeAttribute.class);

             //遍历每一个语汇

             while(stream.incrementToken()){

                 System.out.print(positionAttr.getPositionIncrement() + "、");

                 System.out.print("[" + charAttr + "  : " + offsetAttr.startOffset() + "~" + offsetAttr.endOffset()+ "(" + typeAttr.type()+ "）] " );

             }

             System.out.println();

         } catch (IOException e) {

             e.printStackTrace();

         }

     }

     /**

      * 测试属性的应用

      */

     @Test

     public void test03(){

         String str = "I'm come from Hanlin，I love Hanlin";

         System.out.println("str = " + str);

         System.out.println("====================================");

         //创建Analyzer对象

         Analyzer a1 = new SimpleAnalyzer(version);

         Analyzer a2 = new StopAnalyzer(version);

         Analyzer a3 = new WhitespaceAnalyzer(version);

         Analyzer a4 = new StandardAnalyzer(version);

         //测试TokenStream

         AnalyzerUtil.displayAttributes(str, a1);

         AnalyzerUtil.displayAttributes(str, a2);

         AnalyzerUtil.displayAttributes(str, a3);

         AnalyzerUtil.displayAttributes(str, a4);

     }

FlagsAttribute：标志位属性信息(了解)

PayloadAttribute：负载属性信息(了解)

说明:每一个语汇单元都存在一定的属性.通过Attribute可以获取到相关的语汇信息。

巴特西