NLP（二十三）使用LSTM进行语言建模以预测最优词

原文链接：http://www.one2know.cn/nlp23/

N元模型

预测要输入的连续词，比如

如果抽取两个连续的词汇，则称之为二元模型
准备工作

数据集使用 Alice in Wonderland

将初始数据提取N-grams

import nltk

import string

with open('alice_in_wonderland.txt', 'r') as content_file:

    content = content_file.read()

content2 = " ".join("".join([" " if ch in string.punctuation else ch for ch in content]).split())

tokens = nltk.word_tokenize(content2)

tokens = [word.lower() for word in tokens if len(word)>=2]

N = 3

quads = list(nltk.ngrams(tokens,N))

"""

    Return the ngrams generated from a sequence of items, as an iterator.

    For example:

        >>> from nltk.util import ngrams

        >>> list(ngrams([1,2,3,4,5], 3))

        [(1, 2, 3), (2, 3, 4), (3, 4, 5)]

"""

newl_app = []

for ln in quads:

    new1 = ' '.join(ln)

    newl_app.append(new1)

print(newl_app[:3])

输出：

['alice adventures in', 'adventures in wonderland', 'in wonderland alice']

如何实现

1.预处理：词转换为词向量

2.创建模型和验证：将输入映射到输出的收敛-发散模型（convergent-divergent）

3.预测：最优词预测
代码

from __future__ import print_function

from sklearn.model_selection import train_test_split

import nltk

import numpy as np

import string

with open('alice_in_wonderland.txt', 'r') as content_file:

    content = content_file.read()

content2 = " ".join("".join([" " if ch in string.punctuation else ch for ch in content]).split())

tokens = nltk.word_tokenize(content2)

tokens = [word.lower() for word in tokens if len(word)>=2]

N = 3

quads = list(nltk.ngrams(tokens,N))

"""

    Return the ngrams generated from a sequence of items, as an iterator.

    For example:

        >>> from nltk.util import ngrams

        >>> list(ngrams([1,2,3,4,5], 3))

        [(1, 2, 3), (2, 3, 4), (3, 4, 5)]

"""

newl_app = []

for ln in quads:

    new1 = ' '.join(ln)

    newl_app.append(new1)

# print(newl_app[:3])

# 将单词向量化

from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer() # 词=>词向量

"""

    >>> corpus = [

    ...     'This is the first document.',

    ...     'This document is the second document.',

    ...     'And this is the third one.',

    ...     'Is this the first document?',

    ... ]

    >>> vectorizer = CountVectorizer()

    >>> X = vectorizer.fit_transform(corpus)

    >>> print(vectorizer.get_feature_names())

    ['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']

    >>> print(X.toarray())  # doctest: +NORMALIZE_WHITESPACE

    [[0 1 1 1 0 0 1 0 1]

     [0 2 0 1 0 1 1 0 1]

     [1 0 0 1 1 0 1 1 1]

     [0 1 1 1 0 0 1 0 1]]

"""

x_trigm = []

y_trigm = []

for l in newl_app:

    x_str = " ".join(l.split()[0:N-1])

    y_str = l.split()[N-1]

    x_trigm.append(x_str)

    y_trigm.append(y_str)

x_trigm_check = vectorizer.fit_transform(x_trigm).todense()

y_trigm_check = vectorizer.fit_transform(y_trigm).todense()

# Dictionaries from word to integer and integer to word

dictnry = vectorizer.vocabulary_

rev_dictnry = {v:k for k,v in dictnry.items()}

X = np.array(x_trigm_check)

Y = np.array(y_trigm_check)

Xtrain, Xtest, Ytrain, Ytest,xtrain_tg,xtest_tg = train_test_split(X, Y,x_trigm, test_size=0.3,random_state=1)

print("X Train shape",Xtrain.shape, "Y Train shape" , Ytrain.shape)

print("X Test shape",Xtest.shape, "Y Test shape" , Ytest.shape)

# Model Building

from keras.layers import Input,Dense,Dropout

from keras.models import Model

np.random.seed(1)

BATCH_SIZE = 128

NUM_EPOCHS = 20

input_layer = Input(shape = (Xtrain.shape[1],),name="input")

first_layer = Dense(1000,activation='relu',name = "first")(input_layer)

first_dropout = Dropout(0.5,name="firstdout")(first_layer)

second_layer = Dense(800,activation='relu',name="second")(first_dropout)

third_layer = Dense(1000,activation='relu',name="third")(second_layer)

third_dropout = Dropout(0.5,name="thirdout")(third_layer)

fourth_layer = Dense(Ytrain.shape[1],activation='softmax',name = "fourth")(third_dropout)

history = Model(input_layer,fourth_layer)

history.compile(optimizer = "adam",loss="categorical_crossentropy",metrics=["accuracy"])

print (history.summary())

# Model Training

history.fit(Xtrain, Ytrain, batch_size=BATCH_SIZE,epochs=NUM_EPOCHS, verbose=1,validation_split = 0.2)

# Model Prediction

Y_pred = history.predict(Xtest)

# 测试

print ("Prior bigram words","|Actual","|Predicted","\n")

import random

NUM_DISPLAY = 10

for i in random.sample(range(len(xtest_tg)),NUM_DISPLAY):

    print (i,xtest_tg[i],"|",rev_dictnry[np.argmax(Ytest[i])],"|",rev_dictnry[np.argmax(Y_pred[i])])

输出：

X Train shape (17947, 2559) Y Train shape (17947, 2559)

X Test shape (7692, 2559) Y Test shape (7692, 2559)

_________________________________________________________________

Layer (type)                 Output Shape              Param #

=================================================================

input (InputLayer)           (None, 2559)              0

_________________________________________________________________

first (Dense)                (None, 1000)              2560000

_________________________________________________________________

firstdout (Dropout)          (None, 1000)              0

_________________________________________________________________

second (Dense)               (None, 800)               800800

_________________________________________________________________

third (Dense)                (None, 1000)              801000

_________________________________________________________________

thirdout (Dropout)           (None, 1000)              0

_________________________________________________________________

fourth (Dense)               (None, 2559)              2561559

=================================================================

Total params: 6,723,359

Trainable params: 6,723,359

Non-trainable params: 0

_________________________________________________________________

None

Prior bigram words |Actual |Predicted

595 words don | fit | know

3816 in tone | of | of

5792 queen had | only | been

2757 who seemed | to | to

5393 her and | she | she

4197 heard of | one | its

2464 sneeze were | the | of

1590 done with | said | whiting

3039 and most | things | of

4226 the queen | of | said

训练结果不好，因为单词向量维度太大，为2559，相对而言总数据集的单词太少；除了单词预测，还可以字符预测，每碰到一个空格算一个单词

巴特西

NLP（二十三）使用LSTM进行语言建模以预测最优词

最新文章

热门文章