学习笔记（22）- plato-训练端到端的模型
原始文档
Train an end-to-end model
To get started we can train a very simple model using Ludwig (feel free to use
your favourite deep learning framework here):
input_features:
-
name: user
type: text
level: word
encoder: rnn
cell_type: lstm
reduce_output: null
output_features:
-
name: system
type: text
level: word
decoder: generator
cell_type: lstm
attention: bahdanau
training:
epochs: 100
You can modify this config to reflect the architecture of your choice and train
using Ludwig:
ludwig train \
--data_csv data/metalwoz.csv \
--model_definition_file plato/example/config/ludwig/metalWOZ_seq2seq_ludwig.yaml \
--output_directory "models/joint_models/"
我的笔记
训练端到端模型：
- 输入文件是
metalwoz.csv、metalWOZ_seq2seq_ludwig.yaml - 输出文件是
models/joint_models/
注意模型训练完毕，加载模型文件（使用模型）的时候，还需要（1）写一个类文件，实现plato提供的接口；（2）写一个yaml配置文件，用--config 参数 来告诉plato run 加载的模型的路径。
先准备数据，然后训练模型。
csv文件需要解析得到。
1. 下载metalwoz数据集
https://www.microsoft.com/en-us/research/project/metalwoz/
2. 解压数据
- 以shoping为例. 901个对话，2个角色
解压之后得到：
/Users/huihui/data/metalwoz-v1/dialogues/SHOPPING.txt
文件内容举例：
{"id": "47d85004", "user_id": "891cf0fb", "bot_id": "0f9f7619", "domain": "SHOPPING", "task_id": "5e456a4d", "turns": ["Hello how may I help you?", "i want to order a mattress from walmart", "Great. I can help you with your mattress order.", "how long will it take to arrive", "From the time of purchase it will take three days for us to ship it.", "great, lets start the order", "Once we have shipped it however, we dont know when it will arrive at your location", "how can i find out an exact date for it to arrive", "We ship priority mail through USPS. The length of time will vary with depending on the carrier", "well then i will try somewhere else, thank you anyway", "I am sorry we were not able to accommodate you"]}
3. 将txt文件转化为csv文件
- 3.1 准备yaml文件
编写文件
plato/example/config/parser/Parse_MetalWOZ.yaml
---
package: plato.utilities.parser.parse_metal_woz
class: Parser
arguments:
data_path: /Users/huihui/data/metalwoz-v1/dialogues/SHOPPING.txt
- 3.2 使用脚本执行转换
plato parse --config Parse_MetalWOZ.yaml
解析之后的文件在data/metalwoz.csv
- 注意：只有2个角色。不是多角色会话
user,system
hi,How can I help you today. I am a bot.
Can you help me order on an online shop,"Sure, I would love to help you. What is it you would like to order?"
I like to order bicycle helmet,Which brand helmet would you like to purchase.
yoni,"OK, what size helmet would you like to order."
small,"OK. I found a Yoni bicycle helmet in size small. It comes in black, red, blue or white. Which color would you like to order?"
black,The cost is $39.99. Would you like to go ahead and place an order?
ok pls,Great. Your order has been placed.
thank you,You are quite welcome. Have a great day!
hi,Hello how may I help you?
至此，csv文件准备完毕
4. 准备model_definition_file文件
官方文档给了例子，
plato/example/config/ludwig/metalWOZ_seq2seq_ludwig.yaml
---
input_features:
-
name: user
type: text
level: word
encoder: rnn
cell_type: lstm
reduce_output: null
output_features:
-
name: system
type: text
level: word
decoder: generator
cell_type: lstm
attention: bahdanau
training:
epochs: 100
5. 开始训练模型
ludwig train \
--data_csv data/metalwoz.csv \
--model_definition_file plato/example/config/ludwig/metalWOZ_seq2seq_ludwig.yaml \
--output_directory "models/joint_models/"
一共训练100轮
不等训练完毕
6. 写类文件，加载模型
Write a class inheriting from Conversational Module that loads and queries the model
This class simply needs to handle loading of the model, querying it
appropriately and formatting its output appropriately. In our case, we need to
wrap the input text into a pandas dataframe, grab the predicted tokens from
the output and join them in a string that will be returned. See the class here:
plato.agent.component.joint_model.metal_woz_seq2seq.py
package: plato.agent.component.joint_model.metal_woz_seq2seq
class: MetalWOZSeq2Seq
文件：
plato/agent/component/joint_model/metal_woz_seq2seq.py
"""
MetalWOZ is an MetalWOZ class that defines an interface to Ludwig models.
"""
class MetalWOZSeq2Seq(ConversationalModule):
……
7. 运行Agent
Write a Plato generic yaml config and run your agent!
See plato/example/config/application/metalwoz_generic.yaml for an example generic
configuration file that interacts with the seq2seq agent over text. You can try
it out as follows:
plato run --config metalwoz_text.yaml
plato/example/config/application/metalwoz_text.yaml
8. 测试结果
Dialogue 1 (out of 10)
USER > I want to buy a bicycle
(DEBUG) system> what is the helmet ?
USER > yoni
(DEBUG) system> what size ?
USER > small
……
2020-02-21 15:57:20 效果不好
模型没有训练好？
再继续训练看看
总结上面的流程
最新文章
- Nutch搜索引擎（第1期）_ Nutch简介及安装
- google.GIS小例子
- C#的Invoke和BeginInvoke
- JDK1.5新特性
- windows 和 linux ssh互连
- js 中常用的方法
- WCF bindings comparison z
- 初学Android 一 基本开发环境
- The Black Hole of Numbers (strtoint+inttostr+sort)
- php的ob_flush和flush(转)
- Unity 3D中的菜单项
- 【转】TI蓝牙BLE 协议栈代码学习
- Python经常使用第三方工具、库、骨架
- mixer: 一个用go实现的mysql proxy
- 监听F5刷新，添加路由前缀
- Arch Linux中使用VMware Workstation不能打开vmmon内核模块
- three.js - 渲染并展示三维对象
- JAVA SpringBoot 项目打成jar包供第三方引用自动配置(Spring发现)解决方案
- PathUtil
- 对称加密-AES