1、首先进行数据训的XltraBackup备份,有备无患,切记切记!

2、
mysql -uroot -pD********
-- 导出csv文件
use dsideal_db; MariaDB [dsideal_db]> SELECT * from t_resource_info INTO OUTFILE "/usr/local/mysql/t_resource_info.txt" FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n';
Query OK, rows affected (29.97 sec) 3、切分csv文件,这样分批导入更快,更方便,参考这里:http://www.nowamagic.net/librarys/veda/detail/2495,但是不能使用按容量啊,一定要按行。说日志太大拿不回来的,罚面壁一小时!
mkdir /usr/local/huanghai -p
split -a -d -l /usr/local/mysql/t_resource_info.txt /usr/local/huanghai/prefix
-3秒吧 3、清空原表,修改字段,反正有备份,不怕的
  truncate t_resource_info;
  alter table t_resource_info add huanghai_test int;
4、优化环境配置,准备开始导入
SET autocommit=;
SET unique_checks=;
SET foreign_key_checks=;
set sql_log_bin=; SET @innodb_additional_mem_pool_size=;
set @innodb_buffer_pool_size=;
set @innodb_log_buffer_size=;
set @innodb_log_file_size=; load data infile '/usr/local/huanghai/prefix00' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix01' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix02' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix03' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix04' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix05' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix06' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix07' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix08' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix09' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix10' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix11' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix12' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix13' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix14' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix15' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix16' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix17' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix18' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"';
load data infile '/usr/local/huanghai/prefix19' IGNORE into table dsideal_db.t_resource_info_huanghai fields terminated by ',' enclosed by '"'; commit; 5、恢复现场
SET autocommit=;
SET unique_checks=;
SET foreign_key_checks=;
set sql_log_bin=; 6、建议使用python3进行开发一个程序,这样方便串连起来,这是我目前能想到的最好办法,基本无风险,速度上基本能利用磁盘的最大IO,不建议采用修改frm等暴力办法,那个对于最后面追加字段的可能还行,对于字段在中间的
可能就是灾难,而且没有办法程序化,这个办法是用python3开发起来,基本无困难。

补充一下测试结果,但这台机器实在是太NB了,可能一般的客户没有这样的条件,供参考吧:

测试表:
 t_resource_info
 记录个数:1582937

一、生成
[root@localhost TestLoadFile]# python3 ExportData.py
2017-11-05 17:03:57      成功创建工作目录!
2017-11-05 17:03:59      开始导出数据...
2017-11-05 17:04:29      成功导出数据!
2017-11-05 17:04:29      正在进行分割...
2017-11-05 17:04:32      成功进行分割!
导出需要35秒

二、重新导入
[root@localhost TestLoadFile]# python3 ImportData.py
2017-11-05 16:58:08,378 INFO    : 开始生成SQL脚本...
2017-11-05 16:58:08,380 INFO    : 成功生成SQL脚本!
2017-11-05 16:58:08,380 INFO    : 开始执行SQL脚本...
2017-11-05 16:59:27,223 INFO    : SQL脚本执行成功!
导入需要79秒

合计需要114秒。

===================================================================================================
测试用机:

物理机,4颗CPU
cat /proc/cpuinfo| grep "physical id"| sort| uniq| wc -l
4

CPU核数:
cat /proc/cpuinfo| grep "cpu cores"

逻辑处理器个数:
cat /proc/cpuinfo| grep "processor"| wc -l
64

cpu 型号:
[root@localhost TestLoadFile]# cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c
64  Intel(R) Xeon(R) CPU E7-4809 v4 @ 2.10GHz

内存:
cat /proc/meminfo
MemTotal:       65845352 kB

===================================================================================================

附10.10.14.224测试用例

[root@localhost TestLoadFile]# python3 ExportData.py
2017-11-06 07:51:14      成功创建工作目录!
2017-11-06 07:51:14      开始导出数据...
2017-11-06 07:53:12      成功导出数据!
2017-11-06 07:53:12      正在进行分割...
2017-11-06 07:53:27      成功进行分割!
You have new mail in /var/spool/mail/root
[root@localhost TestLoadFile]# python3  ImportData.py
2017-11-06 07:55:37,622 INFO    : 开始生成SQL脚本...
2017-11-06 07:55:37,629 INFO    : 成功生成SQL脚本!
2017-11-06 07:55:37,630 INFO    : 开始执行SQL脚本...
2017-11-06 08:07:40,093 INFO    : SQL脚本执行成功!

===================================================================================================

附:测试用例 链接:http://pan.baidu.com/s/1dFbCEIl 密码:75j5

最新文章

  1. npm package.json属性详解
  2. 用nhibernate的几点小经验
  3. 深入分析 Java 中的中文编码问题
  4. Java多线程编程——进阶篇一
  5. Legendre polynomials
  6. dom解析器机制 web基本概念 tomcat
  7. MFC UpdateData(true) 失败原因
  8. 我的DbHelper数据操作类(转)
  9. 导入Excel 文件(图片和文字)NPOI+BootStrap fileinput 插件 的结合使用
  10. javascript三角函数的使用
  11. PHP—-模型MODEL 一对多
  12. 转 qInstallMsgHandler实现日志输出
  13. asp.net在配置文件里设置多种编码方式的研究
  14. 受邀与微软售前技术团队参与TFS技术沟通会议
  15. 检测硬件RDMA卡是否存在
  16. 关于putty连接百度云linux服务器那些事
  17. NOIP 初赛笔记
  18. Java 请求webServce接口 不带参数
  19. app生成工具
  20. 227. Mock Hanoi Tower by Stacks【easy】

热门文章

  1. 用HTML5 File API 实现截图粘贴上传、拖拽上传
  2. 查看自己电脑外网IP
  3. pythonweb框架
  4. Flask中使用mysql
  5. webpack 4 :从0配置到项目搭建
  6. ...args剩余参数用法
  7. aspnet_regiis.exe -i 执行报错
  8. mysql查询语句的执行顺序(重点)
  9. 日常开发技巧:使用notify-send发送通知
  10. Linux内核【链表】整理笔记(2) 【转】