postgresql 常用的删除重复数据方法
一、 最高效方法
测试环境验证，6600万行大表，删除2200万重复数据仅需3分钟
delete from deltest a where a.ctid = any(array (select ctid from (select row_number() over (partition by id), ctid from deltest) t where t.row_number > 1));
二、 PG中三种删除重复数据方法
首先创建一张基础表，并插入一定量的重复数据。
create table deltest(id int, name varchar(255));
create table deltest_bk (like deltest);
insert into deltest select generate_series(1, 10000), 'ZhangSan';
insert into deltest select generate_series(1, 10000), 'ZhangSan';
insert into deltest_bk select * from deltest;
1. 常规删除方法
最容易想到的方法就是判断数据是否重复，对于重复的数据只保留ctid最小（或最大）的数据，删除其他的。
explain analyse delete from deltest a where a.ctid <> (select min(t.ctid) from deltest t where a.id=t.id);
-------------------------------------------------------------------------------------------
Delete on deltest a (cost=0.00..195616.30 rows=1518 width=6) (actual time=67758.866..67758.866 rows=0 loops=1)
-> Seq Scan on deltest a (cost=0.00..195616.30 rows=1518 width=6) (actual time=32896.517..67663.228 rows=10000 loops=1)
Filter: (ctid <> (SubPlan 1))
Rows Removed by Filter: 10000
SubPlan 1
-> Aggregate (cost=128.10..128.10 rows=1 width=6) (actual time=3.374..3.374 rows=1 loops=20000)
-> Seq Scan on deltest t (cost=0.00..128.07 rows=8 width=6) (actual time=0.831..3.344 rows=2 loops=20000)
Filter: (a.id = id)
Rows Removed by Filter: 19998
Total runtime: 67758.931 ms
select count(*) from deltest;
count
-------
10000
可以看到，id相同的数据，保留ctid最小的，其他的删除。相当于把deltest表中的数据删掉一半，耗时达到67s多。相当慢。
2. group by删除方法
group by方法通过分组找到ctid最小的数据，然后删除其他数据。
explain analyse delete from deltest a where a.ctid not in (select min(ctid) from deltest group by id);
-------------------------------------------------------------------------------------------
Delete on deltest a (cost=131.89..2930.46 rows=763 width=6) (actual time=30942.496..30942.496 rows=0 loops=1)
-> Seq Scan on deltest a (cost=131.89..2930.46 rows=763 width=6) (actual time=10186.296..30814.366 rows=10000 loops=1)
Filter: (NOT (SubPlan 1))
Rows Removed by Filter: 10000
SubPlan 1
-> Materialize (cost=131.89..134.89 rows=200 width=10) (actual time=0.001..0.471 rows=7500 loops=20000)
-> HashAggregate (cost=131.89..133.89 rows=200 width=10) (actual time=10.568..13.584 rows=10000 loops=1)
-> Seq Scan on deltest (cost=0.00..124.26 rows=1526 width=10) (actual time=0.006..3.829 rows=20000 loops=1)
Total runtime: 30942.819 ms
select count(*) from deltest;
count
-------
10000
可以看到同样是删除一半的数据，使用group by的方式，时间节省了一半。但仍含需要30s，下面试一下第三种删除操作。
3. 高效删除方法
explain analyze delete from deltest a where a.ctid = any(array (select ctid from (select row_number() over (partition by id), ctid from deltest) t where t.row_number > 1));
-----------------------------------------------------------------------------------------
Delete on deltest a (cost=250.74..270.84 rows=10 width=6) (actual time=98.363..98.363 rows=0 loops=1)
InitPlan 1 (returns 0)−>SubqueryScanont(cost=204.95..250.73rows=509width=6)(actualtime=29.446..47.867rows=10000loops=1)Filter:(t.rownumber>1)RowsRemovedbyFilter:10000−>WindowAgg(cost=204.95..231.66rows=1526width=10)(actualtime=29.436..44.790rows=20000loops=1)−>Sort(cost=204.95..208.77rows=1526width=10)(actualtime=12.466..13.754rows=20000loops=1)SortKey:deltest.idSortMethod:quicksortMemory:1294kB−>SeqScanondeltest(cost=0.00..124.26rows=1526width=10)(actualtime=0.021..5.110rows=20000loops=1)−>TidScanondeltesta(cost=0.01..20.11rows=10width=6)(actualtime=82.983..88.751rows=10000loops=1)TIDCond:(ctid=ANY(0)−>SubqueryScanont(cost=204.95..250.73rows=509width=6)(actualtime=29.446..47.867rows=10000loops=1)Filter:(t.rownumber>1)RowsRemovedbyFilter:10000−>WindowAgg(cost=204.95..231.66rows=1526width=10)(actualtime=29.436..44.790rows=20000loops=1)−>Sort(cost=204.95..208.77rows=1526width=10)(actualtime=12.466..13.754rows=20000loops=1)SortKey:deltest.idSortMethod:quicksortMemory:1294kB−>SeqScanondeltest(cost=0.00..124.26rows=1526width=10)(actualtime=0.021..5.110rows=20000loops=1)−>TidScanondeltesta(cost=0.01..20.11rows=10width=6)(actualtime=82.983..88.751rows=10000loops=1)TIDCond:(ctid=ANY(0))
Total runtime: 98.912 ms
select count(*) from deltest;
count
-------
10000
最新文章
- Notepade ++ 替换每行最后的换行符
- 也来玩玩 javascript对象深拷贝，浅拷贝
- sp_MSforeachtable使用方法 查看库中所有表的空间大小
- Jquery ajax 得到返回值
- JavaScript 弹出窗体点击按钮返回选择数据的实现
- Android使用pull解析xml
- JS局部打印两种方法
- Swift编程语言学习1.4——数值型字面量、数值类型转换
- go mode
- jqueryui autocomplete的使用与angular配合的小坑
- J.U.C ThreadPoolExecutor解析
- windows转mac-开发环境搭建（一）：需要搭建的环境及安装的工具
- 4 Values whose Sum is 0 [POJ2785] [折半搜索]
- mysql5.6 主从同步配置
- Linux内核分析——进程的描述和进程的创建
- SQL数据库增量备份还原方式
- C++的字符串多行输入
- Charlie's Change（完全背包+路径记忆）
- 2019第十届蓝桥杯 E题 迷宫
- Python3.x：Selenium+PhantomJS爬取带Ajax、Js的网页