drill 集成开源s3 存储minio
2024-08-22 06:38:21
drill 支持s3数据的查询，同时新版的通过简单配置就可以实现minio 的集成
测试使用docker 运行drill 参考 https://www.cnblogs.com/rongfengliang/p/9846899.html，minio 安装也可以使用docker
minio bucket 创建以及文件上传可以使用UI 或者mc
配置
- s3 配置说明
{
"type": "file",
"connection": "s3a://app/",
"config": {
"fs.s3a.access.key": "<key>",
"fs.s3a.secret.key": "<secret>",
"fs.s3a.endpoint": "<minio server>",
"fs.s3a.connection.ssl.enabled": "false", // default isn't ssl
"fs.s3a.path.style.access": "true"
},
"workspaces": {
"root": {
"location": "/",
"writable": false,
"defaultInputFormat": null,
"allowAccessOutsideWorkspace": false
},
"tmp": {
"location": "/tmp",
"writable": true,
"defaultInputFormat": null,
"allowAccessOutsideWorkspace": false
}
},
"formats": {
"psv": {
"type": "text",
"extensions": [
"tbl"
],
"delimiter": "|"
},
"csv": {
"type": "text",
"extensions": [
"csv"
],
"delimiter": ","
},
"tsv": {
"type": "text",
"extensions": [
"tsv"
],
"delimiter": "\t"
},
"parquet": {
"type": "parquet"
},
"json": {
"type": "json",
"extensions": [
"json"
]
},
"avro": {
"type": "avro"
},
"sequencefile": {
"type": "sequencefile",
"extensions": [
"seq"
]
},
"csvh": {
"type": "text",
"extensions": [
"csvh"
],
"extractHeader": true,
"delimiter": ","
}
},
"enabled": true
}- 查询
select * from s3.`root`.`user.json`- minio 添加数据
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}效果
参考资料
https://blog.minio.io/query-minio-datastore-with-apache-drill-dcaf71d0cee5?_branch_match_id=520116032687566918
https://www.cnblogs.com/rongfengliang/p/9846899.html
http://drill.apache.org/docs/s3-storage-plugin/
最新文章
- 从零开始学 Java - Spring AOP 实现主从读写分离
- [Web开发] 在HTML代码里面如何判断IE版本
- mysql的约束类型
- textarea 在浏览器中固定大小和禁止拖动
- [转]oracle中使用set transaction设置事务属性
- Leetcode 416. Partition Equal Subset Sum
- C++基础知识（1）----文件操作
- javaWeb---文件上传（commons-FileUpload组件）
- 从头学Qt Quick（2）-- QML语法从一个简单的例子说起
- DDD, MVC & Entity Framework
- KMP算法_读书笔记
- 数据库导出excel表数据
- 精华【分布式、微服务、云架构、dubbo+zookeeper+springmvc+mybatis+shiro+redis】分布式大型互联网企业架构！
- 【Linux】查看进程号
- c语言基础学习01
- 笔记：Struts2 拦截器
- [Swift]LeetCode110. 平衡二叉树 | Balanced Binary Tree
- kafka 暂停消费
- MHA-手动Failover流程(传统复制&GTID复制)
- window.name跨域实现