将dbplyr的tbl_sql对象写入MySQL不本地存数时SQL语法报错
问题背景
本问题拓展自Stack Overflow相关提问。
当前需求为:使用自定义函数将R环境中的tbl_sql对象直接写入MySQL生成新表,全程不将数据落地存储到本地。使用场景为:MySQL环境中库与schema均命名为test,R中待写入的tbl_sql对象为my_data,目标生成的MySQL新表名为car_data。
初始实现代码如下:
library(DBI) library(tidyverse) library(dbplyr) # 建立连接并从MySQL读取数据 con <- DBI::dbConnect(RMariaDB::MariaDB(), dbname = "test", host = "127.0.0.1", user = "user", password = "password") my_data <- tbl(con, "mtcars") my_data <- my_data %>% filter(mpg >= 22) # 编写将tbl_sql写入数据库新表的函数 write_to_database <- function(input_tbl, db, schema, tbl_name){ # 获取连接对象 tbl_connection <- input_tbl$src$con # 拼接SQL查询 sql_query <- glue::glue( "SELECT *\n", "INTO {db}.{schema}.{tbl_name}\n", "FROM (\n", dbplyr::sql_render(input_tbl), "\n) AS sub_query" ) result <- dbExecute(tbl_connection, as.character(sql_query)) } # 执行函数 write_to_database(my_data, "test", "test", "car_data")
执行代码后抛出如下语法报错:
Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '.test.car_data FROM ( SELECT * FROM `mtcars` WHERE (`mpg` >= 22.0) ) AS sub_quer' at line 2 [1064] 12. stop(structure(list(message = "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '.test.test.car_data\nFROM (\nSELECT *\nFROM `mtcars`\nWHERE (`mpg` >= 22.0)\n) AS sub_quer' at line 2 [1064]", call = NULL, cppstack = NULL), class = c("Rcpp::exception", "C++Error", "error", "condition"))) 11. result_create(conn@ptr, statement, is_statement) 10. initialize(value, ...) 9. initialize(value, ...) 8. new("MariaDBResult", sql = statement, ptr = result_create(conn@ptr, statement, is_statement), bigint = conn@bigint, conn = conn) 7. dbSend(conn, statement, params, is_statement = TRUE) 6. .local(conn, statement, ...) 5. dbSendStatement(conn, statement, ...) 4. dbSendStatement(conn, statement, ...) 3. dbExecute(tbl_connection, as.character(sql_query)) 2. dbExecute(tbl_connection, as.character(sql_query)) 1. write_to_database(my_data, "test", "test", "car_data")
报错原因
报错由两个语法适配问题导致:
- 原自定义函数是为SQL Server编写的,使用的
SELECT ... INTO ...是SQL Server专属的查询结果建表语法,MySQL不支持该语法实现建表,MySQL中查询结果建表需要使用CREATE TABLE ... AS SELECT ...语法,原生SELECT ... INTO仅支持将查询结果赋值给变量,无法生成新表。 - MySQL中不存在SQL Server体系下“库-schema-表”的三层结构,MySQL的schema与database是完全等价的概念,代码中拼接的
test.test.car_data三层命名会被MySQL解析为test库下test表的car_data字段,直接触发语法错误,这也是报错提示.test.car_data附近语法异常的核心原因。
修复方案
调整函数逻辑适配MySQL语法,去掉重复的库名层级,改用MySQL支持的CTAS(CREATE TABLE AS SELECT)语法拼接语句,修复后代码如下:
write_to_database <- function(input_tbl, schema, tbl_name, overwrite = FALSE){ # 提取数据库连接 tbl_connection <- input_tbl$src$con # 拼接MySQL兼容SQL sql_query <- glue::glue( # 配置覆盖逻辑:overwrite为TRUE时先删除已存在的同名表 ifelse(overwrite, "DROP TABLE IF EXISTS `{schema}`.`{tbl_name}`;\n", ""), "CREATE TABLE `{schema}`.`{tbl_name}` AS\n", dbplyr::sql_render(input_tbl) ) # 执行SQL dbExecute(tbl_connection, as.character(sql_query)) } # 函数调用 write_to_database(my_data, schema = "test", tbl_name = "car_data", overwrite = TRUE)
修复后逻辑说明:
- 全程在数据库端完成计算和表写入,不会将数据拉取到本地内存,满足数据不落地的需求
- 标识符使用反引号包裹,避免表名、库名与MySQL保留字冲突
- 新增
overwrite参数控制是否覆盖已存在的同名表,默认值为FALSE,遇到重名表直接报错,避免误删数据 - 去掉了冗余的
db参数,适配MySQL中schema与database等价的特性
内容的提问来源于stack exchange,提问作者dd_data
相关产品推荐
相关产品推荐

