为何Spark使用predicates读取JDBC的速度远慢于分区参数方式?
Spark JDBC读取PostgreSQL的性能差异问题
我用Spark的spark.read.jdbc接口读取PostgreSQL表,测试了两种读取方式:
方式1:使用upperBound、lowerBound、numPartitions参数
df = spark.read.format("jdbc")\ .option("url",f"jdbc:postgresql://{host}:{port}/{dbname}")\ .option("dbtable",dbtable)\ .option("user",user)\ .option("password",password)\ .option("partitionColumn","person_id")\ .option("numPartitions",2)\ .option("lowerBound",261461)\ .option("upperBound",29283487)\ .load() df.count()
方式2:使用predicates参数
df = spark.read.jdbc( url = f"jdbc:postgresql://{host}:{port}/{dbname}", table = dbtable, predicates = ["(person_id < 14772474 OR person_id is null)", "(person_id >= 14772474)"], properties = {"user": f"{user}", "password": f"{password}"} ) df.count()
我原本预期两种方式执行时间相近,但实际方式1仅耗时10秒,方式2却耗时40秒。查看PostgreSQL日志,两种方式执行的SQL语句完全一致:
LOG: execute <unnamed>: SELECT 1 FROM xxxxx.person WHERE "person_id" < 14772474 or "person_id" is null LOG: execute <unnamed>: SELECT 1 FROM xxxxx.person WHERE "person_id" >= 14772474
内容的提问来源于stack exchange,提问作者willshen
相关产品推荐
相关产品推荐

