PySpark 中的新功能：TVF、子查询、绘图和分析器.pdf

1、Whats New in PySparkTVFs,Subqueries,Plots,and ProfilersTakuya Ueshin,Xinrong MengJune 11,2025IntroductionsTakuya UeshinSr.Software Engineer DatabricksXinrong MengSr.Software Engineer DatabricksAgendaTable-Valued Function APIPython UDTFSubquery Support in the DataFrame APILateral JoinTABLE argumentsP

2、lotting in PySpark WorkflowsProfiling PySpark CodeOther New and Upcoming Features3Your subtitle hereTable-Valued FunctionsTable-Valued Functions in PySparkFunctions that return a table,not a single valueAlready available in SQLBuilt-in TVFs:range,explode,inline,etc.Custom:written via Python UDTFsExa

3、mple TVFfrom pyspark.sql.functions import parse_json,litdf=spark.tvf.variant_explode(parse_json(lit(a:1,b:two)df.show()+-+-+-+|pos|key|value|+-+-+-+|0|a|1|1|b|two|+-+-+-+variant_explodeBuilt-in Table-Valued FunctionsCommonly used built-in TVFs:rangeexplode,explode_outerposexplode,posexplode_outerinl

4、ine,inline_outerstack,json_tuplevariant_explode,variant_explode_outercollations,sql_keywordsAll available under spark.tvf.xxxCan be used as top-level tables or in joinsWhat is a Python UDTF?UDTF=User-Defined Table FunctionReturns multiple rows per input rowschema defined by return typeImplemented by

5、 defining a Python class with:eval()method that yields rowsOptionally polymorphic via analyze()Decorated with udtfExample:RangeWithSquare UDTF from pyspark.sql.functions import udtfudtf(returnType=num INT,squared INT)class RangeWithSquare:def eval(self,start:int,end:int):for i in range(start,end):yi

6、eld(i,i*i)spark.udtf.register(range_with_square,RangeWithSquare)Generating a RangeUsing the UDTF in PySparkfrom pyspark.sql.functions import lit#df=spark.sql(SELECT*FROM range_with_square(1,4)df=RangeWithSquare(lit(1),lit(4)df.show()+-+-+|num|squared|+-+-+|1|1|2|4|3|9|+-+-+Polymorphic UDTFs:Dynamic