在 Apache Spark™ 中管理数据加密.pdf

1、Managing Data Encryption in Apache SparkGidon Gershinsky,Apple Inc.Databricks2023THIS IS NOT A CONTRIBUTIONPresenterGidon GershinskyCommitter and PMC member in Apache ParquetWorks on big data security in Parquet,Spark,Iceberg and other projects Designs and builds data security solutions at AppleAgen

2、daOverview Parquet encryption features in SparkLearn how to run basic encryption and decryption sampleshands-on HelloWorld write/read demoDiscuss encryption management in production platformsQuestionsSQL or MLIngest1.Files(CSV,Avro,Parquet,ORC,JSON)2.Streams(Kafka messages,etc)1.Notebook users2.Spar

3、k applicationsUntrustedStorage BackendBig DataKey ManagerAccess ControlData Encryption in StorageProtect Sensitive Data-at-RestKeep the data Confidentialhiding sensitive information in storagevia encryptionKeep the data Tamper-Proofprotecting integrity of sensitive information in stored filesvia cry

4、pto-signatures and module IDsApache ParquetPopular columnar storage formatBuilt-in encoding,compression Advanced filtering for Big Datacolumnar projection:skip columnspredicate push down:skip row groups,or data pages,or even filesonly small data subset is fetched from storage,and processedBuilt-in e

5、ncryption since 2021=+ColumnarStatisticsRead only the data you needStrata 2017 Parquet Arrow Roadmapread only the data you needParquet Modular EncryptionGoalsEncrypt and sign all modules in Parquet files(data and metadata modules)Preserve full Parquet capabilities(columnar projection,predicate pushd

6、own,compression,etc)in encrypted filesPreserve performance of analytic engines with encrypted files2017 Parquet Arrow RoadmapParquet Modular EncryptionOpen standard for safe storage of analytic dataParquet Format spec for encryption approved in 2019Works the same in any storage cloud or private,file