构建数据湖库来管理自动驾驶车辆数据的 PBS.pdf

1、How Traton built a data lakehouse for autonomous drivingJoachim ZettermanSr.Data&ML platform lead 2Production and assembly sites world wide33Employees as of 2023103,621Sold vehicles in 2023338,183 3Automation will have a revolutionary impact on transport,improving energy efficiency and safety as wel

2、l as relieving congestion issues profoundly changing the future of transportation.WIPAUTONOMOUS SOLUTIONSMININGHUB-TO-HUB development of autonomous trucks for mining 5self-driving trucks ON PUBLIC ROADS 6Autonomous SystemCloudDeploy ML models&new data triggersDevelop&train ML modelsOffloaddataData&A

3、I loop16 x Cameras8 x Lidars8 x Radars1 TB/h per vehicle 7AI capabiltiesMotion planningOccupancySemantic segmentation+stereo Road feature dectionMultimodal object detectionAgent predictionVerification/SBA Data selection+augmentationPerception Prediction&planning Tools 8The long tail problem of auton

4、omous drivingTimeDriving scenarios Flywheel mechanism for continous data curation 9 Complex data and ML tooling Growing infrastructure Knowledge and competenceChallenges with data driven development“Hidden technical debt in machine learning systems”Simplify complex technology and reduces cognitive l

5、oad Self-service data and ML infrastructure Co-development with usersPlatform thinkingData&ML PlatformPipelinesCodeComputeStorageData&ML engineers10 11Per teamPersonal Data groupAll usersHow we use Unity Catalog-Tables-Volumes-Models-Dashboards-GDPR data-GPS-License plates-Human faces-Sensor data-La

6、beled data-System data-Meta dataTeamCatalogsData Products CatalogPersonal Data CatalogUnity CatalogDiscoveryAccess ControlLineageAuditing/System tablesData GovernanceData SharingAll users-Default catalog-Clean it every nightSandboxCatalogData Share Catalog-Open source datasets-Si