大规模管理 Databricks.pdf

1、Managing Databricks at ScaleA Data Journey at Vikas Ranjan June 11,202519+years of experience in Data&AIDriving massive scale hybrid data estatePassionate about using data for goodSeahawks Fan Deep discussions with 12-year-old kid on world history and global politics3Vikas RanjanSr.ManagerNetwork Da

2、ta&AIIntroduction“To be the best in the world at connecting customers to their world.”Our MissionBest ExperienceSustainabilityBest NetworkData For GoodMaximizing ValueConnected Data&Connected Intelligence2025+(700TB)AI Agents Reinforcement Learning Decision Automation Multi-Cloud2016(2TB)Data Engine

3、ering Data Exploration Hadoop Migration2017(20TB)BI Reporting KPI&Statistical Analysis2019(200TB)Anomaly Detection Forecasting Cloud Modernization2021(400TB)Root Cause Analysis Insights-Guided Workflow Open-Source Innovation2024(600TB)Network Copilot Network Knowledge Center Falcon AIGenGen-AIAIPres

4、criptivePrescriptiveDescriptiveDescriptiveBig DataBig DataDiagnostic Diagnostic&Predictive&PredictiveAgentic AIAgentic AIAGICapabilitiesBusiness ValueOur Data&AI Platform JourneyYEAR(Daily Data Volume)80+PB of network logs Multi-Cloud(Hadoop,Databricks,Snowflake,Azure OpenAI,K8)Supporting 5,000+inte

5、rnal users Secured and governedCurrent Data EcosystemData InputsNetwork,Device logs and Location Records from Network Switches and Tools Authoritative Data Sourcew/a Connected Data StrategyData Driven OutcomesEnterprise Security Data Governance Data CatalogAccess FrameworkAI/ML ModelsInsightsReal-Ti

6、meCoverageAI CenterAPIsConnectedDataData Storage/Processing Layer500+TB DailyData CorrelationIMSI Level ViewsTime Series EventsChallengesGovernanceData ProcessingReliabilityUpskillingCostChallenges to Process Data at ScaleThe proverbial Fire HoseVolume&VelocityComplexity Propriety Network Logs Vario