使用基于配置的方法增强数据集成.pdf

1、Supercharge Data Integration Using a Configuration-based ApproachCOLLABORATION FORTRANSFORMATIONIntroductions Bill SweetmanAshish PanditPaul RoeslerJohn GunvaldsonCOLLABORATION FORTRANSFORMATIONIntegrations at Scale UC San Diego needed to build hundreds of integrations in short timeMost integrations

2、 consist of simple data movement,Extract Load,with little TranslationCOLLABORATION FORTRANSFORMATIONIntegration LandscapeOver a dozen domains(Financial,Student,Research,HDH,RMP,)of systems;each domain with 100+integrationsCOLLABORATION FOR TRANSFORMATIONTraditional integration implementation and typ

3、ical issues encounteredCustom coding(Java),potential tools,Talend,IICS,etc.Moved to NiFi,shortening integration development from months to weeksStill needed to move faster to support three enterprise releases(OFC,Kuali,UCPath)Need faster developer onboarding processFaster design and implementationLo

4、wer cost of platformLegacy integration challengesCOLLABORATION FOR TRANSFORMATIONIntegration CapabilitiesStreaming and RealtimeBatch processingAPI ManagementFile ManagementCOLLABORATION FOR TRANSFORMATIONIntegration StackKafkaStreaming DataRealtimeNifiELT PatternSwiss Army KnifeAPI ManagerSelf Servi

5、ce API ManagementGoAnywhereFile ManagementCOLLABORATION FOR TRANSFORMATIONOrient to simple data import/exportReduce time to marketReduce developer onboarding time Reduce development timeEasy to maintain and monitorPlatform OptimizationCOLLABORATION FOR TRANSFORMATIONPattern CatalogSimple Integration

6、sCOLLABORATION FOR TRANSFORMATIONNew Integration Platform Apache Airflow is an open-source platform used to programmatically author,schedule,and monitor workflows It focuses on data processing,such as ETL(Extract,Transform,Load)processes.Provides a user interface to monitor and manage these workflow