GenAI 在客户服务中的可观察性.pdf

1、GenAI Observability in Customer Care EarnInWillem DHaeseleerMatteo CiccozziAgenda3Customer Support At EarnIn5 minutesChatbot Journey5 minutesMetrics5 minutesPlatform10 minutesLooking Forward2 minutesHallucinations8 minutesEarnIn4Make Any Day PayDayMake Any Day PayDayNo interest,no credit checkOver 1

2、9 million downloads and countingCustomer Care5At EarnInTaxonomy of 100+customer support issues24/7 real time support across USHighly regulated by CFPBCustomer Care6At EarnInTaxonomy of 100+customer support issues24/7 real time support across USHighly regulated by CFPB7March 2024FAQ RAGJune 2024Tool

3、callingAugust 2024Datadog-DatabricksOctober 2024Ai/Bi Dashboard February 2025 Agentic CapabilitiesApril 2025Multimodal2025+ReasoningChatbot JourneyAt EarnInMetricsImmediate feedback during experimentationMetricsHuman reviewEvolving definition of metricsHigh cardinality metrics8Closing the feedback l

4、oopMetricsSingle use case evaluationDid we help the customer?Did we contain the conversation?Intent precision Multi use case evaluationMeasured over all chat trafficBottom Line/North starIntent recallContainmentCoverage9Measuring in productionMetrics10Containment/CoveragePlatform12PlatformArchitectu

5、re13PlatformArchitecture14PlatformArchitecture15PlatformArchitecture16PlatformArchitecturePlatform17Publishing Closeup*https:/microservices.io/patterns/data/transactional-outbox.html*Platform18Materialization CloseupPlatform19Materialization Self-ServePlatform20Materialization Self-ServePlatformClou

6、deventsTransactional OutboxDefine contract for eventsPublish events to kafkaDelta Live TablesAccess control w/UnityNear real time optionSelf-Serve Materialization21Developer ExperienceProtobuf or JSON SchemaSchema Registry&Lifecycle Integrated data environmentNotebook explorationTrack experimentsInc