3-4 基于事件图结构的文本-视觉理解.pdf

1、Connecting Text and Vision with Event Graph StructuresManling Limanling2illinois.eduObjectCarEventCarEventBombingItemCarArgumentEventAttackingAttackerprotestersTargetpoliceEventAttackingAttackerpoliceTargetprotesterEventWearingItemmaskAgentpersonEventTreatmentAgentdoctorTargetpatientEventResearching

2、AgentresearcherTargetdropperEventSanitizingAgentpersonToolsprayerEventTestingAgentwomanplacecarEventVaccinationAgentwomanTargetgirlReal-world Multimedia ApplicationsReal-world multimedia applications requires image-language models to understand multiple levels of alignments such as verbs,objects,as

3、well as semantic structures.Zellers,Rowan,et al.From recognition to cognition:Visual commonsense reasoning.CVPR.2019.Visual Commonsense Reasoning 6Real-world Multimedia ApplicationsReal-world multimedia applications requires image-language models to understand multiple levels of alignments such as v

4、erbs,objects,as well as semantic structures.Park,Jae Sung,et al.Visualcomet:Reasoning about the dynamic context of a still image.”ECCV.Springer,Cham,2020.VisualCOMET(Visual Commonsense Reasoning in Time)7Cross-media Structured Common SpaceMultiple levels of Alignments8TextImage/Video FrameSentenceIm

5、ageWordImage RegionEntityVisual ObjectRelationVisual RelationEntity-Relation GraphVisual Scene GraphEvent TriggerVisual ActivityEvent Structure Image Event GraphObject-centric To Event-centricTreat Image/Video as a foreign language9TextImage/Video FrameSentenceImageWordImage RegionEntityVisual Objec

6、tRelationVisual RelationEntity-Relation GraphVisual Scene GraphEvent TriggerVisual ActivityEvent Structure Image Event GraphObject-centricEvent-centricHow to add event knowledge into Vision-Language Pretraining?To improve the event structure understanding of vision-language pretraining models:Where