隐式图神经网络的介绍与进展.pdf

1、隐式图神经网络的介绍与进展刘俊成National University of SingaporeAdvisor:Xiaokui Xiao BackgroundA homogenous graph/network =,contains a node set and an edge set.Node attribute matrix.Given and,several tasks are usually considered in this area:1)Node classification(e.g.,classify research area of each paper in a citat

2、ion network)2)Graph Classification(e.g.,molecule property prediction)3)Link Prediction(e.g.,predict missing links on social network for friendship recommendation)4)BackgroundExplicit Layers v.s.Implicit Layers in Deep LearningConsider the input X and output Z,Explicit Layers:=(+)Implicit Layers,inst

3、ead of specifying how to compute output from input,specify thedesirable condition which the output should satisfy.Zico Kolter,David Duvenaud,and Matt Johnson.Deep Implicit Layers-Neural ODEs,Deep Equilibirum Models,and Beyond(NeurIPS 2020 Tutorial)Implicit Deep LearningA typical k-layer deep network

4、:Infinitely deep modelA fixed point iterationA class of implicit layer model:Deep Equilibrium(DEQ)Model=W+=+DEQ ModelThe goal of a DEQ model:to directly find this equilibrium point,without necessarily performing the forward iteration itself.For backpropagation,use implicit differentiation to directl

5、y get the gradient ofparameters from.Thus,dont need to store hidden states.Achieve constant memory cost O(1)which does not grow with#layers.Bai,Shaojie,J.Zico Kolter,and Vladlen Koltun.“Deep equilibrium models.”NeurIPS 2019.Implicit Model on GraphsImplicit Graph Neural Networks(NeurIPS 2020)Obtain n

6、ode representations through the fixed-point solutionTo train the model,they have a condition on parameters W to ensure convergence.Gu F,Chang H,Zhu W,Sojoudi S,El Ghaoui L.Implicit graph neural networks.NeurIPS 2020EIGNN:Efficient Infinite-Depth Graph Neural NetworksJuncheng Liu,Kenji Kawaguchi,Brya