Graph Representation Learning with Hierarchical Structure ... · Experiment p Facebook social...
46
Graph Representation Learning with Hierarchical Structure and Domain Adaptation Speaker ：Lun Du Affiliation: Microsoft Research Asia 2019/10/11 Graph Representation Learning 1
Transcript of Graph Representation Learning with Hierarchical Structure ... · Experiment p Facebook social...
GraphRepresentationLearningwithHierarchicalStructureandDomainAdaptation
Speaker：LunDuAffiliation:MicrosoftResearchAsia
2019/10/11 GraphRepresentationLearning 1
MainContents
q Background
q Graphembeddingwithhierarchicalcommunitystructure
q Domainadaptivegraphembedding
q Futureworks
2019/10/11 GraphRepresentationLearning 2
Background
q GraphEmbeddingtriestomapgraphverticesintoalow-dimensionalvectorspaceundertheconditionofpreservingdifferenttypesofgraphproperties.
2019/10/11 GraphRepresentationLearning 3
p Node classification
p Link Predictionp Network Visualization
p Community detection
p …...
Background
q Unsupervisedvs.Supervised§ DeepWalk,LINE, node2vec,etc.
§ GCN,GraphSAGE,etc.
q Euclideanvs.Non-Euclidean§ Hyperbolicspace(Tag2Vec,WWW’19)
q Vectorvs.Distribution§ Usingvariancetomodeluncertaintyofsemantic
2019/10/11 GraphRepresentationLearning 4
MainContents
q Background
q Graphembeddingwithhierarchicalcommunitystructure
q Domainadaptivegraphembedding
q Futureworks
2019/10/11 GraphRepresentationLearning 5
Outline
q Conceptually,complexnetworkshavehierarchicalcommunity inrealworld.§ E.g.socialnetworks,airtransportationnetworks,andmetabolicnetworks,
etc.
2019/10/11 GraphRepresentationLearning 6
Outline
q HierarchicalInfocanbeobservedtoacertainextentinonlinenetworks.
2019/10/11 GraphRepresentationLearning 7
Explicit hierarchy with attributes Implicit hierarchy with tags
#
#University
#Major
#EnrollmentYear
TwitterNetworkFacebook Network
Outline
q Goal:§ Encodingtherichhierarchicalstructuralinformation
qMainChallenges:§ Howtorepresentnodesortags?
§ Howtolearntherepresentationseffectivelyandefficiently?
2019/10/11 GraphRepresentationLearning 8
GalaxyNetworkEmbedding:AHierarchicalCommunityStructurePreserving
Approach
LunDu,Zhicong Lu,YunWang,GuojieSon𝑔∱ ,YimingWang,WeiChen.GalaxyNetworkEmbedding:AHierarchicalCommunityStructurePreservingApproach.InProceedingsofIJCAI,2018.
2019/10/11 GraphRepresentationLearning 9
HowtoRepresent?
q Inspiredbygalaxystructureq Embeddingnodesandcommunitiessimultaneously
§ Easytoanalyzethenetworkatdifferentscales.
2019/10/11 GraphRepresentationLearning 10
(The representations of nodes in tree)
Embedding
HowtoLearn?
q Formulatethehierarchicalcommunitypreservingnetworkembedding§ Oneisthelocalinformation,i.e.pairwisenodessimilarityinthesame
community.
§ Theotheristhehierarchicalstructureproperty,i.e.horizontalrelationshipandverticalrelationship.
q Implementandoptimizeefficientlytheembeddingmethod.
2019/10/11 GraphRepresentationLearning 11
HierarchicalPreservingNetworkEmbedding
q PairwiseProximityPreservation
2019/10/11 GraphRepresentationLearning 12
HierarchicalPreservingNetworkEmbedding
q HierarchicalStructurePreservation§ Horizontalrelationship:
§ Verticalrelationship:
𝒄𝒗𝒍𝒄𝒖𝒍 𝒄𝒘𝒍𝒄(𝒍)𝟏
𝒄𝒋𝒍
𝒄𝒌𝒍-𝟏
2019/10/11 GraphRepresentationLearning 13
GalaxyNetworkEmbedding
q Objective
2019/10/11 GraphRepresentationLearning 14
Proof
q PairwiseProximityPreservation
q HierarchicalStructurePreservation
§ Horizontal relationship:
§ Vertical relationship:
HierarchicalPreservingNetworkEmbedding
GalaxyNetworkEmbedding
?=>
2019/10/11 GraphRepresentationLearning 15
Proof
Lemma1
ThecommunityrepresentationslearnedfromrecursivelyoptimizingtheobjectiveEq.(5)with
thestrategyEq.(6)preservetheconstraintsEq.(3)andEq.(4).
⟹Vertical relationship:
Horizontal relationship:
2019/10/11 GraphRepresentationLearning 16
Experiment
p Facebook social network datasets:
p Amherst Collegep Hamilton Universityp Georgetown University
p Hierarchical random graphs (HRG):
p syn_with_125nodesp syn_with_1800nodesp syn_with_2560nodesp syn_with_3750nodes
2019/10/11 Graph Representation Learning 17
Dataset Baselines
p Spectral Clustering [Tang and Liu, 2011]
p DeepWalk [Perozzi et al, 2014]
p Node2vec [Grover and Leskovec, 2016]
p LINE [Tang et al., 2017]
p GraRep [Cao et al., 2015]
p MNMF [Wang et al., 2017]
Experiment
2019/10/11 Graph Representation Learning 18
Hierarchical Community Detection
Experiment
2019/10/11 Graph Representation Learning 19
Network Visualization
Experiment
2019/10/11 Graph Representation Learning 20
Vertex Classification
HierarchicalCommunityStructurePreservingNetworkEmbedding:ASubspaceApproach
𝐋𝐮𝐧𝐃𝐮∗,𝑄𝑖𝑛𝑔𝑞𝑖𝑛𝑔𝐿𝑜𝑛𝑔∗,Y𝑖𝑚𝑖𝑛𝑔𝑊𝑎𝑛𝑔∗,𝐺𝑢𝑜𝑗𝑖𝑒𝑆𝑜𝑛𝑔∱ ,𝑌𝑖𝐿𝑢𝑛𝑗𝑖𝑛,𝑊𝑒𝑖𝐿𝑖𝑛.HierarchicalCommunityStructurePreservingNetworkEmbedding:ASubspaceApproach.AcceptedbyCIKM,2019.
2019/10/11 GraphRepresentationLearning 21
DrawbacksofGNE
q Failedwhenembeddingdeepercommunities§ Radiishrinkexponentially
§ Datasparsityinadeepercommunity
q Probablyovervaluedhierarchicalinformation§ Verticesacrosscommunityareexponentially
distantthanthosewithinthesamecommunity.
2019/10/11 GraphRepresentationLearning 22
HowtoRepresent?
q Subspace§ Naturalhierarchicalstructure
§ Deepercommunitycorrespondingtolowerdimensionalsubspace
2019/10/11 GraphRepresentationLearning 23
HowtoLearn?
q Formulatingtheproblemintoanoptimizationproblemwithsubspaceconstraints§ Modelingcommunityaffiliationbysubspace
§ Reducingtherepresentationdimbyconstrainingtherankofbasevectors
q Designingefficientlearningalgorithm§ Fromglobaltolayer-wiseoptimization
§ Fromdiscretetodifferentiableoptimization
2019/10/11 GraphRepresentationLearning 24
HierarchicalStructurePreserved
q PreservationofStructurewithinIndividualCommunities
q PreservationofStructureamongCommunities
q LowRankConstraints
2019/10/11 GraphRepresentationLearning 25
ℒJ = K log𝜎(||𝑢RS − 𝑢U
S ||) + 𝑘 ⋅ 𝔼[\~ \̂[log𝜎(−||𝑢`S − 𝑢U
S ||)]U ,R ∈c
Where,𝑢U(d) = 𝑆efg 𝑢U
(d-J) for𝑙 = 1…𝐿, 𝑣U ∈ 𝑉
ℒm =KK K ΔURd − ΓURm
|pg|
RqU)J
pg
UqJ
r
dqS
ℒs = KK𝑟𝑎𝑛𝑘(𝑆Ud)pg
UqJ
r
dqS
Experiment
2019/10/11 Graph Representation Learning 26
Vertex Classification
Experiment
2019/10/11 Graph Representation Learning 27
Link Prediction
Experiment
2019/10/11 Graph Representation Learning 28
Network Visualization
Outline
q HierarchicalInfocanbeobservedtoacertainextentinonlinenetworks.
2019/10/11 GraphRepresentationLearning 29
Explicit hierarchy with attributes Implicit hierarchy with tags
#
#University
#Major
#EnrollmentYear
TwitterNetworkFacebook Network
Tag2Gauss:LearningTagRepresentationsviaGaussianDistributioninTaggedNetworks
LunD𝐮∗,YunWan𝑔∗,GuojieSon𝑔∱ ,XiaoMa,LichenJin,WeiLin,FeiSun.Tag2Gauss: LearningTagRepresentationsviaGaussianDistributioninTaggedNetworks.InProceedingsofIJCAI,2019.
2019/10/11 GraphRepresentationLearning 30
HowtoRepresent?
q Representtagsandnodessimultaneouslyq Tagsrepresentnodecommunitieswithintricateoverlappingrelationships
q Distribution:Tag;Samplefromdistributions:Node
2019/10/11 GraphRepresentationLearning 31
Howtolearn?
2019/10/11 GraphRepresentationLearning 32
q Tag2Gauss Framework:
§ Tag-view Embedding § Node-view Embedding § Multi-task Learning
Tag View Embedding Module Node View Embedding Module
MaxMargin Ranking Objective NCE Objective+=Objective
...
Forward
Backward
GCN
...
NodeTag
MultiTask Learning
...
Tag Relation Sequence
ProjectHybrid
Walker
Tag Distribution
Sample
...
Experiments
q Datasets:§ Leetcode (652nodes,1096edges,34tags,3labels)§ Bilibili (11727nodes,187148edges,151tags,10labels)§ Cora.(2707nodes,5429edges,1433tags,7labels)
q Baselines§ DeepWalk (KDD’14)§ Node2vec(KDD’16)§ HybridDeepwalk (NaiveDesign)§ GraphSage (NIPS’17)
2019/10/11 GraphRepresentationLearning 33
Experiment
2019/10/11 Graph Representation Learning 34
The Advantage of Distribution Representations
Segment Tree
ArrayLinked List
DFS
Topological SortDivide and Conquer
Rejection SamplingQueue
TrieDP
Binary Indexed TreeTwo Pointers
Tree RecursionHeap
Sort
BinarySearch
Bit Manipulation
Greedy
Backtracking
BSTMinimax
Brainteaser
Segment TreeArray
Linked List DFS
Topological Sort
Divide and ConquerRejection Sampling
TrieDP
Binary Indexed TreeTwo Pointers
Recursion
Bit Manipulation
BSTBacktracking
Sort
BinarySearch
Minimax
Brainteaser
Greedy
HeapQueue
Tree
Experiment
2019/10/11 Graph Representation Learning 35
Node Classification
MainContents
q Background
q Graphembeddingwithhierarchicalcommunitystructure
q Domainadaptivegraphembedding
q Futureworks
2019/10/11 GraphRepresentationLearning 36
DANE:DomainAdaptiveNetworkEmbedding
YizhouZhang, GuojieSon𝑔∱ ,LunDu,Shuwen Yang,Yilun Jin.DANE:DomainAdaptiveNetworkEmbedding.InProceedings ofIJCAI,2019.
2019/10/11 GraphRepresentationLearning 37
Motivation
q Domainadaptation§ Transferringmachinelearningmodelsacrossdifferentdatasetsto
handlethesametaskq Domainadaptationonnetworksissignificant:
§ Reducethecostoftrainingdownstreammachinelearningmodelsbyenablingmodelstobereusedonothernetworks
§ Handlethescarcityoflabeleddatabytransferringmodelstrainedwellonalabelednetworktounlabelednetworks
q Itisimportanttodesignanetworkembeddingalgorithmthatcansupportdomainadaptation.
2019/10/11 GraphRepresentationLearning 38
Challenges
q Embeddingspacealignment§ Structurallysimilarnodesshouldhavesimilarrepresentationsintheembeddingspace,eveniftheyarefromdifferentnetworks.
q Distributionalignment§ Embeddingvectorsofdifferentnetworksshouldhavesimilardistributioninembeddingspace.
§ Mostmachinelearningmodelsperformasguaranteedonlywhentheyworkondatawithsimilardistributionastheirtrainingdata.
2019/10/11 GraphRepresentationLearning 39
TechniqueFramework:Overall
2019/10/11 GraphRepresentationLearning 40
AdversarialLearningRegularization
q Discriminatortoavoidtheinstabilityofadversariallearning:
q Adversarialtraininglossfunctiontoconfusethediscriminatoris:
qOveralllossfunction
2019/10/11 GraphRepresentationLearning 41
Experiment
2019/10/11 Graph Representation Learning 42
Comparison with Baselines
Experiment
2019/10/11 Graph Representation Learning 43
Comparison with the Variant without adversarial learning
MainContents
q Background
q Graphembeddingwithhierarchicalcommunitystructure
q Domainadaptivegraphembedding
q Futureworks
2019/10/11 GraphRepresentationLearning 44
FutureWorks
q Understandingofgraphneuralnetworks§ Whydoesitwork?
§ Whatkindofgraphisitmoreeffective?
q CustomizedGNNfordifferentkindsofgraphsq Applications
§ Semi-structureddatamining
§ Sourcecodeanalytics
2019/10/11 GraphRepresentationLearning 45
Q&A
2019/10/11 GraphRepresentationLearning 46
Welcome to collaboration or internship!
![CSCE 580 Artificial Intelligence Ch.2 [P]: Agent Architectures and Hierarchical Control](https://static.fdocuments.us/doc/165x107/56812f52550346895d94e351/csce-580-artificial-intelligence-ch2-p-agent-architectures-and-hierarchical.jpg)
