Training Methods

The original GCN has major problems in training and optimisation methods. For example, the embedding of a node is computed recursively using the embeddings of all its neighbours, which means as the number of layers increase, the cost of embedding computation grows significantly. In addition, GCN is trained for a fixed graph, which means that it lacks the ability for inductive learning.

Sampling

  • GraphSAGE [1] as it’s an improvement of the original GCN

  • PinSage – importance-based sampling

  • FastGCN

Receptive Field Control

[Need to do more research]

Data Augmentation

GCN requires many labelled data for validation. To tackle this problem, we can look into the following for increasing training dataset:

  • Co-Training GCN – finds the nearest neighbours of training data

  • Self-Training GCN

Unsupervised Training

There has been a new trend to extend autoencoder to graphs. Graph autoencoders (GAE) represents nodes in low-dimensional vectors using unsupervised training.

General Frameworks

Here’s the space that aims to integrate different models into a single framework. There are three networks:

  1. 7.1.1a – Message Passing Neural Network (MPNN) – combines different GNNs and GCNs

  2. 7.1.1b – Non-Local Neural Network (NLNN) – combines different self-attention mechanisms

  3. 7.1.1c – Graph Network (GN) – combines MPNN and NLNN methods as well as other methods such as Interaction Networks, CommNet, GGNN, Relation Network, Deep Sets and Point Net

Message Passing Neural Network (MPNN)

The framework extracts the common features of different graph models, both spectral and non-spectral methods. There are two phases:

  1. Message passing phase (propagation) – defined by the message and update function

  2. Readout phase – use a readout function to compute a feature vector of the whole graph

The unification comes from the message, update, and readout function where you could easily incorporate different settings for each function, drawing in the different settings from different graph models.

Non-Local Neural Network (NLNN)

Here, we use NLNN to capture long-term dependencies in deep networks. NLNN can be viewed as the combination of different self-attention mechanisms as it’s based around the idea of non-local operation, which computes the response at each position as the weighted sum of features in all positions. There are many ways to perform the non-local operation:

  1. Gaussian

  2. Embedded Gaussian

  3. Dot product

  4. Concatenation

Graph Network (GN)

The GN generalises and combines the different MPNN and NLNN methods! GN block is the core computation unit of the GN. A GN block consists of 3 update functions and 3 aggregation functions. The computation steps of a GN block are as follows:

There are three basic design principles when it comes to building the GN:

  1. Flexible representations – The GN framework should supports different representations of attributes and graph structures. The most common representation is real-valued vectors and tensors

  2. Configurable within-block structure – The functions and inputs within a GN block can have different settings

  3. Composable multi-block architectures – GN blocks should be able to be composed, to be used to construct complex architectures

Ryan

Ryan

Data Scientist

Leave a Reply