Training Methods
The original GCN has major problems in training and optimisation methods. For example, the embedding of a node is computed recursively using the embeddings of all its neighbours, which means as the number of layers increase, the cost of embedding computation grows significantly. In addition, GCN is trained for a fixed graph, which means that it lacks the ability for inductive learning.
Sampling

GraphSAGE [1] as it’s an improvement of the original GCN

PinSage – importancebased sampling

FastGCN
Receptive Field Control
[Need to do more research]
Data Augmentation
GCN requires many labelled data for validation. To tackle this problem, we can look into the following for increasing training dataset:

CoTraining GCN – finds the nearest neighbours of training data

SelfTraining GCN
Unsupervised Training
There has been a new trend to extend autoencoder to graphs. Graph autoencoders (GAE) represents nodes in lowdimensional vectors using unsupervised training.
General Frameworks
Here’s the space that aims to integrate different models into a single framework. There are three networks:

7.1.1a – Message Passing Neural Network (MPNN) – combines different GNNs and GCNs

7.1.1b – NonLocal Neural Network (NLNN) – combines different selfattention mechanisms

7.1.1c – Graph Network (GN) – combines MPNN and NLNN methods as well as other methods such as Interaction Networks, CommNet, GGNN, Relation Network, Deep Sets and Point Net
Message Passing Neural Network (MPNN)
The framework extracts the common features of different graph models, both spectral and nonspectral methods. There are two phases:

Message passing phase (propagation) – defined by the message and update function

Readout phase – use a readout function to compute a feature vector of the whole graph
The unification comes from the message, update, and readout function where you could easily incorporate different settings for each function, drawing in the different settings from different graph models.
NonLocal Neural Network (NLNN)
Here, we use NLNN to capture longterm dependencies in deep networks. NLNN can be viewed as the combination of different selfattention mechanisms as it’s based around the idea of nonlocal operation, which computes the response at each position as the weighted sum of features in all positions. There are many ways to perform the nonlocal operation:

Gaussian

Embedded Gaussian

Dot product

Concatenation
Graph Network (GN)
The GN generalises and combines the different MPNN and NLNN methods! GN block is the core computation unit of the GN. A GN block consists of 3 update functions and 3 aggregation functions. The computation steps of a GN block are as follows:
There are three basic design principles when it comes to building the GN:

Flexible representations – The GN framework should supports different representations of attributes and graph structures. The most common representation is realvalued vectors and tensors

Configurable withinblock structure – The functions and inputs within a GN block can have different settings

Composable multiblock architectures – GN blocks should be able to be composed, to be used to construct complex architectures