• Contribute
• What's New
• Submit News

1.0 Introduction 

Figure 1: Biological Neuron

Simple neuron (Figure 2) introduced by McCulloch and Pitts in 1940s, consists of input layer, activation function, and output layer. Input layer receive input signal from external environment (or other neuron). Activation function is the neuron internal states that calculates and sum the input signals. The signals are then transmitted to output layer. The input layer, activation function and output layer in artificial neuron are similar to the function of dendrites, soma and axon in biological neuron.

Figure 2: McCulloch-Pitts Neuron Model

2.0 Learning in Neural Network

y_in_j =
where w_ij is the connection weights of input unit x_i and output unit y_j.

The network output (y_j) is calculated using the activation function f(x). In which y_j = f(x), where x is y_in_j. The computed weight from the training is stored and will become the information or knowledge for the future application.

NN can be divided into three architectures, namely single layer, multilayer network and competitive layer. The number of layers in a net is defined based on the number of interconnected weight in the neuron. Single layer network consists only one layer of connection weights. Whereas, multilayer networks consists of more than one layer of connection weights. The network also consists of additional layer called hidden layer. Multilayer networks can be used to solve more complicated problems compared to single layer network. Both of the network are also called feedforward network where the signal flows from the input units to the output units in a forward direction. The competitive layer network, for example the Recurrent Networks is a feedback network where there are closed-loop signal from a unit back to itself. 

Learning Mechanisms

NNs learning algorithms can be divided into two main groups that are supervised (or Associative learning) and unsupervised (Self-Organisation) learning. Many supervised and unsupervised learning NN have been invented. Some are listed in NN FAQ (frequently-ask-question) and discussion group web page, but many other are not. 

Supervised Learning
Supervised learning learns based on the target value or the desired outputs. During training the network tries to match the outputs with the desired target values. This method has two sub varieties called auto-associative and hetero-associative. In auto-associative learning, the target values are the same as the inputs, whereas in hetero-associative learning, the targets are generally different from the inputs. 

One of the most commonly used supervised NN model is backpropagation network that uses backpropagation learning algorithm. Backpropagation (or backprop) algorithm is one of the well-known algorithms in neural networks. Backpropagation algorithm has been popularized by Rumelhart, Hinton, and Williams in 1980s as a euphemism for generalized delta rule. Backpropagation of errors or generalized delta rule is a decent method to minimize the total squared error of the output computed by the net (Fausett, 1994). The introduction of backprop algorithm has overcome the drawback of previous NN algorithm in 1970s where single layer perceptron fail to solve a simple XOR problem. 

Unsupervised Learning
Unsupervised learning method is not given any target value. A desired output of the network is unknown. During training the network performs some kind of data compression such as dimensionality reduction or clustering. The network learns the distribution of patterns and makes a classification of that pattern where, similar patterns are assigned to the same output cluster. Kohonen network is the best example of unsupervised learning network. According to Sarle (1997) Kohonen network refers to three types of networks that are Vector Quantization, Self-Organizing Map and Learning Vector Quantization. 

3.0 Training the Network

Figure 3: Training and validation curve

Discussion and Conclusion

References