Home » Articles » Neural Networks » Beginner

An Introduction to Neural Networks

Introduction

Neural-networks is one of those words that is getting fashionable in the new era of technology. Most people have heard of them, but very few actually know what they are. This essay is designed to introduce you to all the basics of neural networks — their function, generic structure, terminology, types and uses.

The term 'neural network' is in fact a biological term, and what we refer to as neural networks should really be called Artificial Neural Networks (ANNs). I will use the two terms interchangeable throughout the essay, though. A real neural network is a collection of neurons, the tiny cells our brains are comprised of. A network can consist of a few to a few billion neurons connected in an array of different methods. ANNs attempt to model these biological structures both in architecture and operation. There is a small problem: we don't quite know how biological NNs work! Therefore, the architecture of neural networks changes greatly from type to type. What we do know is the structure of the basic neuron.

The Neuron

Although it has been proposed that there are anything between 50 and 500 different types of neurons in our brain, they are mostly just specialized cells based upon the basic neuron. The basic neuron consists of synapses, the soma, the axon and dendrites. Synapses are connections between neurons - they are not physical connections, but miniscule gaps that allow electric signals to jump across from neuron to neuron. These electrical signals are then passed across to the soma which performs some operation and sends out its own electrical signal to the axon. The axon then distributes this signal to dendrites. Dendrites carry the signals out to the various synapses, and the cycle repeats.

Just as there is a basic biological neuron, there is basic artificial neuron. Each neuron has a certain number of inputs, each of which have a weight assigned to them. The weights simply are an indication of how 'important' the incoming signal for that input is. The net value of the neuron is then calculated - the net is simply the weighted sum, the sum of all the inputs multiplied by their specific weight. Each neuron has its own unique threshold value, and it the net is greater than the threshold, the neuron fires (or outputs a 1), otherwise it stays quiet (outputs a 0). The output is then fed into all the neurons it is connected to.

Learning

As this talk about weights and thresholds leads to an obvious question. How are all these values set? There are nearly as many training methods as there are network types (a lot!), but some of the more popular ones include back-propagation, the delta rule and Kohonen learning.

As architectures vary, so do the learning rules, but most rules can be categorized into two areas - supervised and unsupervised. Supervised learning rules require a 'teacher' to tell them what the desired output is given an input. The learning rules then adjusts all the necessary weights (this can be very complicated in networks), and the whole process starts again until the data can be correctly analyzed by the network. Supervised learning rules include back-propagation and the delta rule. Unsupervised rules do not require teachers because they produce their own output which is then further evaluated.

Architecture

This area of neural networking is the "fuzziest" in terms of a definite set of rules to abide by. There are many types of networks - ranging from simple boolean networks (Perceptrons), to complex self-organizing networks (Kohonen networks), to networks modelling thermodynamic properties (Boltzmann machines)! There is, though, a standard network architecture.

The network consists of several "layers" of neurons, an input layer, hidden layers, and output layers. Input layers take the input and distribute it to the hidden layers (so-called hidden because the user cannot see the inputs or outputs for those layers). These hidden layers do all the necessary computation and output the results to the output layer, which (surprisingly) outputs the data to the user. Now, to avoid confusion, I will not explore the architecture topic further. To read more about different neural nets, see the Generation5 essays.

Even after discussing neurons, learning and architecture we are still unsure about what exactly neural networks do!

The Function of ANNs

Neural networks are designed to work with patterns - they can be classified as pattern classifiers or pattern associators. The networks can takes a vector (series of numbers), then classify the vector. For example, my ONR program takes an image of a number and outputs the number itself. Or my PDA32 program takes a coordinate and can classify it as either class A or class B (classes are determined by learning from examples provided). More practical uses can be seen in military radars where radar returns can be classified as enemy vehicles or trees (read more in the Applications in the Military essay).

Pattern associators takes one vector and output another. For example, my HIR program takes a 'dirty' image and outputs the image that represents the one closest to the one it has learnt. Again, at a more practical level, associative networks can be used in more complex applications such as signature/face/fingerprint recognition.

The Ups and Downs of Neural Networks

There are many good points to neural-networks and advances in this field will increase their popularity. They are excellent as pattern classifiers/recognizors - and can be used where traditional techniques do not work. Neural-networks can handle exceptions and abnormal input data, very important for systems that handle a wide range of data (radar and sonar systems, for example). Many neural networks are biologically plausible, which means they may provide clues as to how the brain works as they progress. Advances in neuroscience will also help advance neural networks to the point where they will be able to classify objects with the accuracy of a human at the speed of a computer! The future is bright, the present however...

Yes, there are quite a few down points to neural networks. Most of them, though, lie with our lack of hardware. The power of neural-networks lie in their ability to process information in a parallel fashion (that is, process multiple chunks of data simultaneously). Unfortunately, machines today are serial - they only execute one instruction at a time. Therefore, modelling parallel processing on serial machines can be a very time-consuming process. As with everything in this day and age, time is of the essence, which often leaves neural networks out of the list of viable solutions to a problem.

Other problems with neural networks are the lack of defining rules to help construct a network given a problem - there are many factors to take into consideration: the learning algorithm, architecture, number of neurons per layer, number of layers, data representation and much more. Again, with time being so important, companies cannot afford to invest to time to develop a network to solve the problem efficiently. This will all change as neural networking advances.

Conclusion

Hopefully, by now you have a good understanding of the basics of neural networks. Generation5 has recently had a lot of information added on neural networking, both in essays and in programs. We have examples of Hopfield networks, perceptrons (2 example programs), and even some case-studies on back-propagation. Please browse through the site to find out more!

Submitted: 31/03/2000

Printed from: http://www.generation5.org/content/2000/nnintro.asp