At the forefront of Artificial Intelligence

Home » Articles » Neural Networks » Advanced

Some Reviews on Distributed Learning in Neural Networks

1.0 Introduction

Training single neural network is a difficult task. In each training session, the network is trained for hundred of epochs and some problem may involve a large amount of variables and data. Thus, training single network is time consuming. Therefore, distributed learning approaches such as hierarchical, multi-stage, parallel neural network computing and multi-modal neural network was introduced.

2.0 Hierarchical Network

Hierarchical Neural Network (HNN) is neural network architecture in which the problem is divided and solved in more than step (Ohno-Machado, 1996). Ohno-Machado divide hierarchical network into two architectures that are bottom-up and top-down architectures. In bottom-up designs, several specialized network are used to classify the instances and a top-level network (triage network) aggregates the results. In this design, all instances are used in all networks. However, the specialized networks work only on certain features. In contrast, in top-down hierarchical architecture design, the top-level network divides the inputs to be classified in specialized networks.

Figure 1: Hierarchical Neural Network

Many version of HNN have been introduced and applied in various applications. Erenshteyn and Laskov (1996) examine the application of hierarchical classifier to recognition of fingerspelling. They refer hierarchical NN as multi-stage NN. The approach aimed to minimize the network's learning time without reducing the accuracy of the classifier. Currently, Mat Isa et al (2002) used Hierarchical Radial Basis Function (HiRBF) to increase RBF performance in diagnosing cervical cancer. HiRBF cascading together two RBF networks, where both network have different structure but using the same algorithms. The first network classifies all data and performs a filtering process to ensure that only certain attributes to be fed to the second network. The study shows that HiRBF performs better compared to single RBF.

3.0 Multi-Stage Network

Cardot et al. (1994) developed a verification module for signature verification system by cooperating several neural network architectures in three levels. The first level (input level) is formed by two unsupervised NNs that is kohonen map. The second layer formed of two multi-layer networks using an error gradient backpropagation-learning algorithm. The final layer is formed by one NN of the backpropagation type. The network in the final level takes the output of previous level as the input and makes the final decision.

Figure 2: Multi-Stage Network in Cardot et al (1994)

Ahmed and Farag (1998) uses two self-organizing maps (SOM) in two stages, self-organizing principal components analysis (SOPCA) and self-organizing feature map (SOFM) for automatic volume segmentation of medical images. They performed a statistical comparison of the performance of the SOFM with Hopfield network and ISODATA algorithm. The results indicate that the accuracy of SOFM is superior compared to both networks. In addition, SOFM was claimed to have advantage of ease implementation and guaranteed convergence.

4.0 Parallel Neural Network Computing

Parallelism is a key feature of neural network processing. Typically, there are two approaches for parallel neural network computing that are parallel simulation on general-purpose computers and using parallel machine (Serbedzija, 1996). Many parallel NN simulator have been developed using whether general-purpose computers or parallel machine.

Misra (1996) surveyed the area of parallel environment for implementing NN. Misra has also listed four prescriptions that have to be considered when implementing NN in parallel machine. The prescriptions are theoretical analysis of the inherent parallelism, portability, ease of use from the user's perspective and access to ANN model description at various levels. Misra also suggests that portability and ease to use environment are an ideal environment for implementing NN on parallel machines.

Sulaiman and Evans (1996) introduced NEUCOMP2 that is a parallel NN compiler. NEUCOMP2 run on a shared-memory parallel machine was an extended version of NEUCOMP, a simulation NN compiler that executed sequentially. Another simulator known as PANNS (or Parallel Artificial Neural Network Simulator) was developed as a general-purpose tool for building, running and analyzing neural network models (Furle and Schikuta, 1997). PANNS distributes NN among the processors of a workstation clusters based on the data parallel model. The analysis shows that PANNS having a good speedup and scaleup behaviour.

Parallel NN simulator also can be developed in a conventional computing environment with one processor (Weigang and Silva, 1999). Weigang and Silva developed Parallel-SOM (Parallel Self-Organizing Map) and tested meteorological radar images. However, the approach did not work well in parallel processing.

5.0 Multi-Modal Neural Network

Multi-Modal Neural Networks (MNNs) share the same concept of parallel NN computing. MNNs utilizing many neural networks at the same time (Yoshihara et al., 2001). Yoshihara et al developed MNNs that composed of multi-layer neural networks and decision module to identify exon-intron boundaries in DNA base sequences. The NN modules learn the same data using different initial weights. The approach is found to produce better performance than using single NN.

Figure 3: Multi-Modal Neural Network

6.0 Conclusion

Distributed learning approaches reviewed in this article is an alternative approach for training Neural Network. This approach is essential, as too many data is needed in training which can create a complex network. A complex network is more difficult to train and usually takes more time, as more epochs are required to complete the tasks. On the other hand, most of the problem domain involves large amount of data and variables. Removing some of the data or variables, even though some consider less relevant could affect the system knowledge. As in human brain function, even tiny information could affect the whole decision process.
The approaches does not require an alteration of the NN learning algorithm. Instead, a large network is split into several groups or smaller networks. These networks are trained separately and another network integrates the outputs to produce the final result.

References

Ohno-Machado, L. (1996). Medical Applications of Artificial Neural Networks: Connectionist Model of Survival. Ph.D Dissertation. Stanford University.

Erenshteyn, R., and Laskov, P. (1996). A multi-stage approach to fingerspelling and gesture recognition. Proceedings of the Workshop on the Integration of Gesture in Language and Speech, (pp. 185-194), Wilmington, DE.

Mat Isa, N. A., Mashor, M. Y., and Othman, N. H. (2002). Diagnosis of Cervical Cancer using Hierarchical Radial Basis Function (HiRBF) Network. In Sazali Yaacob, R. Nagarajan, Ali Chekima (Eds.), Proceedings of the International Conference on Artificial Intelligence in Engineering and Technology (pp: 458-463). June 17-18 2002, Kota Kinabalu, Sabah, Malaysia

Cardot, H., Revenu, M., Victorri, B., Revillet, M. J. (1994). A Static Signature Verification System Based on a Cooperating Neural Networks Architecture.
International Journal of Pattern Recognition and Artificial Intelligence. 8(3), 679-692.

Ahmed, M. N., and Farag, A. A. (1998). Two-stage Neural Network for Medical Volume Segmentation. Accepted for Publication in the Journal of Pattern Recognition Letters, 1998.

Serbedzija, N. B. (1996). Simulating Artificial Neural Networks on Parallel Architectures. Computer, 29(3), 56-63.

Misra, M. (1996). Parallel Environments for Implementing Neural Networks. Neural Computing Surveys, 1, 48-60.

Sulaiman, M. N., and Evans, D. J. (1996). NEUCOMP2 – Parallel Neural Network Compiler. Malaysian Journal of Computer Science, 9(2), 54-70.

Furle, T., and Schikuta, E. (1997). PANNS – A Parallelized Artificial Neural Simulator. In Proceedings Fourth International Conference on Neural Information Processing (ICONIP’97), Dunedin, New Zealand, Springler-Verlag.

Weigang, L., and Silva, N. C. (1999). A Study of Parallel Neural Networks. In Proceedings of International Joint Conference on Neural Networks, 2, (pp: 1113-1116).

Yoshihara, I., Kamimai, Y., and Yasunaga, M. (2001). Feature Extraction from Genome Sequence using Multi-Model Neural Network. Genome Informatics 12, 420-422.

Submitted: 09/04/2004

Article Toolbar

BibTeX entry