At the forefront of Artificial Intelligence

Home » Articles » Neural Networks » Advanced

Multi-Backpropagation Network: Concept and Modeling

1.0 Introduction

Backpropagation network is able to deal with various types of data and also has the ability to model a complex decision system. However, some problem domains might involve a large amount of data. Backpropagation network with four input units and two hidden units for example required certain epochs, to create classification or prediction model. More input units or hidden units could increase the complexity of the model and also increase its computational complexity. In other words, additional input unit or hidden unit could increase the model complexity and increase training time. This is because a larger network is more difficult to train. Like human learning, a complex problem requires certain period of time to establish learning.

In some studies, backpropagation network and in general neural network has been considered "not efficient". The network, appears very time consuming even for small network architectures (Sima, 1994; Sima, 1996). Sima (1996) highlighted that the failure of effort to speed up the algorithm is caused by the network's learning complexity problem. Therefore, in this study multi-backpropagation network approach is proposed as an alternative training method for backpropagation network to reduce the complexity.

2.0 Modeling With Multi-Backpropagation Network
The first stage of multi-backpropagation training is to minimize the complexity of data used to train the network. Minimizing the complexity means reducing the complexity of each pattern by normalizing its attributes. Normalization referred in this study is the same as applied in relational databases where attributes are grouped into several categories to minimize the relationship between attributes. In representing an object, not all of the attributes can be represented. Instead, only necessary attributes are presented to fulfill user's information need (Kroenke, 1997). This technique could also reduce the redundancy of data. In the second stage, each category is represented in the specialized networks. Figure 1 shows the semantic object for student's record.

Figure 1: Semantic Object Diagram for Student's Record

As illustrated, the student record can be split into three objects; personal details, education and activities. Splitting the record into three specialized objects reduces the complexity of student's record. Several specialized networks were then constructed to represent each object in student's record. The specialized networks are PERSONAL_DETAILS, EDUCATION and ACTIVITIES (see Figure 2).

Figure 2: Multi-Network for Student's Object

An integrating network, STUDENT is constructed to integrate the outputs from the specialized network. This is the third stage or final stage in implementing multi backpropagation network. Figure 3 summarizes the processes involved.

Figure 3: The Multi Backpropagation Modeling Process

3.0 Applying Multi Backpropagation

The construction of Multi backpropagation involves several phases as shown in Figure 3. The main modules are normalization, training module, storing weights and application module.

Normalization
Normalization is a process of splitting data into several smaller data sets. Typically, in developing a system or particularly in intelligent system development, the knowledge engineer is responsible to design the system's processes and decides which attributes or parameters that are important and relevant to the system. The decision is based on the expert advice and his understanding on the problem domain. In this study the grouping criteria are based on the questionnaires forms, which were designed by the experts.

Training the backpropagation network is based on the training algorithm as discussed in previous chapter. The general training process is as follow:

Step 1:

Loading:

Load input data

Step 2:

Feeding:

Broadcast the input signals to the next layer

Step 3:

Calculate Error:

Calculate error information

Step 4:

Updating:

Update the weights

Step 5:

Stopping:

Checks stopping criteria

Storing Weights
As a result of the training or the learning process the best weights will be produced. All specialized network and integrating network will have different sets of weights as they are trained individually. The weights are knowledge that represents the networks. The weights of specialized networks are the knowledge that describes the sub-domain or category presented, whereas the weights of integrating network are a predefined knowledge or rules that tell how the network should interpret the combination of specialized networks. Figure 4 shows samples of weights and how the weights are stored.

Figure 4: Storing Weights

Application Module
Application module uses output from the specialized networks as its input and produced the final output. Prior to that, the network itself is trained with all possible specialized network output that randomly generated by the computer. The complete step-by-step processes in multi backpropagation network application and the role of integrating network are as follows:

Step 1:
For each specialized networks

Loading:

Load input data.
Load weights files.

Testing:

Feed into the network.
Apply the weights.

Output:

Store the output.

Step 2:
For integrating network

Loading:

Get the output of all specialized networks as input.
Load weights files.

Testing:

Feed into the network.
Apply the weights.

Output:

Store the output.

Step 3:

Display report.

Figure 5 shows a complete architecture in training multi backpropagation network. As shown, there are two sections: training and application sections. Training section is where the networks are trained and their weights are stored while, in application section the weights and a new data are used by the networks to produce the output. The outputs of the specialized networks are then fed into the integrated network.

Figure 5: Incorporating Specialized and integrating networks

4.0 Conclusion

Backpropagation network is one of the well known NN model. However, large network is too complex and takes a long time to train. Giving the network some help by splitting the data would enable the network to learn better. In addition, the multi network approach could reduce the network learning time and epoch and increase its learning capability. Therefore, the large network could be divided into several specialized networks. Each network represents a small group of the data and trained separately. Another network will be used to integrate the result and provide the final output.

References
Sima, J. (1994). Loading Deep Networks is Hard. Neural Computation, 6(5), 842-850.

Sima, J. (1996). Back Propagation is Not Efficient. Neural Networks, 9(6), 1017-1023.

Kroenke, D. M. (1997). Database Processing: Fundamentals, Design and Implementation (6th ed). Prentice Hall; USA.

Submitted: 18/04/2004

Article Toolbar

BibTeX entry