Initialization and Estimation of Weights and Bias using Bayesian Technique

Abstract

Researches on artificial neural network models have shown that the method used to initialize and estimate weights and bias always determines the rate at which the network will converge and how efficient the model will perform. Although there are several methods that can be used to initialize and estimate network weights and bias, Bayesian approach is widely considered as a more efficient method for modelling artificial neural networks because it can easily compute the inverses of covariance matrices with high dimension that otherwise are computationally expensive. This study has developed a new filter, the First order Extended Ensemble Filter(FoEEF),that applies numerical solution in solving the inverses of high dimensional covariance matrices from I ^ to ′ s stochastic state-space dynamical models. The research applies the new FoEEF Filter to initializes and estimates the weights and bias of artificial neural network. Comparison on the performance of FoEEF filter in estimating weights and bias are done against the performance of EKF on function estimations by using two different functions, sin(x)+Q function and 3sin(x)3− 3sin(x)2 − sin(x) + 1 function, within 18 and 22 epochs. Emphasis of the performance is placed on the rate at which the models converge. This study gauges the performance by determining the minimum value of the mean square error produced from each epoch and average mean square error minimum value.The outcome from the study showed that FoEEF filter had the lowest value of mean square error and average mean square error which was achieved with the least number of epochs compared to EKF filter. This study concluded that the new FoEEF performed better than EKF and is a more suitable filter for that can be used for initialization and estimation of weights and bias in artificial neural network.