How to Compute Weight Decay in Neural Networks

Neuronal networks are formed by the circuitry of electrically excitable cells. These networks are composed of a varying number of input and output neuron layers. The synapses store parameters called "weights," used to manipulate data in the calculations. By means of "weight decay," one can calculate a generalized output of an entire network of neurons. This weight decay provides a certain penalty on the larger weights to regularize output data. If not decayed, the large weights can cause major variances in output data. Therefore, it is very important to use weight elimination and shrink larger weights in order to attain a smooth output.

Things You'll Need

  • MATLAB/ NetLab toolbox
Show More

Instructions

  1. Calculation

    • 1

      Set-up inputs in MATLAB. Open the NetLab toolbox -- "Mathworks" > "NetLab" -- and assign the input parameters as follows: matrix for input vectors, matrix for target values, number of units in the hidden layer, maximum number of iterations for non-convergent networks and the parameters for weight decay.

    • 2

      Use Gaussian regularization statistics for each network layer. Set the weight decay parameters to mlpprior, so that: prior = mlpprior (nin, nhidden, nout, aw1, ab1, aw2, ab2), where "aw1" is the weight decay of first layer weights; "ab1" is the weight decay parameter for the first layer biases; "aw2" is the weight decay parameter for the second layer weights; "ab2" is the weight decay parameter for the second layer of biases. Set-up the network by training the input and output data such that net = mlp(nin, nhidden, nout, function, prior) and [net, options] = netopt (net, options, trainIn, trainOut, method).

    • 3

      Employ standardization of inputs and targets or adjust the penalty term for standard deviations for all inputs and targets. A good generalization can be possible if at least 3 different decay constants are available for the input-hidden, hidden-hidden, and hidden-output weights in the network.

Learnify Hub © www.0685.com All Rights Reserved