Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment

Training neural networks for neuromorphic deployment is non-trivial.  There have been a variety of approaches proposed to adapt back-propagation or back-propagation-like algorithms appropriate for training. Considering that these networks often have very different performance characteristics than traditional neural networks, it is often unclear how to set either the network topology or the hyperparameters to achieve optimal performance. One such algorithm is Whetstone (developed at Sandia National Laboratories), which trains networks that have binary communication, and are amenable for mapping onto spiking neuromorphic hardware. This approach not only has all of the hyperparameters associated with traditional neural network or deep learning network training, but also additional hyperparameters of its own. These hyperparameters have a significant effect on the performance of the algorithm. The presented framework provides a way to automatically discover an optimal hyperparameters that drastically improves the performance of the network. For a simple case study, the researchers also showed that this hyperparameter optimization approach can discover similar optimal hyperparameter set as a grid search, but with far fewer evaluations and far less time.