Wu Shuang, a PhD Candidate at the Center for Brain-Inspired Computing Research (CBICR) at Tsinghua University, gave an oral report at the 2018 International Conference on learning representations (ICLR). It is the only paper included by ICLR as an oral report with a Chinese institution as the first affiliation.

ICLR is the top event in the field of deep learning and is highly regarded as the uncrowned king, attracting much attention and involvement from many hi-tech companies, such as Google, Facebook, Deepmind, Amazon, and IBM. The four-day ICLR 2018 was held at the Vancouver Convention Center, Canada, on April 30th, 2018, local time. The chairmen were Yoshua Bengio from the University of Montreal and Yann LeCun from New York University & Facebook, alongside three prominent figures in the deep learning field. Only 23 papers were selected from more than one thousand submissions to the ICLR 2018.

Picture 1. Wu Shuang (on the left) and Li Guoqi (on the right), authors of the paper selected at ICLR 2018.

Wu’s oral report, entitled Training and Inference with Integers in Deep Neural Networks, as the 15th one, was arranged on May 2nd.

The report explores deep neural networks with discrete parameters and their deployment in embedded systems, which have been active and promising topics. Although previous works have successfully reduced precision in inference, transferring both training and inference processes to low-bitwidth integers has not been demonstrated simultaneously. In the report, they develop a new method termed “WAGE” to discretize both training and inference, where weights (W), activations (A), gradients (G) and errors (E) among layers are shifted and linearly constrained to low-bitwidth integers. To perform pure discrete dataflow for fixed-point devices, they further replace batch normalization by a constant scaling layer and simplify other components that are arduous for integer implementation. Improved accuracies can be obtained on multiple datasets, which indicates that WAGE somehow acts as a type of regularization. Empirically, they demonstrate the potential to deploy training in hardware systems such as integer-based deep learning accelerators and neuromorphic chips with comparable accuracy and higher energy efficiency, which is crucial to future AI applications in variable scenarios with transfer and continual learning demands.

Picture 2. Wu Shuang was giving an oral report at the ICLR 2018.