Baidu Research today unveiled the next generation of DeepBench, the open source deep learning benchmark that now includes measurement for inference. The announcement was made at the O’Reilly AI Conference in New York.

In September of 2016, Baidu released the initial version of DeepBench, which became the first tool to be opened up to the wider deep learning community to evaluate how different processors perform when they are used to train deep neural networks. Since its initial release, several companies have used and contributed to the DeepBench platform, including Intel, Nvidia, and AMD.

Following positive feedback from peers across the AI industry and academia, Baidu Research has now incorporated requests to include the measurement of deep learning inference, in addition to training, across different hardware platforms. Inference involves using a previously trained model to make predictions on a new data set.

“Measuring inference is critical,” said Dr. Greg Diamos, Senior Researcher at Baidu Research Silicon Valley AI Lab. “It covers the operations needed to run neural networks on a device, be it in the cloud, on a phone or a wearable. A better understanding of performance of inference means better chips and neural networks in real products.”

Benchmarking inference is a challenging problem. Many applications that have been enabled by deep learning each have their own unique performance characteristics and requirements. In addition, there are several different deployment platforms. DeepBench attempts to solve this problem by benchmarking fundamental operations required for inference.

“Speed is key to training neural networks, and the first step to improving speed is having an accurate measurement of performance,” said Sharan Narang, Systems Researcher at Baidu Research Silicon Valley AI Lab. “With the addition of the ability to measure inference, researchers will now have a more comprehensive benchmark for the performance of their AI hardware”.

In addition to measuring inference performance, DeepBench provides new kernels for training from several different deep learning models. It also sets new minimum precision requirements for training. Based on a study conducted by Baidu Research, DeepBench establishes 16 bit floating point for multiplication and 32 bit floating point for addition for training operations. It also establishes 8 bit fixed point for multiplication and 32 bit fixed point for addition.

DeepBench also provides results for training and inference across a variety of processors. Baidu Research has collected results for server deployment as well as mobile deployment platforms such as the iPhone.

For more information on DeepBench, please visit https://svail.github.io/DeepBench-update/