Advanced vs. embedded ML - bridging the gap
Many industrial actors have today not one, but two distinct ML departments - one mostly dedicated to R&D into advanced ML, and the other to the actual embedded implementation of ML algorithms. The embedded ML department usually considers rather simple (e.g. feedforward) networks in inference mode and focuses on providing real-time and numerical precision guarantees using compilers and run-times with limited control over optimization and resource allocation. But advanced ML algorithms require complex control involving back-propagation training (in on-device training and reinforcement learning contexts), or more generally stateful (e.g. recurrent) and conditional (e.g. gated mixture of experts) behaviors. Such advanced control is not readily covered by existing embedded back-ends, and is also often hidden in layers of Python code. We propose an approach to address and bridge this gap.
