Edge deployments

Requirements

• You should understand the difficulties of deploying machine learning models on edge devices (e.g. smartphones, raspberry pi's, etc.).
• You should understand the following model optimization techniques and their benefits and drawbacks:
  • Quantization
  • Pruning
  • Knowledge distillation
• You should be able to create a TensorFlow Lite model from a TensorFlow model.
• You should be able to apply the following model optimization techniques to a TensorFlow model:
  • Quantization
  • Pruning

Theory

Articles

• Deep Learning on the Edge: https://www.kdnuggets.com/2018/09/deep-learning-edge.html
• Why Machine Learning on the Edge?: https://towardsdatascience.com/why-machine-learning-on-the-edge-92fac32105e6
• Gemini Nano lets you complete helpful AI tasks without a network connection: https://store.google.com/intl/en/ideas/articles/gemini-nano-google-pixel/
  • Gemini Nano (product page): https://deepmind.google/technologies/gemini/nano/
• Quantization: https://huggingface.co/docs/optimum/concept_guides/quantization
• 8-Bit Quantization and TensorFlow Lite: Speeding up mobile inference with low precision: https://fritz.ai/8-bit-quantization-and-tensorflow-lite/
• A Visual Guide to Quantization: https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-quantization
• Pruning in Deep Learning Model: https://medium.com/@souvik.paul01/pruning-in-deep-learning-models-1067a19acd89
• Knowledge Distillation: Principles, Algorithms, Applications: https://neptune.ai/blog/knowledge-distillation

Video

• TensorFlow in 100 seconds: https://www.youtube.com/watch?v=i8NETqtGHms
• Quantization in deep learning: https://www.youtube.com/watch?v=v1oHf1KV6kM
• Pruning a neural Network for faster training times: https://www.youtube.com/watch?v=tUWnOz0c4pU
• TensorFlow model optimization: Quantization and pruning (TF World '19): https://www.youtube.com/watch?v=3JWRVx1OKQQ
• Inside TensorFlow: TF Model Optimization Toolkit (Quantization and Pruning): https://www.youtube.com/watch?v=4iq-d2AmfRU
• Inside TensorFlow: Quantization aware training: https://www.youtube.com/watch?v=Q1oBXdizXwI

Manuals

• TensorFlow Lite: https://www.tensorflow.org/lite/guide
• TensorFlow Lite examples: https://www.tensorflow.org/lite/examples
• TensorFlow model optimization: https://www.tensorflow.org/model_optimization/guide
• Post-training quantization: https://ai.google.dev/edge/litert/models/post_training_quantization
• Quantization aware training comprehensive guide: https://www.tensorflow.org/model_optimization/guide/quantization/training_comprehensive_guide
• Pruning comprehensive guide: https://www.tensorflow.org/model_optimization/guide/pruning/comprehensive_guide

Online Courses

• Learning TinyML: A Hands-On Course, LinkedIn Learning: https://www.linkedin.com/learning/learning-tinyml-a-hands-on-course/getting-started-with-tinyml
  • HOGENT students have free access to LinkedIn Learning via Academic Software.