TensorFlow.js on CPU now faster with an XNNPACK-powered
#WebAssembly
backend! Whopping 4-20x over previous TF.js CPU backend in pure JavaScript🚀, near-universal coverage, and Node.js compatibility - available right now in the Alpha release
We’re excited to release the Alpha of our WebAssembly backend for TensorFlow.js! 🎉
WASM has wider device support and better numerical stability while getting competitive with WebGL for smaller models.
Share your feedback here →
Today at
@CVPR
we are presenting the joint work () with colleagues from
@DeepMind
and
@GoogleAI
that proves sparsity in neural network weights practically useful for accelerating ConvNet inference on general-purpose processors.
Third-generation NNPACK-family library is here, at ! This time the focus is on accelerating FP32 models in NHWC layout, and it supports both mobile and Web platforms.
@soumithchintala
@jeremyphoward
In hardware, it is cheaper to scale compute units than memory bandwidth, thus the more powerful an accelerator is, the more it is typically imbalanced towards compute performance vs memory performance.
The FP16 inference blog post I wrote while still at Google is finally up:
Enjoy the 2X speedup for floating-point models across most ARM devices, including recent mobile phones, Apple Silicon Macs, ARM64 Windows laptops, and Raspberry Pi 5!
It is already 2024, and I just received the statue for the award I won in 2023 for the work I did in 2022. Big companies’s gears take a long time to turn.
My most awaited
#TFDevSummit
announcement today: XNNPACK is coming in TF Lite 2.3! But if you're adventurous, you can already try it today following instructions at
Today is a historic day for Web computing:
#WebAssembly
#SIMD
proposal progressed to stage 4 of standardization process, i.e. browsers now can enable it by default.
@soumithchintala
@jeremyphoward
Grouped convolution reduce number of parameters and FLOPs by number of groups, but doesn’t affect the number of input & output elements. Thus, overall grouped convolution is more memory-intensive than normal convolution, especially when there’re many groups (e.g. DW convolution).
New goodies from colleagues at
@GoogleAI
! All
#MediaPipe
effects - edge detection, face detection, hair segmentation, and hand tracking - now can run inside a Web browser, powered by XNNPACK and
#WebAssembly
Thumbs up, MediaPipe! 👍👍👍
Check out the blog to see MediaPipe graphs running live in the web browser, enabled by WebAssembly and accelerated by XNNPack ML Inference Library.
Read here →
With sparsity you get up to 3X faster inference and also smaller models (e.g. 16.5 MB -> 6 MB for our sparse MobileNet v1 model). Experimental implementation is available right now in the
@TensorFlow
Lite through the XNNPACK backend:
Sparse inference for computer vision is how you do more (mind-blowing AI processing) with less (milliseconds, joules, and bytes), and it is not just production-ready, it is in production! Try background effects in Google Meet or MediaPipe Hand Tracking to see for yourself!
Today we announce the release of new sparsity features in the
#XNNPACK
acceleration library that is powering
#TensorFlowLite
! Sparse inference improves efficiency without degrading quality in applications like Google Meet's background effects.
The Fresno River Viaduct in Madera County is one of the first completed high-speed rail structures. At nearly 1,600 feet long, high-speed trains will travel over the riverbed and will run parallel with the BNSF Railroad.
#BuildHSR
Accelerating TF Lite 🏁⚡️
See how integration of the XNNPACK library with TensorFlow Lite improves neural network inference performance by 2.3X on average.
Learn more ↓
@soumithchintala
@jeremyphoward
On mobile, where DW convolutions are particularly popular, the situation is different: with XNNPACK, DW 3x3 convolutions deliver ~1/3 GFLOPS of a normal convolution, and the ~9X reduction in the total number of FLOPS in DW sep. conv. more than compensates for the efficiency loss.
In the next 10 years we are going to have:
- super human AI
- full self driving everywhere in the world
- humans on mars
- internet everywhere on earth
- supersonic commercial jets
- cures for major diseases
Keep building, there’s still more to do 🚀
@abacaj
Amusing that many people believe competition will drive LLM cost to zero, some people believe it will drive GPU cost to zero, and no people believe competition will drive electricity cost to zero.
@paulg
A passage from your old essay came a lot to my mind in the recent days: “You could parachute [Sam] into an island full of cannibals and come back in 5 years and he'd be the king.”
@soumithchintala
@jeremyphoward
High compute performance is great for normal convolutions, particularly with many input/output channels, but barely helps the commonly used 3x3 depthwise convolutions, which are bandwith-limited even on mobile CPUs.
Sparsification, or pruning of weights, in convolutional neural networks has a long history as a compression technique, and good support in deep learning frameworks, e.g. Model Optimization Toolkit in TensorFlow. [1/4]
Today Facebook publicly released QNNPACK, open source library for low-precision neural network computations on mobile. Caffe2+QNNPACK = 2x speedup over TFLite + support for grouped conv (CondenseNet, ShuffleNet, RexNeXt).
Until recently, XNNPack was not memory-efficient for throughput-optimized inference, e.g. server-side or burst-mode. Weight cache fix that, by enabling internally repacked weights to be shared across multiple threads.
@cHHillee
We should be optimizing for minimal memory bandwidth. Unfortunately, its very hard to do. First, utilized memory bandwidth is dependent on both hardware and software stack. Secondly, there aren't simple models for it, you need to understand the whole stack.
@sedielem
BatchNorm is free at inference time (fused into preceding convolution/matmul), LayerNorm/GroupNorm has cost (and a big one on highly parallel hardware).
v1.6: native mixed-precision support from NVIDIA (~2x perf improvement), distributed perf improvements, new profiling tool for memory consumption, Microsoft commits to developing and maintaining Windows PyTorch.
Release Notes:
Blog:
One more happy XNNPACK +
#WebAssembly
user: "XNNPACK was great. After some adjustment to the build system we were able to build our code with XNNPACK for WASM. It took us overall 10 days. The result? Almost the same speed as the C++ version. "
The gap between native and in-browser compute is shrinking. Relaxed SIMD augments WebAssembly SIMD with FMA, 8-bit Integer Dot Product, efficient minimum/maximum, efficient float-to-int conversions, efficient lane-wise selects, and more.
@Deepmind
@GoogleAI
The micro-kernels for sparse inference on ARM64 and WebAssembly are already open-sourced in XNNPACK [], and so are pre-trained sparse models [] [4/4]
@giffmana
There’s no business case for running ZLUDA on AMD/Intel = AMD/Intel don’t want to lock themselves into forever being the second-best CUDA implementation. This has little to do with ZLUDA being a viable tech.
State of mind: someone who claims to be Will Smith, but could be as well be a fake, posted a video that claims to be a fake, but could as well be real!
Our recent work [] with colleagues from
@DeepMind
and
@GoogleAI
demonstrates that with a right layout and optimizations sparse inference delivers practical and non-negligible speedups of 1.3X-2.4X on a range of MobileNet and EfficientNet models. [3/4]
@ashvardanian
disagrees (although they don't have Sapphire Rapids, but Alder Lake P should be similar): 3 cycle without masking for any width, 5 cycles with
@Nextremer_nb_o
Operators supported by XNNPACK get offloaded to XNNPACK, others fall back to built-in TFLite kernels, which use Ruy for matrix multiplication.
@GregoryDiamos
@realSharonZhou
Per GeekWire “Adept co-founder and CEO David Luan, the former vice president of engineering at OpenAI, will join Amazon. Adept co-founders Augustus Odena, Maxwell Nye, Erich Elsen, and Kelsey Szot will also move to Amazon, along with a few other employees.”