Think of Matmul like a special kind of multiplication that helps neural networks learn and represent complex relationships between data.

Imagine you have two sets of numbers or matrices that represent features or patterns in your data.

Now Matrix Multiplication or Matmul combines these sets by multiplying and adding corresponding elements, creating a totally new matrix that represents the relationship between the original matrices or features.

This process helps neural networks transform input into meaningful output like predictions or classification.

Imagine you have a matrix of images and a matrix of filters that detect edges.

Matmul combines these matrices to produce a new matrix that represents the edges in each image.

This paper, which you can see on your screen, has taken the social media of AI by storm by giving it a new twist.

Before I tell you in detail what exactly this paper has done here, let me give a bit more technical detail on this Matmul because it's a pivotal concept in this whole scenario to understand this better.

Matrix Multiplication or Matmul, as I said, it's a quite dominant operation in most of the models these days, where dense layers involve vector matrix multiplication.

Convolutions can be implemented as block sparse VMMs with shared weight and self-attention relies on matrix-matrix multiplication.

The prevalence of Matmul is primarily due to graphics processing units or GPUs being optimized for Matmul operations.

By leveraging Compute Unified Device Architecture or CUDA and highly optimized linear algebra libraries such as Kaplas, the Matmul operation can be efficiently parallelized and accelerated.

This optimization was a key factor in the victory of AlexNet, by the way, which is a very famous competition.

Despite its prevalence in deep learning, Matmul operations account for the dominant portion of computational expense, often consuming the majority of the execution time and memory access during both training and inference phases.

A lot of work has already been done where Matmul has been replaced with simpler operations through two main stages.

First, strategy or stage involves substituting Matmul with elementary operations and the second approach employs binary or ternary quantization, simplifying Matmul to operations where values are either flipped or zeroed out before accumulation.

In this paper, these researchers have developed the first scalable Matmul-free language model or Matmul-free LM by using additive operations in dense layers and element-wise enameled products for self-attention-like functions.

Specifically, ternary weights eliminate Matmul in dense layers, similar to BNNs.

To remove Matmul from self-attention, they have optimized the GRU to rely solely on element-wise products and show that this model competes with state-of-the-art transformers while eliminating all Matmul operations.

In this diagram, you can see the overview of this Matmul-free LM where the sequence of operations are shown for vanilla self-attention and the Matmul-free token mixer which is on the top right and ternary accumulations.

The Matmul-free LM employs a Matmul-free token mixer and a Matmul-free channel mixer to while reducing compute cost.

Similarly, if you look at this diagram, this primarily shows you what exactly is the comparison with other models and a lot of other stuff around performance comparison and analysis of different models.

I will also drop the link to this paper in Vute's description and you can read it as your layer because it's quite an interesting read in my humble opinion.

Now, look at this diagram.

This actually shows you a bit more in-depth and easy way as what is happening here.

So, to test the power usage and effectiveness of the Matmul-free LM on custom hardware that can better exploit ternary operations, these researchers have created an FPGA accelerator in System Verilog and this is the whole overview of it.

There are four functional units in this design, row-wise operation, root-means-square, load-store and ternary matrix multiplication and they each allow for simple out-of-order execution.

They also wrote a custom assembler for their custom instruction set which was used to convert assembly files into an instruction row and there is a lot of detail around there that they also have this register router as you can see in the middle that delegates incoming instructions into available registers.

The register file consists of eight registers, each storing one vector in a separate SRAM array.

Each register SRAM array has a read and write port that are delegated to at most one instruction at a time.

If an instruction requests access to a functional unit or a register that is busy, the program counter will stall until the functional unit or register has been freed.

If two instructions do not block each other, they execute simultaneously and then also we see there is a root-means-square functional unit that uses a specialized hardware.

I'll go to preserve precision and there are few stages of it and then we have this ternary matrix multiplication functional unit that takes in a DRAM address for a ternary matrix then performs a T-MATMUL on specified vector.

This architecture entirely places a ternary matrices and DRAM while running a T-MATMUL instruction and SRAM FIFO is simultaneously filled with sequential DRAM fetch result.

So the results are simply amazing and the performance and area impact shows that this is the simplest case where the core only receives 8-bit at a time from memory.

All in all, amazing demonstration of the feasibility and effectiveness of the first scalable MATMUL-free language model.

This work challenges the paradigm that MATMUL operations are indispensable for building high-performance language models and paves the way for development of more efficient and hardware-friendly architectures.

They have also achieved performance on par with state-of-the-art transformers while eliminating the need for MATMUL operations with an optimized implementation that significantly enhances both training and inference efficiency reducing both memory usage and latency.

As the demand for deploying language models on various platforms grows, MATMUL-free LMs present a promising direction for creating models that are both effective and resource efficient.

So amazing, amazing stuff and by prioritizing the development and deployment of MATMUL-free architecture such as this one, the future of LMs will only become more accessible, efficient and sustainable and probably that is why this paper has really taken the AI social media by storm and sort of everyone is talking about it because this seems real and I don't think so this will just remain present in the paper.

I think this we are going to see a lot of implementation of it.

We will see a lot of memory reduction, efficiency gains and hardware optimization.

Let me know what you think.

If you like the content, please consider subscribing to the channel and if you are already subscribed then please share it among your network as it helps.

Thanks for watching.