Running LLaMA on Raspberry Pi, NPX, and Pixel 6: Exploring the Possibilities of Open Source LLMs

source: TIME The increasing demand for large language models (LLMs) in various applications, including language translation, content generation, and conversational AI, has led to the emergence of several models such as GPT-3, T5, and RoBERTa. However, these models are mostly owned by large companies and come with certain limitations and fees, which have led to the need for open-source solutions. Open-source solutions allow for greater accessibility, flexibility, and customization of LLMs, enabling researchers and developers to build and experiment with models without censorship or financial constraints.

The Rise of LLaMA: An Open Source LLM

Meta AI recently released LLaMA, an open-source LLM with parameter sizes ranging from 7B to 65B, claiming to match the quality and speed of OpenAI's GPT-3. While open-source alternatives like GPT-J exist, they require significant GPU RAM and storage space, making them inaccessible to most consumers. LLaMA, on the other hand, can run on readily available consumer-level hardware, making it an attractive alternative to GPT-3.

The Challenge with LLaMA: Restricted Weights

While LLaMA was open source, Meta AI held back the "weights," which are the trained "knowledge" stored in a neural network, for qualified researchers only. This limitation meant that LLaMA could not be fully utilized and optimized by everyone.

The Leaked LLaMA Weights: The Turning Point

On March 2, 2023, someone leaked the LLaMA models via BitTorrent, breaking the restrictions imposed by Meta AI. This event triggered an explosion of development surrounding LLaMA, making it accessible to a broader audience. The leak enabled independent AI researchers to experiment with LLaMA, leading to several groundbreaking advancements in a short period. Source: The Hans India

The Advancements in LLaMA: Lightning Speed Progress

In just over a week since the leak, LLaMA has undergone significant advancements, making it possible to run the model on various devices, including Mac laptops, Windows, Raspberry Pi, and Pixel 6 phones. These developments have made it possible to run the LLM on readily available hardware, making it more accessible and convenient for researchers and developers.

Georgi Gerganov and the llama.cpp Tool

One notable development is the creation of the llama.cpp tool by a software developer named Georgi Gerganov. The tool can run LLaMA locally on a Mac laptop, and soon after, people worked out how to run it on Windows as well. Gerganov's tool has made it possible for researchers to experiment with LLaMA without relying on cloud-based APIs, which can be costly and have limitations. Also Read: The First AI-Driven Radio Service in the World

LLaMA on Raspberry Pi, NPX, and Pixel 6

Another groundbreaking development is the successful running of LLaMA on Raspberry Pi, albeit slowly, and on NPX, a node.js execution tool. LLaMA also got to run on a Pixel 6 phone, though slowly, showcasing the vast possibilities that come with the open-source model.

Stanford's Alpaca 7B

On March 13, 2023, Stanford released Alpaca 7B, an instruction-tuned version of LLaMA 7B that "behaves similarly to OpenAI's "text-davinci-003" but runs on much less powerful hardware." This version of LLaMA is an improvement on the original version, and it is expected that more developments and optimizations will follow as more people experiment with the open-source LLM.