Meet react-native-ai: LLMs Running on Mobile, for Real

At the end of April, we announced the release of react-native-ai—a new React Native library that enables LLM execution on-device with React Native apps, powered by MLC LLM Engine. In this blog post, we'll explain our motivation behind this module, our plans, and the challenges we faced during development.

From window.ai demo to Callstack Incubator: How react-native-ai was born

During last summer, I saw a demo of window.ai running inside the latest Canary builds of Google Chrome, and I thought “Is it possible to run AI on-device?”, and the most important question “How hard is it?”.

After some research, I found out there’s a universal engine to run models everywhere. This is somewhat similar to the React mental model, where we also have the possibility of running code everywhere.

I spent a few long evenings creating the first PoC of running AI on-device with React Native, which I presented at the first edition of thegeekconf in Berlin. The demo on stage went great. I also wanted to hack streaming support, but unfortunately, the time between the speaker’s dinner and the next breakfast wasn’t enough 😅

I open-sourced the repo and posted a demo on X, which went pretty well. Then, this topic stayed in the drawer for a few months until I spoke with Mike Grabowski, who wanted to give this project a proper API and a proper release.

So we moved the project under Callstack Incubator, a place for new, exciting (not always fully stable!) stuff.

The API: Plugging into Vercel AI SDK with just a few lines

We decided to expose API from the native side that is compatible with Vercel AI SDK, which Web-based developers might know well because of its flexibility and simplicity. With literally one-line changes, you can switch between models.

import { generateText } from 'ai';

const { text } = await generateText({
    model: openai('gpt-4o-mini'),
    prompt: "why React Native is good choice?"
});

And when I learned that AI SDK has a public API for third-party providers, which can plug into generateText(), streamText, etc. I thought of this kind of API:

import { prepareModel, getModel } from 'react-native-ai';
import { streamText } from 'ai';

const modelId = '';

prepareModel(modelId);

const { textStream } = streamText({
	model: getModel(modelId), // Passes reference into AI SDK
	prompt: "why React Native is a good choice?"
});

After a few weeks, together with Kewin, who wrote the whole implementation on Android and helped with API design a lot, we turned this API into reality.

Under the hood: MLC LLM powering react-native-ai package

Under the hood module is leveraging MLC LLM universal LLM Deployment Engine with ML Compilation that after some research was the best fit for us!

We’ll have a dedicated blog post coming soon regarding how MLC LLM engine works under the hood!

Getting Started: Installing and running models on-device

To start using react-native-ai in your projects, you'll need to set up the environment and understand which models to use. Here's what you need to know:

Installation Process

Setting up react-native-ai requires several steps to ensure the native modules are properly linked:

1. Install the package via npm: npm install react-native-ai
2. Clone the MLC LLM Engine repository and set the MLC_LLM_SOURCE_DIR environment variable
3. Install the mlc_llm CLI tool following the official guide
4. Create a mlc-config.json file in your project root to specify which models you want to use
5. For Android, set up the required NDK environment variables
6. For iOS, add the "Increased Memory Limit" capability in Xcode and install CocoaPods
7. Run npx react-native-ai mlc-prepare to generate all necessary binaries and libraries
8. Add the required polyfills to make Vercel AI SDK work with React Native

To check what are the latest steps to get module working in your project we highly recommend checking out Installation section in README.

Available Models

The library supports various quantized models that can run efficiently on mobile devices. You can specify these in your mlc-config.json file. Some popular options include:

• Mistral-7B-Instruct (various quantization levels)
• Llama-2-7B
• Phi-2
• Gemma-2B

Each model is available in different quantization levels (like q3f16_1, q4f16_1) which affect the model size and performance. To get started we recommend using some models from “Model” section in mlc-ai organization on HuggingFace.

Usage

After you’re done with package setup, and you chose your perfect model you can use API which we described before to download model from HuggingFace, then load it to the memory and generate (or stream) output from LLM 🔥 To learn more about usage check out API section in the README file.

Moving beyond AI SDK and integrating with more model architectures

Just like React allows you to write the code once and then “compiles” it to run on multiple devices, so does MLC with AI. Currently, we focus on integration with the AI SDK because we see great value in it and love using it in our own projects. But since MLC focuses on preparing the ideal runtime for the model, in the future, it should be possible to run other models as well.

Imagine a dedicated runtime for the models you have. Optimized and prepared to get the most out of the device it runs on, with a simple command, preparing the model created in any framework available to run on any device available. This is possible. And even though we know there’s a lot of work ahead, we are excited to start that journey!

Challenges and things to improve

Currently, the setup is quite complex. It requires a lot of steps to correctly set up linking native static libraries, making everything compile correctly on both platforms, etc.

We’ve created a CLI that improves the experience and validates some required env variables, reads config and calls relevant CLI to properly prepare required pieces. We believe that there’s room for improvements to lower the number of steps required to make things work.

In our backlog, there’s also a place for more guides, tutorials, and content presenting what’s possible, what the limits are, and where it can be leveraged—so stay tuned.

To wrap things up

There’s a lot of excitement around on-device LLMs, where we can have our “own” AI accessible from devices that we all carry in our pockets. To catch up with the excitement and align with the DevX on the web, we wanted to bring the best API possible.

We hope that this API will be future-proof and will fit the developers' needs. We’re more than happy to hear the feedback and hopefully implement some improvements!

Thank you

Huge thanks to Kewin for supporting me in development, for creating the whole Android module, and for helping design the user-facing API 🙏 Also thanks to Jamon for sharing react-native-ai package 🫡