Quantization Impact on LLM Quality: A Case Study of Apex
Video coming soon
Presented
Speakers
How much model quality do you give up when quantizing an LLM? At AI Tinkerers Poland’s fourth Wrocław meetup, Artur Morys-Magiera and Lech Kalinowski will present experiments with Apex, our LLM specializing in mobile app development. They’ll compare quantization methods across knowledge, language modeling, and React Native coding tasks to show where smaller models preserve quality and where accuracy starts to drop.
Quantization impact on LLM quality: A case study of Apex, a Gemma-based model
Quantization reduces model size and can improve inference speed, but the effect on quality depends on the method and the task. Artur and Lech will begin with a practical introduction to quantization: how it works and which trade-offs to measure when choosing a configuration.
They will then present experiments with Apex, our LLM specializing in mobile app development. Running on a DGX Spark, the study compares quantized configurations against a 61 GB BF16 Gemma 4 31B-it baseline with a merged rank-8 LoRA adapter.
The experiments cover IQ2_M, IQ3_M, Q3_K_M, IQ4_XS, Q4_K_S, Q4_K_M, Q5_K_S, Q5_K_M, Q6_K, and Q8_0. Evaluation includes a subset of MMLU, WikiText-2, and selected React Native Evals tasks.
Across 25 calendar days, the study includes 344 benchmark runs and 255 model loads, processing 6 million input tokens and 891,000 output tokens. Stored model files total 235 GB.
The talk compares decode speed, accuracy, model size, and Kullback–Leibler divergence to examine changes in the model’s answer distributions. Attendees will see where quantization preserves quality and where the losses become more pronounced.
What you’ll learn
- How the tested quantization methods affect Gemma-based Apex across knowledge, language modeling, and React Native coding tasks.
- The measured quality, decode speed, and model size of each evaluated configuration.
- How Kullback–Leibler divergence reveals shifts in model confidence and decision stability.
- A methodology for running analogous comparisons on other model families using on-premise hardware.
