Solving Quartiles Puzzle with LLM
Ever since becoming an Apple News+ subscriber, I have greatly enjoyed solving the daily Quartiles puzzle. It’s become part of my daily routine and an effective way to relieve stress. I have solved over 100 puzzles so far, achieving my longest streak of 43 days, which unfortunately ended due to travel, and I’m currently on a 33-day streak. Occasionally, the puzzles get a bit challenging, prompting me, given my background, to seek hints or explore how a Large Language Model (LLM) might approach solving them. Given the strong language understanding capabilities of LLMs, I was curious to see their methods for tackling these puzzles. In this article, I explore various LLMs and evaluate their performance on solving Quartiles.
Before we get started, let’s briefly go over the rules of Quartiles.
Understanding the Rules of Quartiles
Quartiles is a word puzzle game where players aim to form words by selecting tiles from a 4x5 grid. Each tile contains 2 to 4 letters. The objective is to combine these tiles to create valid English words, with longer words earning more points.
Scoring System:
- 1-tile word: 1 point
- 2-tile word: 2 points
- 3-tile word: 4 points
- 4-tile word (Quartile): 8 points
In each puzzle, there are five 4-tile words known as “Quartiles.” Finding all five Quartiles grants an additional 40-point bonus. Players can continue forming words to accumulate points, aiming for the highest possible score.
The game features a ranking system based on the total points scored:
- Expert: 100+ points
- Wordsmith: 50+ points
- Artisan: 30+ points
- Builder: 15+ points
- Apprentice: 5+ points
- Novice: 0+ points
Players aim to climb these ranks by forming words and earning points, with the ultimate goal of reaching “Expert” status.
For more details and to play the game, visit the official Quartiles website.
Methodology
To evaluate the performance of various LLMs on the Quartiles puzzle, I will use two primary metrics:
- Number of correctly identified words
- Total points scored
To ensure a fair and consistent comparison, every LLM will receive the exact same puzzle and prompt. Specifically, I selected the puzzle dated 05 May 2025 (solutions available here), which includes 25 words in total with a maximum possible score of 132 points. For reference, solutions to previous Quartiles puzzles can be found here.
The following prompt was provided to each model:
You are solving a word puzzle called Quartiles. The puzzle consists of a 4x5 grid of tiles. Each tile contains 2 to 4 letters. Your goal is to form valid English words by combining 1 to 4 tiles.
Rules:
Use each tile at most once per word.
Do not rearrange letters within a tile.
Tiles must be selected in sequence (left to right).
Words can be made from 1, 2, 3, or 4 tiles.
Scoring:
1-tile word = 1 point
2-tile word = 2 points
3-tile word = 4 points
4-tile word = 8 points
There are exactly five special 4-tile words (called Quartiles) in each puzzle. Finding all five gives a 40-point bonus.
Instructions:
Identify and list as many valid English words as you can.
Prioritize finding the five Quartiles (4-tile words), but also include shorter valid words.
For each word, list the exact tiles used in order.
Input tiles (text format):
[NS] [SCR] [SHU] [NO]
[NG] [PRO] [TYP] [CHE]
[VI] [SIO] [AR] [DER]
[DD] [ERI] [CE] [IPT]
[NDE] [ES] [PLA] [HOL]
(Refer to attached image for visual layout if needed.)
Output format:
Word: RESTART
Tiles used: [RE] [ST] [ART]
Length: 3 tiles
Points: 4
Word: CATER
Tiles used: [CA] [TER]
Length: 2 tiles
Points: 2
Results
Get Hatim Kagalwala’s stories in your inbox
Join Medium for free to get updates from this writer.
OpenAI — ChatGPT 4o (Chat Link)
GPT-4o (“o” for “omni”) is OpenAI’s flagship multimodal model, offering GPT-4-level intelligence with real-time reasoning across text, images, and audio, all at improved speed and efficiency. However, on this particular Quartiles puzzle, the model struggled significantly. It failed to identify valid word combinations, producing mostly invalid outputs, and ultimately scored just 1 point by identifying a single 1-tile word.
OpenAI — ChatGPT o4-mini (Chat Link)
o4-mini is a lightweight, efficient thinking variant of OpenAI’s GPT-4 family, designed to deliver strong performance with lower computational overhead. On the Quartiles puzzle, it performed reasonably well, identifying five valid words: a 1-tile word (Pro), a 2-tile word (No), a 3-tile word (Provider), another 3-tile word (Script), and one Quartile (Provisions). In total, the model scored 16 points, demonstrating a balanced mix of accuracy and tile-length variation.
Gemini — 2.5 Pro (Chat Link)
Gemini 2.5 Pro is Google’s most advanced model for complex tasks. With thinking built in, it showcases strong reasoning and coding capabilities. However, on the Quartiles puzzle, the model performed extremely poorly. It took over a minute to generate a response and identified only two valid 1-tile words (PRO and NO), scoring just 2 points. It also hallucinated several invalid words such as VISIOARDER, PROTYPCHE, and DDERICEIPT, indicating weak pattern recognition in this context.
Qwen3–235B-A22B (Chat Link)
The most powerful Qwen model currently available is Qwen3–235B-A22B. While the earlier Qwen2.5–72B model was seen as a strong performer, Qwen3–235B-A22B is now considered the flagship, delivering competitive results across coding, math, and general reasoning benchmarks — often compared to other top-tier models like DeepSeek R1 and Gemini 2.5 Pro.
On the Quartiles puzzle, however, the model performed poorly. It took approximately 8 minutes to respond and identified only a single valid 1-tile word (NO), scoring 1 point in total.
DeepSeek R1
DeepSeek R1 is an open-source language model designed for efficient reasoning and competitive performance across a range of academic and real-world tasks. On the Quartiles puzzle, the model took an unusually long time to respond — 419 seconds (~7 minutes) — and identified only two valid 1-tile words (Pro and No), scoring a total of 2 points. Given its reasoning focus, I expected stronger performance on this task.
Anthropic Claude 3.7 Sonnet (Chat Link)
Claude 3.7 Sonnet is Anthropic’s most advanced model. On the Quartiles puzzle, the model identified four valid words: Script (2 points), Visions (4 points), Holder (2 points), and Place (2 points), for a total score of 8 points. Surprisingly, it missed several straightforward 1-tile words like Pro and No, which most other models detected with ease.
Summary
Observations
Many models incorrectly identified short segments like ES and AR as valid words. This suggests that the models tend to favor common English letter combinations even when they do not form complete words. All models also generated made-up or incorrect words, which points to challenges in accurately combining tiles under the game’s rules.
One-tile words such as Pro and No were easier for models to find, with the exception of Claude 3.7 Sonnet, which missed both. This was unexpected given how simple these words are.
Among all the models tested, ChatGPT o4-mini performed the best. It identified a mix of short and long words, including one Quartile, and achieved the highest score. Other models were less effective either due to incorrect outputs, slower response times, or fewer valid words.
Next Steps
The Quartiles puzzle includes a shuffle feature that rearranges the tiles. I plan to explore how different LLMs respond to various shuffles by simulating multiple configurations of the same puzzle. I will be working on programming this over the next few days and sharing the results soon.
