Qwen3.8-27B

At xhigh effort, Qwen3.8-27B scores 87.5% on ARC-AGI v1 Semi-Private at $0.2152/task and 42.4% on ARC-AGI v2 Semi-Private at $0.4469/task. Medium reasoning performs similarly to low on v1 but scores lower on both v2 sets, despite using more completion tokens. Baseten confirmed that Qwen's chat template adds effort-specific instructions for low and xhigh, but not medium. This may help explain the regression: the settings change the instructions given to the model, rather than simply increasing its thinking budget. Evaluated using dedicated inference on Baseten. Estimated cost uses recorded token usage priced at Alibaba Cloud International's listed rates of $0.425 per million input tokens and $2.55 per million output tokens, using the September 30, 2026 pricing snapshot. This estimate does not represent actual Baseten charges.

ARC-AGI 2 leaderboard

Verified scores

Tasks & environments

Pass/fail per reasoning level across each benchmark.

ARC-AGI-2 Public Eval

120 tasks

Task	XHigh ✓✗	Medium ✓✗	Low ✓✗	None ✓✗
0934a4d8	✓	✗	✗	✗
135a2760	✗	✗	✗	✗
136b0064	✓	✗	✓	✗
13e47133	✗	✗	✗	✗
142ca369	✗	✗	✗	✗
16b78196	✗	✗	✗	✗
16de56c4	✓	✗	✗	✗
1818057f	✗	✗	✓	✗
195c6913	✗	✗	✗	✗
1ae2feb7	✓	✓	✓	✓
20270e3b	✓	✗	✗	✗
20a9e565	✓	✗	✗	✗
21897d95	✗	✗	✗	✗
221dfab4	✗	✗	✗	✗
247ef758	✗	✗	✗	✗
269e22fb	✗	✗	✗	✗
271d71e2	✗	✗	✗	✗
28a6681f	✗	✗	✗	✗
291dc1e1	✗	✓	✓	✗
2b83f449	✗	✗	✗	✗
2ba387bc	✓	✓	✓	✓
2c181942	✗	✗	✗	✗
2d0172a1	✓	✗	✗	✗
31f7f899	✓	✓	✓	✗
332f06d7	✗	✗	✗	✗
35ab12c3	✓	✗	✗	✗
36a08778	✗	✗	✗	✗
38007db0	✗	✓	✓	✗
3a25b0d8	✗	✗	✗	✗
3dc255db	✓	✗	✗	✗
3e6067c3	✗	✗	✗	✗
409aa875	✓	✗	✗	✗
446ef5d2	✗	✗	✗	✗
45a5af55	✗	✗	✗	✗
4a21e3da	✓	✓	✓	✗
4c3d4a41	✗	✓	✗	✗
4c416de3	✗	✗	✗	✗
4c7dc4dd	✗	✗	✗	✗
4e34c42c	✗	✗	✗	✗
53fb4810	✗	✓	✗	✗
5545f144	✗	✗	✗	✗
581f7754	✓	✗	✗	✗
58490d8a	✗	✗	✓	✗
58f5dbd5	✓	✗	✗	✗
5961cc34	✗	✗	✗	✗
5dbc8537	✗	✗	✗	✗
62593bfd	✗	✗	✗	✗
64efde09	✗	✗	✗	✗
65b59efc	✓	✓	✓	✗
67e490f4	✓	✗	✗	✗
6e453dd6	✗	✗	✓	✗
6e4f6532	✗	✗	✗	✗
6ffbe589	✗	✗	✗	✗
71e489b6	✗	✗	✗	✗
7491f3cf	✓	✗	✗	✗
7666fa5d	✓	✗	✗	✗
78332cb0	✗	✗	✗	✗
7b0280bc	✗	✗	✗	✗
7b3084d4	✓	✗	✗	✗
7b5033c1	✗	✗	✗	✗
7b80bb43	✗	✗	✗	✗
7c66cb00	✓	✗	✗	✗
7ed72f31	✗	✗	✗	✗
800d221b	✗	✗	✗	✗
80a900e0	✓	✗	✗	✗
8698868d	✗	✗	✗	✗
88bcf3b4	✗	✗	✗	✗
88e364bc	✗	✗	✗	✗
89565ca0	✗	✗	✗	✗
898e7135	✓	✗	✗	✗
8b7bacbf	✗	✗	✗	✗
8b9c3697	✗	✗	✗	✗
8e5c0c38	✓	✗	✗	✗
8f215267	✓	✓	✓	✗
8f3a5a89	✗	✓	✓	✗
9385bd28	✗	✗	✗	✗
97d7923e	✗	✗	✗	✗
981571dc	✓	✓	✓	✗
9aaea919	✗	✗	✗	✗
9bbf930d	✗	✗	✗	✗
a251c730	✗	✗	✗	✗
a25697e4	✗	✗	✗	✗
a32d8b75	✗	✗	✗	✗
a395ee82	✗	✗	✓	✗
a47bf94d	✗	✗	✗	✗
a6f40cea	✓	✗	✗	✗
aa4ec2a5	✗	✗	✗	✗
abc82100	✗	✗	✗	✗
b0039139	✗	✗	✗	✗
b10624e5	✗	✗	✓	✗
b5ca7ac4	✓	✗	✗	✗
b6f77b65	✗	✗	✗	✗
b99e7126	✓	✗	✗	✗
b9e38dc0	✗	✗	✗	✗
bf45cf4b	✓	✓	✓	✓
c4d067a0	✗	✗	✗	✗
c7f57c3e	✗	✗	✗	✗
cb2d8a2c	✓	✗	✗	✗
cbebaa4b	✗	✗	✗	✗
d35bdbdc	✓	✗	✗	✗
d59b0160	✓	✗	✗	✗
d8e07eb2	✓	✓	✓	✗
da515329	✗	✗	✗	✗
db0c5428	✓	✗	✗	✗
db695cfb	✗	✓	✗	✗
dbff022c	✓	✗	✗	✗
dd6b8c4b	✗	✗	✗	✗
de809cff	✗	✗	✗	✗
dfadab01	✗	✗	✗	✗
e12f9a14	✗	✗	✗	✗
e3721c99	✗	✗	✗	✗
e376de54	✓	✗	✗	✗
e8686506	✓	✓	✓	✗
e87109e9	✗	✗	✗	✗
edb79dae	✓	✗	✗	✗
eee78d87	✗	✗	✗	✗
f560132c	✗	✗	✗	✗
f931b4a8	✓	✗	✗	✗
faa9f03d	✗	✗	✗	✗
fc7cae8d	✗	✗	✗	✗

ARC-AGI-1 Public Eval

400 tasks

Task	XHigh ✓✗	Medium ✓✗	Low ✓✗	None ✓✗
00576224	✓	✓	✓	✓
009d5c81	✓	✓	✓	✓
00dbd492	✓	✓	✗	✗
03560426	✓	✓	✓	✓
05a7bcf2	✗	✓	✗	✗
0607ce86	✓	✗	✓	✗
0692e18c	✓	✓	✓	✗
070dd51e	✓	✓	✓	✗
08573cc6	✗	✗	✓	✗
0934a4d8	✓	✗	✗	✗
09c534e7	✗	✗	✗	✗
0a1d4ef5	✓	✓	✓	✓
0a2355a6	✓	✓	✓	✗
0b17323b	✓	✓	✓	✗
0bb8deee	✓	✓	✓	✓
0becf7df	✓	✓	✓	✓
0c786b71	✓	✓	✓	✓
0c9aba6e	✓	✓	✓	✓
0d87d2a6	✗	✗	✗	✗
0e671a1a	✓	✓	✓	✓
0f63c0b9	✓	✓	✓	✗
103eff5b	✓	✓	✓	✗
11e1fe23	✓	✗	✗	✗
12422b43	✓	✓	✓	✓
12997ef3	✓	✓	✓	✓
12eac192	✓	✓	✓	✓
136b0064	✓	✓	✓	✗
13713586	✓	✗	✗	✓
137f0df0	✓	✓	✓	✓
140c817e	✓	✓	✓	✓
14754a24	✗	✗	✗	✗
15113be4	✓	✗	✗	✗
15663ba9	✓	✓	✓	✗
15696249	✓	✓	✓	✓
16b78196	✗	✗	✗	✗
17b80ad2	✓	✗	✗	✗
17cae0c1	✓	✓	✓	✓
18419cfa	✗	✗	✗	✗
184a9768	✓	✓	✓	✗
195ba7dc	✓	✓	✓	✗
1990f7a8	✓	✓	✓	✓
19bb5feb	✓	✓	✓	✓
1a2e2828	✓	✓	✓	✓
1a6449f1	✓	✓	✓	✗
1acc24af	✗	✗	✗	✗
1c02dbbe	✗	✓	✗	✓
1c0d0a4b	✓	✓	✓	✗
1c56ad9f	✓	✓	✓	✗
1d0a4b61	✓	✓	✓	✓
1d398264	✓	✓	✓	✓
1da012fc	✗	✗	✗	✗
1e81d6f9	✓	✓	✓	✗
1e97544e	✓	✓	✓	✓
2037f2c7	✓	✓	✓	✗
2072aba6	✓	✓	✓	✓
20818e16	✓	✓	✓	✗
20981f0e	✓	✓	✓	✗
212895b5	✗	✗	✗	✗
21f83797	✓	✓	✓	✗
22a4bbc2	✓	✓	✓	✗
25094a63	✗	✗	✗	✗
2546ccf6	✗	✗	✗	✗
256b0a75	✗	✗	✓	✗
2685904e	✓	✓	✓	✗
2697da3f	✓	✓	✓	✗
2753e76c	✓	✓	✓	✓
27a77e38	✓	✓	✓	✗
27f8ce4f	✓	✓	✗	✓
281123b4	✓	✓	✓	✓
292dd178	✓	✓	✓	✓
29700607	✓	✓	✓	✓
2a5f8217	✓	✗	✓	✓
2b01abd0	✓	✓	✓	✓
2c0b0aff	✓	✓	✓	✗
2c737e39	✓	✓	✓	✓
2f0c5170	✗	✓	✗	✗
310f3251	✓	✓	✓	✓
3194b014	✓	✓	✓	✓
319f2597	✓	✓	✓	✓
31adaf00	✗	✗	✓	✗
31d5ba1a	✓	✓	✓	✗
32e9702f	✓	✓	✗	✓
332efdb3	✓	✓	✓	✓
3391f8c0	✓	✗	✓	✓
33b52de3	✓	✓	✓	✓
3490cc26	✓	✗	✗	✗
34b99a2b	✓	✓	✓	✓
351d6448	✓	✗	✗	✗
358ba94e	✓	✓	✓	✓
37d3e8b2	✓	✗	✓	✗
3979b1a8	✓	✓	✓	✗
3a301edc	✓	✓	✓	✓
3b4c2228	✓	✓	✓	✓
3d31c5b3	✓	✓	✓	✓
3ed85e70	✓	✗	✗	✗
3ee1011a	✓	✓	✓	✓
3f23242b	✓	✓	✓	✓
40f6cd08	✓	✗	✗	✗
414297c0	✓	✓	✓	✗
423a55dc	✓	✓	✓	✓
42918530	✓	✓	✓	✓
42a15761	✓	✓	✗	✗
4364c1c4	✓	✓	✓	✗
456873bc	✓	✗	✗	✗
45737921	✓	✓	✓	✓
45bbe264	✓	✓	✓	✓
477d2879	✓	✗	✗	✗
47996f11	✓	✗	✗	✗
48131b3c	✓	✓	✓	✗
4852f2fa	✓	✓	✓	✓
48f8583b	✓	✓	✓	✓
4aab4007	✓	✗	✓	✗
4acc7107	✓	✓	✗	✗
4b6b68e5	✗	✗	✗	✗
4c177718	✓	✓	✓	✗
4cd1b7b2	✓	✓	✓	✗
4e45f183	✓	✗	✓	✗
4e469f39	✓	✓	✓	✓
4f537728	✓	✓	✓	✗
4ff4c9da	✓	✓	✗	✗
505fff84	✓	✓	✓	✓
506d28a5	✓	✓	✓	✓
50a16a69	✓	✓	✓	✓
50aad11f	✓	✓	✓	✓
50f325b5	✗	✗	✗	✗
516b51b7	✓	✓	✓	✓
5207a7b5	✓	✓	✓	✗
5289ad53	✓	✓	✓	✓
52fd389e	✓	✗	✗	✗
54db823b	✓	✓	✓	✗
55059096	✓	✗	✗	✓
551d5bf1	✗	✗	✗	✗
55783887	✓	✓	✓	✓
575b1a71	✓	✓	✓	✗
5783df64	✓	✓	✓	✓
5833af48	✓	✓	✗	✗
58743b76	✓	✓	✓	✓
58e15b12	✓	✓	✓	✗
59341089	✓	✓	✓	✓
5a5a2103	✓	✓	✓	✗
5af49b42	✓	✓	✓	✓
5b526a93	✓	✓	✓	✓
5b692c0f	✓	✗	✓	✗
5b6cbef5	✓	✓	✓	✓
5d2a5c43	✓	✓	✓	✓
5ffb2104	✓	✓	✓	✗
604001fa	✓	✓	✓	✗
60a26a3e	✓	✓	✓	✓
60c09cac	✓	✓	✓	✗
626c0bcc	✓	✓	✓	✗
62ab2642	✓	✓	✗	✗
62b74c02	✓	✓	✓	✗
639f5a19	✗	✗	✗	✗
642248e4	✓	✓	✓	✓
642d658d	✓	✓	✓	✗
64a7c07e	✓	✓	✓	✗
66e6c45b	✓	✓	✓	✓
66f2d22f	✓	✓	✓	✗
67636eac	✓	✓	✓	✗
67b4a34d	✓	✓	✓	✓
67c52801	✓	✓	✓	✗
68b67ca3	✓	✓	✓	✓
692cd3b6	✓	✗	✓	✗
695367ec	✓	✓	✓	✓
696d4842	✓	✓	✓	✗
69889d6e	✓	✓	✓	✗
6a11f6da	✓	✓	✓	✓
6ad5bdfd	✓	✓	✓	✗
6df30ad6	✓	✓	✓	✓
6ea4a07e	✓	✓	✓	✗
6f473927	✓	✓	✓	✓
7039b2d7	✓	✓	✓	✗
705a3229	✓	✓	✓	✓
712bf12e	✓	✓	✓	✗
72207abc	✓	✗	✓	✓
72a961c9	✓	✓	✓	✗
73182012	✓	✓	✓	✗
73c3b0d8	✗	✓	✗	✗
73ccf9c2	✗	✗	✗	✗
759f3fd3	✓	✓	✓	✗
762cd429	✓	✓	✓	✗
770cc55f	✓	✓	✓	✗
782b5218	✓	✓	✓	✓
79369cc6	✓	✗	✓	✗
7953d61e	✓	✓	✓	✓
79fb03f4	✗	✗	✗	✗
7bb29440	✓	✓	✓	✓
7c8af763	✓	✓	✓	✓
7c9b52a0	✓	✓	✓	✓
7d18a6fb	✓	✓	✓	✓
7d1f7ee8	✗	✗	✗	✗
7d419a02	✓	✗	✗	✗
7e02026e	✓	✓	✓	✓
7ee1c6ea	✓	✓	✓	✓
817e6c09	✓	✓	✓	✓
81c0276b	✓	✓	✓	✓
833dafe3	✓	✓	✓	✓
845d6e51	✓	✓	✓	✗
84db8fc4	✓	✓	✓	✓
84f2aca1	✓	✓	✓	✓
8597cfd7	✓	✓	✓	✗
85b81ff1	✓	✓	✓	✗
85fa5666	✓	✓	✓	✓
8719f442	✓	✓	✓	✗
88207623	✓	✓	✓	✓
891232d6	✗	✗	✗	✗
896d5239	✗	✗	✗	✗
8a371977	✓	✓	✓	✓
8b28cd80	✗	✗	✗	✗
8ba14f53	✓	✓	✗	✗
8cb8642d	✗	✗	✗	✗
8dae5dfc	✓	✓	✓	✓
8e2edd66	✓	✓	✓	✓
8ee62060	✓	✓	✓	✓
8fbca751	✗	✗	✗	✓
90347967	✓	✗	✓	✓
903d1b4a	✓	✓	✓	✗
9110e3c5	✓	✓	✓	✓
917bccba	✓	✓	✓	✓
929ab4e9	✓	✓	✓	✗
92e50de0	✓	✓	✗	✗
9356391f	✓	✓	✓	✗
93b4f4b3	✓	✓	✓	✗
93c31fbe	✗	✗	✗	✗
94133066	✓	✗	✗	✗
94414823	✓	✓	✓	✓
94be5b80	✗	✓	✓	✗
95a58926	✓	✗	✗	✗
963f59bc	✓	✓	✓	✗
96a8c0cd	✓	✗	✗	✗
97239e3d	✓	✓	✓	✗
9772c176	✓	✓	✓	✗
981571dc	✓	✓	✓	✗
992798f6	✓	✓	✓	✗
99306f82	✓	✓	✗	✗
9a4bb226	✓	✓	✓	✓
9b2a60aa	✓	✓	✗	✗
9b365c51	✓	✓	✓	✓
9b4c17c4	✓	✓	✓	✗
9bebae7a	✓	✓	✓	✓
9c1e755f	✓	✓	✓	✓
9c56f360	✓	✓	✓	✓
9caba7c3	✓	✗	✓	✗
9ddd00f0	✓	✓	✓	✓
9def23fe	✓	✓	✓	✗
9f27f097	✓	✓	✓	✓
a04b2602	✓	✓	✗	✗
a096bf4d	✓	✓	✗	✗
a3f84088	✗	✗	✗	✗
a406ac07	✓	✓	✓	✓
a57f2f04	✓	✗	✗	✗
a59b95c0	✓	✓	✓	✓
a680ac02	✓	✓	✓	✓
a8610ef7	✓	✗	✗	✗
a934301b	✓	✓	✓	✗
aa18de87	✓	✓	✓	✓
aa300dc3	✓	✓	✓	✓
aa4ec2a5	✗	✗	✗	✗
aab50785	✓	✓	✓	✓
ac0c5833	✗	✗	✗	✗
ac2e8ecf	✗	✗	✓	✗
ac3e2b04	✓	✓	✓	✗
ac605cbb	✓	✓	✓	✗
ad7e01d0	✓	✓	✓	✓
ae58858e	✓	✓	✓	✓
aee291af	✓	✓	✓	✓
af22c60d	✓	✓	✓	✓
af24b4cc	✓	✓	✓	✓
b0722778	✓	✗	✓	✗
b0f4d537	✓	✓	✓	✗
b15fca0b	✓	✓	✓	✗
b1fc8b8e	✓	✓	✓	✗
b20f7c8b	✓	✓	✓	✗
b457fec5	✓	✗	✗	✗
b4a43f3b	✓	✓	✓	✓
b7999b51	✗	✓	✓	✗
b7cb93ac	✓	✓	✓	✗
b7f8a4d8	✓	✗	✓	✗
b7fb29bc	✓	✓	✓	✓
b942fd60	✓	✗	✓	✗
b9630600	✗	✗	✗	✗
ba9d41b8	✓	✗	✗	✗
baf41dbf	✓	✓	✗	✗
bb52a14b	✓	✗	✗	✗
bbb1b8b6	✓	✓	✓	✗
bc4146bd	✓	✓	✓	✓
bcb3040b	✓	✓	✓	✗
bd14c3bf	✓	✗	✗	✗
be03b35f	✓	✓	✓	✗
bf32578f	✓	✗	✗	✗
bf699163	✓	✓	✓	✗
bf89d739	✓	✓	✓	✓
c074846d	✓	✓	✓	✗
c1990cce	✓	✓	✗	✗
c3202e5a	✓	✓	✓	✗
c35c1b4c	✓	✗	✗	✗
c48954c1	✓	✓	✓	✓
c62e2108	✓	✓	✓	✗
c64f1187	✓	✓	✓	✗
c658a4bd	✓	✓	✓	✓
c663677b	✓	✓	✓	✓
c6e1b8da	✓	✗	✗	✗
c7d4e6ad	✓	✓	✓	✓
c87289bb	✓	✗	✗	✗
c8b7cc0f	✓	✓	✓	✓
c92b942c	✗	✗	✓	✗
c97c0139	✗	✗	✗	✗
ca8de6ea	✓	✓	✓	✓
ca8f78db	✓	✓	✓	✓
cad67732	✓	✓	✓	✗
cb227835	✓	✓	✓	✓
ccd554ac	✓	✓	✓	✓
cd3c21df	✓	✓	✓	✓
ce039d91	✓	✗	✗	✗
ce8d95cc	✓	✓	✓	✗
cf133acc	✓	✓	✓	✗
cfb2ce5a	✓	✓	✓	✗
d017b73f	✗	✓	✗	✗
d19f7514	✓	✓	✓	✓
d282b262	✓	✓	✓	✗
d2acf2cb	✓	✓	✓	✓
d304284e	✓	✓	✓	✗
d37a1ef5	✓	✓	✗	✗
d47aa2ff	✓	✗	✗	✗
d492a647	✓	✓	✗	✓
d4b1c2b1	✓	✓	✓	✓
d4c90558	✓	✓	✓	✓
d56f2372	✓	✓	✓	✗
d5c634a2	✓	✓	✓	✗
d931c21c	✗	✗	✗	✗
d94c3b52	✓	✗	✗	✗
da2b0fe3	✓	✓	✓	✓
da515329	✗	✗	✗	✗
dc2aa30b	✓	✓	✓	✗
dc2e9a9d	✗	✗	✗	✗
dd2401ed	✓	✓	✓	✓
de493100	✓	✓	✓	✗
df8cc377	✗	✓	✓	✗
e0fb7511	✓	✓	✓	✓
e133d23d	✓	✓	✓	✗
e1baa8a4	✓	✓	✓	✓
e1d2900e	✓	✗	✓	✗
e2092e0c	✓	✗	✗	✗
e21a174a	✓	✓	✓	✓
e345f17b	✓	✓	✓	✓
e4075551	✓	✓	✓	✗
e41c6fd3	✗	✓	✓	✗
e57337a4	✓	✓	✓	✓
e5790162	✓	✓	✓	✗
e5c44e8f	✓	✗	✓	✗
e619ca6e	✓	✓	✗	✗
e633a9e5	✓	✓	✓	✓
e66aafb8	✓	✓	✓	✓
e681b708	✗	✓	✓	✗
e69241bd	✓	✓	✓	✓
e6de6e8f	✓	✗	✓	✗
e74e1818	✓	✓	✓	✗
e760a62e	✗	✓	✗	✗
e7639916	✓	✓	✓	✓
e78887d1	✓	✓	✓	✗
e7a25a18	✓	✓	✓	✗
e7b06bea	✓	✓	✓	✓
e7dd8335	✓	✓	✓	✓
e872b94a	✓	✓	✓	✗
e88171ec	✓	✓	✓	✗
e95e3d8e	✓	✓	✓	✓
e99362f0	✓	✓	✓	✓
e9ac8c9e	✓	✓	✓	✓
e9b4f6fc	✓	✓	✓	✗
e9bb6954	✓	✓	✓	✓
e9c9d9a1	✓	✓	✓	✗
ea959feb	✓	✓	✓	✓
ea9794b1	✓	✓	✗	✓
ecaa0ec1	✓	✓	✓	✗
ed74f2f2	✓	✓	✓	✓
ed98d772	✓	✓	✓	✗
ef26cbf6	✓	✓	✓	✓
f0afb749	✓	✓	✓	✗
f0df5ff0	✓	✓	✓	✓
f21745ec	✓	✗	✓	✗
f3b10344	✗	✗	✗	✗
f3cdc58f	✓	✓	✓	✓
f3e62deb	✓	✓	✓	✗
f4081712	✓	✓	✓	✗
f45f5ca7	✓	✓	✓	✓
f5aa3634	✓	✗	✗	✗
f5c89df1	✓	✓	✓	✗
f823c43c	✓	✓	✓	✓
f83cb3f6	✓	✓	✓	✓
f8be4b64	✓	✗	✓	✗
f9a67cb5	✓	✗	✗	✗
f9d67f8b	✓	✗	✗	✗
fafd9572	✓	✓	✓	✗
fb791726	✓	✓	✓	✗
fc754716	✓	✓	✓	✓
fd096ab6	✗	✗	✗	✗
fd4b2b02	✓	✓	✗	✗
fe9372f3	✓	✗	✗	✗
fea12743	✓	✗	✓	✗
ff72ca3e	✗	✓	✗	✗