Doesn't matter, people built harnesses that solves arc agi 3, so all you need is to train your model to work like that harness by default. That makes a model specialized at solving arc agi 3 without making it smarter in general.
It is very hard to make a benchmark you can't do that for, but it is very easy to make your own personal test that others can't do that for since now it isn't a benchmark they can target.
They didn't. Kaggle is still running for a few more months, best result atm is ~2% with 9h runtime on one rtx6kPRO. Also note that these new results are on the semi-private set, not the public 25 games ones. Any announcement where you see "solved ARC3" is likely only dealing with the 25 public games. And that's highly questionable, until you get to see the code. (which, to my knowledge the team that claimed 99% hasn't yet published).
Yes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.
> # FRAMEWORK ARTEFACT: the run's very first transition is replayed WITHOUT advancing state
# (tools.py:954 and agent.py:468 both `continue` before `state = next_state`). So on the
# level that contains that step (level 0) our counters start exactly one action behind.
# That skipped step was action 1 with BOTH avatars moving, so seeding n=1, bumps=0 reproduces
# the framework's lagged state exactly.
# CAVEAT: this seed is only right while level 0 has never been RESET. If you ever RESET
# level 0, change the seed to n=0 (after a reset the rollout re-inits and no longer skips).
That tells me that there is some leakage between runs. The idea of ARC3 is that agents start working blind, on new tasks, via API. A RESET is counted as one action. Without seeing the actual code that produced these traces we have no way of knowing how many iterations it took, if the "framework" played the same level multiple times (comment hint above makes it likely) and so on. That's why I said that before we actually see the code / can replicate / ARC team confirms it on new envs, this should be taken with a grain of salt.
The comment more likely means the harness source was read, not memory from a previous run and the first few turns of bp35 appear to be a cold start.
Sure none of this is certain without the source.
I do believe the authors that this schema significantly improves over the base, particularly given that it took 22x simulated turns over 14 hours, which is moving the trial and error to context rather than to game. I also don’t doubt there is some contamination.
Regardless, the approach is sound and I do believe it would significantly improve scores, even if that was +20-30 over baseline (49% in this case) it does imply the benchmark is measuring the harness more than the model.
You really should play the 25 games before stating that it's "simple". The benchmark doesn't just track "completion", it also tracks the number of steps, and the score is based on the median steps took by human players. So in order to get 99% it would mean that the model solved every level of every game in less steps than the median humans. Which, having played the games and having setup harnesses for local models, I find hard to believe.
Also the models have to figure out what "end" means. And each game involves some kind of "gotchas" thrown in the harder levels. Some games are only solved by about 2/10 people trying them.
The 99% result most likely has some leakage somewhere, either in the preparation of the environments, or from session to session.
