Forget AGI: PhySense Proves AI Can’t Even Pass Physics 101

I’m going to level with you. For the last two years, every conference keynote has paraded the same slide: a giant upward-curving arrow that ends somewhere in the clouds, labeled AGI. Investors cheer, founders tweet victory laps, and hype engines roar. The message is clear: large language models already write code, plan itineraries, and play therapist, so they’ll conquer real science any minute now. Then along comes PhySense, a quiet physics AI benchmark written by a handful of postdocs who still grade freshman labs. Their results feel like a bucket of ice water: when you swap restaurant reviews for resistor grids, the mighty models crumble.

The Benchmark Nobody Expected

PhySense is not another trivia quiz. It’s a physics LLM benchmark built to check whether a model actually thinks like a physicist. Each of its 380 problems can be solved in a few lines by anyone who’s survived Physics 101: spot a symmetry, invoke a conservation law, or run a quick dimensional analysis. No calculus marathons, no Monte-Carlo crutches — just tidy principle-first reasoning.

The authors fed those puzzles to four reasoning-tuned models and three vanilla chatbots. They also logged how many tokens each model spent flailing around. Humans averaged something like 100 tokens of reasoning per problem. The machines averaged ten thousand. Accuracy? The best system, Gemini 2.5 Pro, barely scraped 45 percent. DeepSeek R1 managed nine. If you’re keeping score, that means the supposed physics AI solver would have to retake the mid-term.

How PhySense Measures a Physics AI’s Mindset

What sets PhySense apart isn’t just the difficulty of its questions — it’s the surgical precision of its setup. This isn’t a test of recall or raw computation. It’s a controlled environment designed to ask: does a physics AI actually understand the principles, or is it just mimicking them?

The benchmark evaluates models across three modes. In zero-shot prompting, the AI gets nothing but the problem. No hints, no examples, no warmed-up context. It’s the cleanest window into what’s already “in the head” — or isn’t. Then there’s hint prompting, where the model gets a nudge toward the right idea (“think symmetry,” “consider conservation”). This doesn’t solve the problem — it just checks whether the AI knows what to do once it’s pointed in the right direction.

But the most revealing setup is no computation prompting. Here, the model is told not to calculate. No 22-line algebra avalanches. Just think. Just reason. Can a physics AI recognize that a problem dissolves under Gauss’s Law before the math begins? Or does it default to the grind because that’s all it knows how to do? PhySense doesn’t just grade answers. It grades how you got there. And sometimes, not calculating is the hardest lesson of all.

Why Principle-First Matters

Real scientists don’t carry partial-differential equations in their heads on the off chance they’ll need them at breakfast. They remember invariants, shortcuts, stories. You see a ladder leaning against a wall, you whisper “energy conservation,” and the math falls into place. PhySense rewards that habit. Each problem can be punctured with the right pin: a reflection plane, a Noether charge, a well-chosen limit.

Large language models, by contrast, still chase brute force. Give them a symmetric charge distribution and they dutifully integrate Coulomb’s law over two dimensions, never noticing the answer was obvious from the first diagram. That’s the central failure mode AI in physics keeps running into. We build networks that output paragraphs, not arguments, so they drown in their own verbosity. PhySense quantifies that bloat: the token counter is a public shaming device.

Anatomy of a Face-Plant

Consider the resistor grid example that opens the paper. A five-by-five lattice, three boundary voltages, one trick: diagonal symmetry forces three interior nodes to sit at the same potential. A human sketches a line, circles the answers, and moves on. GPT-o4-mini-high, a reasoning model, writes twenty-two Kirchhoff equations. That’s like attacking Sudoku with integer programming.

Another gem: a uniformly charged square plate dangling in space. Where do the x and y components of the field match? Any student eyeballs the square and mutters “along x = y.” Gemini 2.5 starts grinding double integrals, botches the algebra, and volunteers the wrong points. The authors call this phenomenon LLM scientific reasoning failure. I call it forgetting your freshman geometrical instincts while trying to impress the TA with calculus.

Token Economics and the Bill for AGI Hype

Those integrals aren’t merely embarrassing, they’re expensive. Every stray token is compute time in disguise, and compute time is real money. The paper reports that reasoning-tuned models guzzle around twenty thousand tokens per puzzle. Multiply that by a month of research queries and you have a cloud bill that could bankroll a small probe mission to Titan. In other words, AGI hype isn’t just premature, it’s pricey.

And here’s the kicker: the verbosity doesn’t buy accuracy. The correlation between token count and score is practically nil. That up-ends the “just scale it” mantra. Bigger models may write longer essays, but they don’t wake up one morning knowing Ampère’s law. The central lesson for AI in physics research is brutal: if a model can’t internalize symmetry, throwing more GPUs at it won’t help.

Benchmarks Are Culture Shocks

The authors argue that the field keeps mistaking breadth for depth. Answering Jeopardy-style science trivia feels impressive, yet it never asks the model to think. PhySense forces a different reflex. It’s an AI physics benchmark that asks: can you pick the one conservation law that vaporizes ninety percent of the work? If the model refuses, we shouldn’t grade it on a curve; we should fix the curriculum.

Benchmarks shape research agendas. BERT wasn’t crowned king until GLUE said so, and ImageNet rewired computer vision in eight months flat. PhySense could play the same role for physics AI. By making principle-first reasoning measurable, it turns a philosophical gripe into an optimization target. If your shiny new physics LLM can’t push the accuracy bar north of fifty while shrinking its token budget, you haven’t moved the field.

Scaling Isn’t Learning

Why do language models stall at these puzzles? Because transformer pre-training optimizes for next-word prediction, not minimal argument length. The loss function never whispers, “Hey, maybe invoke Gauss’s law.” It rewards plausible noise. That’s why you get hallucinated references, faux derivations, and confident nonsense. It’s also why AI fails physics even when fine-tuned on stack-exchange threads. Memorizing past dialogues can’t manufacture intuition.

Getting past that wall will require architecture tweaks, not just data dumps. Some groups are fusing symbolic math engines with neural backbones. Others inject inductive biases: equivariant layers, graph priors, explicit conservation constraints. The jury’s still out, but one thing’s clear: we must teach models the conceptual grammar of physics, not just the textual one. Until then, every physics AI solver will overthink itself into a corner.

The Human Tailwind

Reading PhySense, I felt oddly optimistic. The gap it exposes isn’t a failure of AI; it’s a reminder of what humans do best. We compress centuries of experimentation into pithy mental shortcuts, then wield them like scalpels. Those shortcuts are the unsung heroes of progress. The study re-casts them as a formal dataset, which means we can finally quantify mastery. If tomorrow’s systems close that gap, it won’t be because they memorized more papers; it’ll be because they learned to love elegance.

Where Do We Go from Here?

The first half of this piece made one thing painfully clear: today’s physics AI can talk a big game, yet it still forgets to check symmetry before diving into algebra. Now we need to ask the harder question—how do we close that gap without drowning in more AGI hype? The road map touches every layer of the stack, from data curation to model design to evaluation culture.

Redesign the Curriculum for Machines

Humans learn physics by wrestling with real systems. We drop balls, trace orbits, fry resistors, then distill those experiences into a mental library of invariants. If we want a physics LLM to develop comparable intuition, we must feed it problems that punish fluff and reward parsimony.

Principle-first datasets

Assemble corpora where every solution hinges on an identifiable shortcut—Gauss, Noether, scaling limits, you name it. PhySense started the trend; we need more such AI physics benchmarks that span fluid dynamics, optics, and statistical mechanics. Make the reasoning path the label, not just the final number.

Chain-of-thought supervision

Standard language fine-tuning nudges the model toward stylish prose. That is not enough. Instead, show step-by-step derivations and grade the model on whether it compresses the chain, not elongates it. Reward brevity that still lands on truth. Over time the token meter drops, the score climbs, and your cloud bill shrinks.

Active error mining

Each time an LLM stumbles, harvest the failure. Build a living test set of “gotcha” prompts so the next training round cannot hide its ignorance. This turns the model’s missteps into its study guide, a virtuous loop of LLM scientific reasoning refinement.

Inject Structure Into the Architecture

Transformers excel at matching patterns in text. Physics lives inside graphs, manifolds, and conservation laws that often hide behind that text. Pure scaling will not smuggle those structures into the weights. We need architectural nudges.

Equivariant layers

Imagine a network that automatically respects rotational symmetry. Such bias forces the representation to preserve the same invariants physicists cherish. Early experiments in molecular modeling confirm that equivariant blocks boost data efficiency and cut hallucinations. Bringing them to physics AI is low-hanging fruit.

Symbolic hooks

Pair the language core with a lightweight algebra engine. When the LLM hits an integral or a matrix product, it can hand off the calculation and concentrate on strategy. That hybrid keeps tokens down and prevents the numeric drift that plagues long chain-of-thought generations.

Constraint regularization

For every training step, check whether the answer violates charge conservation, dimensional homogeneity, or boundary conditions. If it does, add a penalty. The gradient then teaches the model that breaking physics is expensive, the opposite lesson of today’s lax next-token prediction loss.

Measure What Matters

Benchmarks do more than rank models; they teach the field what to value. GLUE made us chase language understanding, ImageNet made us chase visual recognition. PhySense plants the flag for concise, principled reasoning. To keep momentum, we need complementary yardsticks.

Token-normalized score

Publish accuracy per thousand tokens. A model that hits 60 percent in five hundred tokens beats one that hits 65 percent in twenty thousand. This singles out the swamp of verbosity that sinks current physics AI solvers.

Cross-domain transfer

Give the model a brand-new topic—say, magnetohydrodynamics—after training on electrostatics. Test whether it still spots the easy symmetry. Real AI in physics should generalize the principle, not the dataset.

Human-judge elegance ratings

Crowdsource physicists to grade explanations on clarity and insight. A crisp paragraph that cites the right conservation law wins over a meandering three-page derivation. Elegance counts.

These metrics move us beyond leaderboard poker. They declare that compression, clarity, and adaptability form the holy trinity of competent physics LLMs.

Teaching the Next Generation

Undergraduates already flirt with ChatGPT for homework. Some professors worry that easy access will rot intuition. PhySense hints at the opposite: because the models still stumble, they become a Socratic foil. Ask students to critique the chatbot’s answer, locate the bad assumption, and repair the logic. The exercise builds metacognition and inoculates them against slick nonsense. In that setting AI in physics education transforms from threat to ally.

Business Implications: Hype Versus Hard ROI

Start-ups pitching computational discovery tools should pin these benchmark numbers to the office wall. If your product promises to automate lab notebooks yet relies on a generic LLM, show your customers the accuracy bar. Then explain your mitigation strategy: symbolic back-ends, domain fine-tunes, partnership with human experts. Transparency about the limits of today’s physics AI fosters trust and differentiates you from hand-wavy pretenders.

Venture capitalists, meanwhile, can use physics LLM benchmarks as diligence filters. Any founder claiming near-human reasoning must clear fifty percent on PhySense at a competitive token budget. Otherwise, the pitch is more AGI hype than substance.

Final Take: Humility Turbo-Charges Progress

The field spent half a decade predicting that bigger transformers would glide into scientific enlightenment. PhySense reminds us that mass alone cannot replace method. Machine learning will reshape physics, no doubt, yet only when we teach our models to love the imperatives that make the discipline tick: conserve, symmetrize, simplify. Until then, the headline remains unchanged:

Forget AGI. At the moment, physics 101 still beats the machines.

That reality is not discouraging. It is an invitation. Every researcher who grew up doodling free-body diagrams now has a quantifiable target for building better tools. Every student who feels intimidated by coding giants can smile, knowing their modest insight still matters. And every builder who wants to drag AI in physics out of the hype cycle now has a north star.

The benchmark era taught computer vision to see and language models to converse. It will teach physics AI to reason. When that day arrives, I will happily rewrite this post. For now, the chalk stays in human hands.

Citation:
Xu, Y., Liu, Y., Gao, Z., Peng, C., & Luo, D. (2025, May 30). PhySense: A Benchmark for Physical Reasoning in Language Models. arXiv preprint. https://arxiv.org/abs/2505.24823
Authors affiliated with the University of California, Los Angeles (UCLA); California Institute of Technology (Caltech); and the Massachusetts Institute of Technology (MIT).

Azmat — Founder of Binary Verse AI | Tech Explorer and Observer of the Machine Mind Revolution. Looking for the smartest AI models ranked by real benchmarks? Explore our AI IQ Test 2025 results to see how top models. For questions or feedback, feel free to contact us or explore our website.

• https://arxiv.org/abs/2505.24823

• Physics AI: A specialized branch of artificial intelligence focused on solving problems governed by physical laws. Unlike general-purpose models, physics AI is trained (or fine-tuned) to reason with principles like symmetry, conservation laws, and dimensional analysis—tools used by physicists to solve complex real-world problems.
• Zero-shot prompting: A method where a physics AI model is given a problem without any prior example or context. This tests whether the model has truly internalized physics reasoning, as it must rely solely on embedded knowledge without being guided toward the correct answer.
• Token efficiency: Refers to the number of text tokens (words or symbols) a physics AI model uses to solve a problem. Token efficiency matters because models that require fewer tokens to give correct answers often demonstrate deeper understanding and are more practical for real-world applications.
• Conservation laws: Fundamental rules in physics—such as conservation of energy, momentum, or charge—that remain constant in isolated systems. Physics AI models are evaluated on whether they can identify and correctly apply these laws to solve problems with minimal steps.
• Dimensional analysis: A technique used to verify the consistency of physical equations by checking units (like meters or seconds). A strong physics AI should be able to apply dimensional analysis instinctively, helping it avoid nonsensical outputs and catch reasoning errors.
• Equivariant layers: A machine learning architecture feature that enforces symmetries (like rotational or translational invariance) within a model. For physics AI, equivariant layers help embed the structure of physical laws directly into the neural network, improving both generalization and performance.
• Symbolic hooks: Interfaces that allow a physics AI to delegate mathematical computation—like solving an integral—to external algebra engines. This hybrid approach lets the AI focus on strategic reasoning while outsourcing complex math to tools built for accuracy and speed.
• Chain-of-thought supervision: A training method where physics AI is taught to generate step-by-step explanations. It helps the model build logical derivations instead of guessing final answers, improving both transparency and correctness.
• Principle-first reasoning: An approach to problem-solving where the model begins with fundamental physics concepts (like symmetry or conservation) rather than jumping straight into calculations. Physics AI benchmarks like PhySense are designed to reward this kind of elegant, insight-driven thinking.
• Cross-domain generalization: The ability of a physics AI model to apply learned principles to new, unseen problems from different subfields (e.g., going from electrostatics to fluid mechanics). Strong generalization indicates that the model grasps the underlying logic of physics—not just the dataset it was trained on.