On Artificial (General) Intelligence
Why AGI is not around the corner, and why current AIs are still amazing.
When it comes to AI, particularly the elusive idea of ‘human-level intelligence,’ one question consistently comes to mind. It originates from an interview Dwarkesh Patel conducted with Dario Amodei, CEO of Anthropic (makers of the famous Claude AI chatbot) roughly a year ago (link with timecode):
One question I had for you while we were talking about the intelligence stuff was, as a scientist yourself, what do you make of the fact that these things have basically the entire corpus of human knowledge memorized and they haven't been able to make a single new connection that has led to a discovery?
Whereas if even a moderately intelligent person had this much stuff memorized, they would notice - Oh, this thing causes this symptom. This other thing also causes this symptom.
There's a medical cure right here.
Shouldn't we be expecting that kind of stuff?
Dario is struggling to answer the question - he replies with a mix of “well, models can be creative when they write a poem in the style of Barbie…” followed by “Actually, I don’t have a good answer for that…” and concluding with “Scaling will solve the problem.”
Although I’m reading a lot about the advances of AI, I have not found a satisfying answer to this question to this day, but it’s genuinely surprising that the CEO of one of the leading companies in the space lacks a compelling explanation for this phenomenon. Even more surprising is, how confidently many people discuss ‘human-level intelligence’ or ‘AGI’ (Artificial General Intelligence) without considering this fundamental fact. If there’s widespread concern about AI’s potential to outmaneuver humans and make autonomous decisions, shouldn’t we prioritize understanding how AIs actually “think”?
What is human intelligence?
Before we can a closer look at the question, I’d like to define the semantics quickly. When we talk about ‘human-level intelligence,’ we have to distinguish between two kinds of intelligence, that Amazon or Google brush over in the coverage of their AI products:
School-level intelligence: This is tested using standardized methods with discrete answers (either multiple-choice or open-ended responses) that get graded. This is how current benchmarking for LLM AIs is done (Example from Meta on the latest Llama 3.3 release on X).
Core human intelligence: This involves creativity—the ability to find novel solutions to problems. It’s the type of intelligence that investigates problems (in the widest sense of the word), and finds new explanations and solutions. Like, noticing the slightly incorrect positions of planets (compared to their calculated ones), and devising a revolutionary new framework (i.e., Einstein’s discovery of the theory of relativity).
Why is this distinction important? Because school-level intelligence (the first type) is essentially recalling information, like a smart Wikipedia bot, while real human creativity is about identifying and resolving unprecedented challenges and problems.
Yet, in the AI space, we conflate these two types of intelligence and assume that throwing more computational power at the problem will magically lead to digital consciousness.
How AIs work: Simplified technical foundations
Quick note: While writing the following paragraph, I stumbled across this short, 4-minute explainer by 3Blue1Brown on YouTube, who—as always—does a better job than I ever could. Save yourself the next paragraph and just watch this video here:
In a very abstract and simplified way, Large Language Model (LLM) AIs work like this:
User provides input (pink).
AI model processes that input in a model (blue).
AI model predicts a response to the user (green).
Imagine you’re trying to predict the next word in a sentence, like completing the phrase, "The cat sat on the __." Based on the context (ie. the first part of the sentence), the AI predicts the word "mat" because it has seen millions of examples where "cat" and "mat" often appear together. It does this by activating specific connections (called weights, illustrated by the blue circles above) in its network, which represent the relationships between words it has learned from its training data. These weights help determine the most likely next word, just as our brains might recall patterns when filling in blanks.
The AI model ‘learns’ to predict the right responses by studying vast amounts of text data—imagine practically the whole internet and more. It does this through a process called ‘training,’ where it adjusts billions of these tiny settings/weights, in its neural network. These weights act like dials that determine how strongly the input (words, phrases, or even context) influences the model’s predictions. During training, the model processes example after example, checking how close its prediction is to the correct answer and tweaking the weights to improve accuracy.
This process is computationally intensive and expensive, requiring billions of dollars in energy and infrastructure. Instead of the 12 weights in our earlier example, real models use billions, making the task incredibly complex. To visualize how these networks work, think of an AI learning to recognize numbers from handwritten digits: each training step refines its ability to connect squiggles on a page to actual numerical values. The same principle applies here, but scaled to a massive linguistic level.
For a more complex visualization of AIs making predictions, see this example of neural networks predicting numbers from handwritten input.
This mechanism underpins every LLM. However, it’s a fallacy to believe that adding more compute or examples will make these models start “thinking” by themselves. That’s akin to believing that putting more horsepower in a car will turn it into a spaceship.
Why LLMs don’t resemble human intelligence
Considering the above explanation, we can deduce several ways in which LLMs fundamentally differ from actual human intelligence:
Necessity for human input
Current AI models require human input. And, while AIs can direct other AIs, humans must still initiate the first prompt. We can see from our analysis above why this is the case. Without any input, no prediction can be derived by performing the calculation across the different layers of weights.
This is fundamentally different from human intelligence. Nobody needed to ask Einstein to rethink Newton’s model of planetary positions; he noticed the discrepancies and pursued the problem autonomously, leading to the theory of relativity.
AIs can’t improve themselves
A common belief is that AIs will eventually improve themselves, often depicted as an exponential feedback loop leading to superintelligence. However, this notion clashes with the fundamental limitations of how current models operate. While it’s theoretically possible to design an AI capable of modifying its own architecture or training processes, the practical challenges are immense. First, there’s the problem of evaluation: how would an AI determine that a self-applied modification has genuinely improved its capabilities? Without an external metric of success or understanding of abstract goals, the system risks amplifying flaws rather than addressing them.
Critically, meaningful self-improvement requires creativity—the ability to generate novel ideas for improvement and systematicly testing them. For an AI to enhance itself, it would need to identify limitations in its design, hypothesize solutions, and experiment with changes to validate those solutions. This goes far beyond pattern matching; it demands the kind of inventive thinking that allows humans to leap beyond existing frameworks. Current AIs lack the intrinsic qualities necessary for such innovation: they do not explore concepts independently, question assumptions, or infer new possibilities from gaps in their knowledge. Without creativity as a driving force, the idea of AI self-improvement remains speculative and rooted more in fiction than fact.
AIs can’t logically reason
Perhaps the strongest argument for the inability for AIs to ‘think’ is this research paper from Apple. In a nutshell, Apple researchers tried a methodical approach that only changed individual words in an LLM prompt.
They did this by creating GSM-Symbolic, a dataset built by modifying existing grade-school math problems from a specific dataset (GSM8K). The researchers made templates and generated new questions by substituting various names and numerical values while keeping the core problem structure consistent. This allowed them to test how LLMs would perform when faced with slight variations of the same problem.
The experiments revealed that even small alterations, such as changing the numerical values, significantly impacted the LLMs' performance. This suggests that LLMs rely heavily on pattern matching - recognizing familiar patterns from their training data - rather than truly understanding the mathematical concepts involved. To further investigate this, the researchers created another dataset that included irrelevant information within the math problems (GSM-NoOp). For instance, a problem might mention the size of some fruit, a detail that doesn't affect the calculation required to solve the problem.
The findings showed a considerable drop in performance across all tested models when presented with GSM-NoOp, indicating that LLMs struggle to discern relevant information from irrelevant details. This supports the conclusion that LLMs are primarily engaging in pattern matching, blindly converting statements into operations without grasping the underlying meaning.
As the paper concludes, this reliance on pattern recognition rather than logical reasoning represents a significant limitation in the current state of AI development.
Lack of self awareness
Humans possess the unique ability to reflect on their thoughts and intentions, enabling a level of introspection that LLMs cannot achieve. For instance, when a person makes a mistake, they can consciously analyze why it happened, learn from it, and adjust their future behavior. An AI, on the other hand, lacks this meta-awareness; it doesn’t know when it’s wrong unless explicitly told so. For example, an AI answering a question inaccurately might confidently assert its response, unable to recognize that it has made an error or revisit its internal logic to self-correct. This absence of self-awareness means that LLMs operate in a kind of vacuum, oblivious to the implications of their outputs or the processes that generated them.
This absence of introspection has profound implications for reasoning. Humans have the capacity to evaluate their thought processes, question their own assumptions, and refine their reasoning over time. For example, a scientist working on a complex hypothesis might reconsider their methodology after discovering a contradiction or anomaly, thereby improving their approach. LLMs, however, cannot evaluate their reasoning or adjust their approach without external intervention. When asked a question, they generate a response based on statistical patterns, unable to assess whether their logic is sound or whether they’ve overlooked a key detail. This limitation becomes evident in tasks requiring careful judgment, where human introspection plays a crucial role in refining outcomes.
Understanding causality
Another fundamental gap is in understanding causality. Humans intuitively grasp cause-and-effect relationships, which is essential for solving complex problems and adapting to novel scenarios. For instance, a doctor diagnosing a patient doesn’t just recognize patterns in symptoms; they infer causal links, such as how a specific virus might trigger inflammation that then leads to a fever. This ability to connect multiple layers of cause and effect allows doctors to recommend treatments that target the root cause, not just the symptoms. LLMs, by contrast, operate purely by recognizing correlations in their training data, which makes them blind to these deeper connections.
While some might argue that providing more comprehensive data—such as lifestyle factors or environmental conditions—could bridge this gap, it doesn’t solve the fundamental issue: causality requires reasoning beyond patterns. Consider a case where multiple factors contribute to a disease, such as genetic predisposition, diet, and stress. A human doctor can hypothesize how these factors interact dynamically over time and test those hypotheses to refine their understanding. An LLM, however, lacks the capacity to hypothesize or test theories. It can only correlate data points that are explicitly present in its training set, making it unable to extrapolate unobserved causal mechanisms or prioritize interventions for outcomes it hasn’t encountered before.
This limitation becomes even clearer in situations where causality is subtle or indirect. For example, a person exposed to chronic stress may develop heart disease over decades. Identifying stress as the root cause involves understanding long-term physiological changes—something that requires not just recognizing patterns but constructing a conceptual model of how stress interacts with the cardiovascular system over time. LLMs can neither construct such conceptual models nor reason through potential interventions, leaving them fundamentally incapable of handling complex, dynamic systems.
This inability to infer and reason about causality restricts LLMs’ usefulness in dynamic, real-world problem-solving. Without a true grasp of how and why events unfold, they remain limited to superficial correlations, unable to provide the deep insights or adaptive solutions required in fields like medicine, engineering, or scientific discovery.
With all that laid out, let’s address the big elephant in the room:
Why does nobody talk about this?
I believe the short answer is conflict of interest. Attention is a scarce resource on the internet, and I suspect that investors are being sold on a vision where large parts of the economy become entirely automated—a future of limitless scalability.
This narrative is lucrative for those at the forefront of AI development, like Elon Musk, Sam Altman, and Ilya Sutskever. They are incentivized to maintain the perception that AGI is just over the horizon to secure funding and fuel the hype cycle. Meanwhile, thinkers like Yuval Noah Harari and Eliezer Yudkowsky amplify the conversation with doomsday scenarios, drawing significant public attention (and selling books).
The result? A lack of critical discussion about the feasibility of AGI with current AI architectures. Admitting that true AGI might be fundamentally unattainable under present paradigms could stall investment, dampen excitement, and shift the focus toward less sensational but more grounded advancements.
A more realistic vision of the future
If this article sounds very negative towards AI, I want to emphasize that I absolutely love AI technology, including LLMs, natural language- and speech computer interfaces, and image generation tools. I believe they provide a glimpse into a future where we can interact more naturally with computers and the vast repository of human knowledge.
Today, computers and phones are tools for searching and retrieving information, fostering creativity, and assisting with tasks. AI technologies promise to enhance this experience by making information more accessible, intuitive, and multimodal. Imagine a future where we can engage with computers not just through screens and keyboards but through natural conversation and sensory-rich interactions.
There’s an intriguing analogy: information technology has been inching closer to human senses over the decades. It started with desktop screens, then moved to mobile screens in our pockets, and now wearable devices like smartwatches. VR goggles bring screens mere centimeters from our eyes. AI takes this evolution further by allowing us to converse, listen, and even visualize ideas through dynamically generated images and spoken responses.
Computing is becoming truly immersive, enabling us to access and interact with information in ways previously unimaginable.
The caveat? We’ll still need to think for ourselves. There is no economic singularity or perpetuum mobile where machines produce something from nothing. At least not for now.