The Intelligence Curse in the Age of AGI

Economic research indicates that countries that rely on selling natural resources like oil for income rather than citizen taxes are affected by the resource curse, resulting in the elite seizing vast wealth while ceasing investment in the general populace. Will humanity face a similar “intelligence curse” after the advent of AGI? In a previous article titled “Capital, AGI and Human Ambition“, Rudolf pointed out that after AGI arrives, labor-replacing AI will change the relative importance of human and non-human production factors, which will decrease society’s attention to humans while making existing power structures more effective and entrenched. In this article, based on the resource curse phenomenon, Luke Drago describes Rudolf’s core argument as the intelligence curse and provides more specific and persuasive data and examples to support his argument, additionally exploring potential solutions. Some key points of this article include:

1. AGI is more like a resource than just a technology. AGI will not be limited by fields—it aims to achieve superhuman capabilities across all domains, replacing programmers, writers, analysts, and even CEOs.

2. Due to the gains of oligarchs (powerful actors) adopting labor-replacing AI systems coming from “always available intelligence,” they will face a motivation dilemma similar to that of rentier states, but with more severe consequences.3. After the arrival of AGI, the income sources of oligarchs will primarily include: revenue from corporate taxes (mainly from companies developing advanced AI systems); AI labs becoming new rentiers; transactions between businesses and between them and oligarchs.4. The intelligence curse will massively generate motivations similar to those of rentier states, with typical constraints absent, resulting in oligarchs being more inclined to invest in AI rather than humans.

5. A few elite in the post-AGI era will control all oligarchs, while others will struggle to meet basic needs.

(This article is translated by OneFlow, for reprint authorization please contact us. Original text: https://lukedrago.substack.com/p/the-intelligence-curse)

Author | Luke Drago

Translated by OneFlow

Translation | Zhang Xuedan

“Tell me the motivation, and I can tell you the outcome.” — Charlie Munger

Economists tend to view AI as an important auxiliary tool, thus they find it hard to understand how AI could render humans irrelevant. Past technological revolutions have further unleashed human potential; the agricultural revolution nurtured civilization, and the industrial revolution made civilization flourish.

However, compared to plows, steam engines, or computers, AGI is more like coal or oil, sharing the following similarities:

• Requires extremely wealthy participants to discover and utilize.

• Control will be concentrated in a few players, mainly the laboratories that develop them and the countries they are located in.

• Countries and companies that rely on these resources for income will no longer need to depend on their people to generate revenue.

• Will replace the previous power sources that drove civilization. For coal, the previous power source was wood; for AGI, the previous power source was humans.

On December 28, 2024,Rudolf published the article “Capital, AGI, and Human Ambition”, summarizing his main points as follows:

Labor-replacing AI will change the relative importance of human and non-human production factors, which will decrease society’s attention to humans while making existing power structures more effective and entrenched.

In this article, I will strive to name this phenomenon and build an evidence base for it. In later articles, I will also discuss potential solutions.

This issue is quite similar to the dilemmas faced by rentier states. Rentier states primarily rely on resource rents rather than citizen taxes to acquire wealth, thus falling into the resource curse—despite having natural income sources, their performance in improving the living standards of ordinary citizens often lags behind that of economically diversified countries.

Because the gains of oligarchs (powerful actors) adopting labor-replacing AI systems come from “always available intelligence” rather than human labor, they will face a motivation dilemma similar to that of rentier states, but with more severe consequences.Oligarchs cannot derive returns from investments we consider essential for survival, like education to cultivate employability, obtain job positions, and earn wages, or provide welfare systems for the unemployed. Therefore, they will not make these investments, and the people in their countries will also be unable to sustain themselves. Humans no longer need to participate in labor, hence there is no pay to be earned.

This is the “intelligence curse”—when oligarchs develop and implement general intelligence, they will lose the motivation to invest in humans.

1My Assumptions About AGI

I believe that specifically, AGI is a system that is “highly autonomous and surpasses humans in most economically valuable tasks,” technically feasible, and has a 90% probability of being achieved within the next 1 to 20 years (honestly, 10 years is too long). You should believe this too.[1]

In most economic activities, once superior, cheaper, faster, and more reliable AI systems are widely adopted, the intelligence curse will begin to take effect. We should expect that within 1 to 5 years after this curse takes effect, we will be trapped in this outcome.

2Why Oligarchs Will Care About You

The so-called oligarchs (powerful actors) refer to large organizations like states, companies, and bureaucracies that shape the world we live in and how we interact with it.

Rudolf provided the following explanation for why states care about their citizens:

Since the Industrial Revolution, the interests of states and citizens have been extraordinarily aligned. To be economically competitive, a powerful state needs efficient markets, a good education system to cultivate skilled workers, and a prosperous middle class to create demand.

No matter where talent comes from, the state benefits. The state also benefits from allowing a high degree of freedom to promote the prosperity of science, technology, and culture, which can enhance the state’s global soft power and cultural influence. Competition among states constantly drives progress in these areas—like the success of the United States and China promoting the development of efficient markets and educated wealthy citizens, while incentivizing free market space to promote China’s science and startups. In contrast, feudalism was characterized by a strategy of establishing an exploitative elite class to rule over an uneducated peasantry, using most of the wealth gained for foreign wars.

The concern of oligarchs for the people is not out of goodwill, but for two reasons:

1. You can bring investment returns, usually manifested in taxes or profits.

2. You will influence their ability to maintain power, which can be realized through democratic means (like voting) or through potential threats to the regime (referring to whether the acting party in a dynamic game should believe that the responding party will act in their favor or against them).

Most countries in today’s world are diversified economies, meaning their value comes from various industries and human activities, rather than relying on a single or a few sources. States acquire revenue by taxing individuals and businesses, and the best way to increase revenue is to enhance citizen productivity. While this can also be achieved by raising tax rates, taxation must be based on existing economic output, with a ceiling on total tax revenue. Therefore, states tend to cultivate engineers, entrepreneurs, innovators, and other economically productive labor forces, creating conditions for these investments to yield returns. To achieve this, they typically take the following measures:

• Create quality schools, research institutions, and universities

• Build infrastructure like roads and public transport

• Construct reliable governance systems and courts to protect property rights

• Ensure freedom of speech and information flow

• Support the establishment of small and micro enterprises

• Promote market competition

• Create social safety nets to provide risk support

These measures not only enhance citizen productivity but also expand the likelihood of innovation occurring. Equally important, they help people escape extreme poverty, improve living standards, and promote political and economic freedom. With quality schools, infrastructure, and competitive markets, citizens can receive training and find jobs that pay above basic subsistence needs. With reliable governance systems, fair judiciary, and freedom of speech, citizens can express their demands to the government without fearing being political prisoners. Citizens gain negotiating power through their votes and economic output, thus driving reforms to improve living standards. Therefore, sometimes, even at a cost, states will yield to the demands of citizens.

A similar phenomenon also affects businesses. Take Silicon Valley as an example, as of now, tech workers with skills urgently needed by companies can command high salaries. This type of employee is in high demand, and the competition among companies to attract them is fierce. To vie for these talents, companies offer high salaries, stock options, pool tables, free meals prepared by Michelin-starred chefs around the clock, and even laundry services. Although no one genuinely believes that the laundry service provided by the company can multiply income tenfold, it may attract potential employees to join or prevent dissatisfied employees from jumping ship to competitors. These employees have bargaining power and can demand the company provide generous benefits to improve their quality of life.

This forms a feedback loop—ordinary people earn more money for the oligarchs, who are then more likely to cater to the needs of the former. If education can increase the lifetime income of citizens (and the state) tenfold, then schools should be built. If paid parental leave can attract better employees, then policies should be changed.

3The Resource Curse

We have seen some countries separating their economic output from human capital, these countries are called rentier states, such as Venezuela, Saudi Arabia, Norway, and Oman. Their main source of income comes from resources (usually oil), rather than productive activities of citizens.

Theoretically, you might think that countries with natural resources should have very wealthy citizens. They just need to extract these resources and sell them to buyers in need; why bother developing a diversified economy? After all, wealth lies beneath their feet.

However, consider the example of the Democratic Republic of the Congo: the country has untapped mineral reserves worth over $24 trillion underground. But how do its people fare? According to a World Bank report:

Most people in the Democratic Republic of the Congo have not benefited from this wealth. Prolonged conflict, political turmoil, and instability, coupled with authoritarian rule, have led to severe and ongoing crises. Violence-induced displacement is also a contributing factor. Since the end of the Congo War in 2003, these conditions have hardly improved.

The Congo is also one of the poorest countries in the world. It is estimated that by 2024, about 73.5% of Congolese live on less than $2.15 a day. Among the extreme poor in sub-Saharan Africa, one in six lives in the Congo.

What has happened? Why have resources worth trillions of dollars not lifted the country out of extreme poverty?

Economists and political scientists refer to this phenomenon as the “resource curse.” Countries rich in natural resources often experience less economic growth than economically diversified countries and have higher poverty rates.

Many factors contribute to the “resource curse,” but I will focus on one core factor: natural resource revenues weaken the motivation of states to care about the economic welfare of their citizens.

Because their income sources are natural resources, rentier states lack the incentive to pay current ordinary citizens wages or invest in their future. For example, building more schools will not bring additional revenue to these countries. They will only invest enough resources to ensure that oil can be extracted, transported to ports, and sold[2]. It is not that citizens of these countries lack the ability to create taxable value, but that there is no necessity to develop them into a taxable group.

When you can make money directly from underground resources, why tax citizens?

In the absence of funding, ordinary people struggle to create demand. In authoritarian regimes, unless citizens pose a potential threat to those in power, rulers have no motivation to care for them. Those who control the flow of rents can seize wealth without concern for the lives of others.

So, how do the citizens of these countries fare? Dr. Ferdinand Ebil and Dr. Stephen Hertog (https://www.cambridge.org/core/journals/american-political-science-review/article/from-rents-to-welfare-why-are-some-oilrich-states-generous-to-their-people) provide two contrasting scenarios:

In comparative political theory, few topics display such starkly contrasting predictions as the relationship between resource rents and government welfare provision. On one hand, scholars adhering to traditional rentier state theory argue that resource-rich rulers may issue expansive welfare policies to appease the populace. On the other hand, scholars proposing political formal models of oil-rich states argue that rulers often neglect the welfare of their citizens. Resource rents are usually seized by a small elite class, who can self-enrich without relying on the domestic economic base, leading to minimal government welfare services and a worsening social poverty issue.

In practice, both developmental paths have cases to support them. For instance, Oman and Equatorial Guinea had similar per capita natural resource rent levels during the period from 1995 to 2014—slightly above $8000 per capita. Both countries have been led by the same authoritarian ruler since the 1970s, when both were in extreme poverty. However, their developmental trajectories are starkly different. Under Sultan Qaboos, Oman rapidly improved public services, reducing infant mortality from 159 deaths per 1000 live births in 1971 to 9 deaths per 1000 live births in 2010, far below the average of 32 deaths per 1000 live births in the Middle East, making it one of the fastest countries to reduce child mortality globally. In contrast, under President Teodoro Obiang, Equatorial Guinea has developed almost exclusively security service institutions, with little development in other areas, leaving most of its population in extreme poverty. Its infant mortality only decreased from 263 deaths per 1000 live births in 1971 to 109 deaths per 1000 live births in 2010, still above the average of 89 deaths per 1000 live births in sub-Saharan Africa. The revenues from resource rents are almost entirely monopolized by the president’s inner circle.

Sometimes, rentier states establish larger social safety nets[3], but in most cases, apart from a few oligarchs controlling the flow of resource rents, everyone else lives in extreme poverty[4]. Why is this? Ebil and Hertog provide the answer:

We agree with the insights provided by the political formal models of resource-rich countries, which reveal that the primary goal of ruling elites is to maintain their power. Public policy formulation is oriented around this core goal and reflects the elites’ assessment of the threats to their rule. Within this framework, elites will strive to maximize personal benefits from resource income.

At the same time, we also agree with existing literature’s conclusion that in resource-dependent regimes, the economic returns of providing welfare policies are relatively low, and politically, rulers do not expect the potential modernization effects (referring to the possible social, economic, and political changes) produced by these welfare policies. Based on this, we expect that regimes with abundant oil resources will typically establish narrow “kleptocratic coalitions,” providing only limited welfare policies, while the self-accumulation of wealth among elites is rampant.

However, this effect does not apply to widely disseminated technology, as the application of technology relies on ordinary people incorporating it into daily workflows to enhance productivity. So, will AGI be different?

4AGI is More Like a Resource Than Just a Technology

Imagine you are the CEO of a large company. Hiring employees is an investment behavior—you pay them salaries, which constitute a significant portion of the company’s total budget, in return for their work generating income for the company. Each year, you hire thousands of junior analysts responsible for basic tasks like data collection, report writing, or creating polished PowerPoint presentations. Additionally, you train them, and as other employees gradually retire, these new hires will take their places. Their work not only brings you current profits, but they will also grow into senior employees over the next twenty years, potentially even replacing you.

The purpose of hiring analysts is twofold: to build a workforce for current basic tasks; to cultivate reserve talents to replace gradually retiring existing employees.

In the 2010s, laptops became increasingly common. Compared to bulky desktop computers, laptops allowed analysts to work anytime and anywhere. They could take detailed notes during meetings and collaborate during group discussions. However, laptops cannot replace analysts’ jobs because you cannot directly command a laptop to perform specific tasks. Instead, analysts need to proactively use the laptop to gain its benefits.

Therefore, you equip all analysts with laptops, which makes nearly all their work more efficient, thus increasing the company’s profits. However, laptops are merely tools; they cannot: 1) allow one analyst to do the work of ten; 2) fully automate the tasks of analysts.

Fast forward to 2030, a company called BigLab releases an AI agent powered by GPT-8. It completes tasks 20% faster than human analysts and performs 10% better. Moreover, the annual cost of running such an agent is only $10,000—saving at least 80% compared to hiring an analyst. It can even allow your top analysts to do the work of ten, or completely “clone” these excellent analysts, automating the entire analyst team.

This AI agent not only performs better but is also more reliable. AI eliminates the bottlenecks of searching for talent, making the challenges of finding, accurately assessing, and hiring talent in all fields obsolete. Rudolf’s viewpoint is as follows:

If you want to convert money into results, the most important question may be recruiting the right talent. There are several issues in this process: 1. Unless you have considerable talent in the same field, it is often difficult to assess talent. Therefore, if you try to find talent, you often come up empty. 2. Talent is scarce (qualified talent is even scarcer—many institutions cannot rely on any other types of talent, refer to point 1), such talent is already rare. 3. Even if you find top talent, they are often not easily swayed by money.

AGI will not only outperform your analysts but will also be more reliable. Before or shortly after system integration, you will have a clear understanding of its performance. You can predict how much progress it will make in updates and improvements, and in the months or years after it surpasses analysts, it may even outpace you in making company strategic decisions.

Perhaps you are very fond of your existing analyst team and are skeptical about this new system. You decide to run a trial period. A year later, the performance of AGI has fully surpassed all analysts. Furthermore, human intervention actually slows down the system’s efficiency and may lead to “human-like” results (which could be less efficient or accurate than automated systems). At this point, will you continue to hire more analysts? Clearly not. The future size of the analyst team will be drastically reduced. Moreover, if the company faces difficulties, what comes to your mind is: firing most employees may yield better results.

In this case, why would you continue to keep analysts? They are more costly, perform worse than the system, and are unreliable. Perhaps some employee (like Mike) performed well in interviews and is quite likable, but companies often fire those employees whom management personally likes. If your company does not fire them, you will be beaten by competitors who do.

Is there anything that behaves similarly to AGI? That would be natural resources. We know the uses of oil very well, know how much oil is needed to perform tasks requiring energy, and know which type of oil is best suited for a specific purpose. When it’s time to refuel a car, we don’t need to investigate or compare ten gas stations, nor do we need to “gamble” on which gas station’s fuel will allow your car to run smoothly. We just stop, confirm the type of oil the car needs, and fill the tank.

AGI will play a role in the “intelligence” domain similar to that of oil in the energy sector. It can easily be integrated into workflows, completing tasks reliably, and its performance far exceeds any previous tools or personnel (including you). Whether in companies, bureaucracies, or governments, all participants will face competitive pressure to replace humans, making AI the successor. AGI will not be limited by fields—the goal is not only to achieve superhuman capabilities in a single domain but to cover all fields, replacing programmers, writers, analysts, and even CEOs.

This is not just a hypothesis. We are already beginning to see some systems nearing AGI reducing analyst teams, altering personnel strategies, and conducting layoffs. Remember, today’s systems are just starting. As time goes on, their capabilities will only continue to enhance. As they become stronger, their impact on the labor market will grow dramatically. As Aschenbrenner (a former employee of OpenAI’s Superalignment department) said, “You don’t need to believe in science fiction, just believe in the upward trajectory of intelligence on the chart.”

We are heading towards a default outcome driven by default motivations. So, what are these motivations? What kind of world will they create?

5The Definition of the Intelligence Curse

The “intelligence curse” describes the dynamics of the economy in the post-AGI era, which will prompt oligarchs to be more inclined to invest in AI rather than humans.If AI can complete your work at a lower cost and higher efficiency, then there is no reason to hire you. More importantly, there is no economic reason to invest in your lifetime productivity, care for your living needs, or keep you involved in economic activities. We may create unprecedented value through full automation, but if the distribution of the results is as unfair as the most rentier states (i.e., monopolized by a few), these results will not lead to a better life for the masses.

A common counterargument is that we will insist on requiring humans to perform certain tasks that can never be automated. This viewpoint is often applied to the teaching profession. I have heard many parents express their preference for real, empathetic teachers to care for and educate their children. However, this perspective completely overlooks the larger picture: the issue is not whether we need teachers, but whether there will still be sufficient motivation to fund schools. The same logic can be infinitely applied to anything that requires investment in ordinary people’s productivity, any luxury that relies on disposable income, and any basic services necessary for human survival.[5] By default, oligarchs will not build things that can employ humans or provide resources for humans; they have no need to do so.

There will still be tax revenues in government income sources, but primarily from corporate taxes. Similarly, the income of companies will come from their AI systems rather than the output of human labor.The high-quality living investments common in developed countries, including wages, education, infrastructure, stable governance, etc., will no longer yield returns, and humans will no longer create value for oligarchs.

So, where will oligarchs obtain their income?

States will obtain revenue through corporate taxes. Companies developing advanced AI systems and those using these systems will generate massive revenues, and as they scale, the taxes the state collects from them will increase. In 2022, corporate taxes accounted for an average of 11.5% of fiscal revenue for OECD member countries (https://taxfoundation.org/data/all/federal/us-tax-revenue-by-tax-type-2024/), while in the United States, this figure was only 6.5%. In the future, the structure of state fiscal revenue may increasingly rely on AI-related enterprises, resembling resource-dependent countries like Norway, Saudi Arabia, or the Democratic Republic of the Congo, contrasting sharply with the current average income level of OECD member countries. Once you notice that the state’s tax structure is leaning towards these resource countries, you will realize that the “intelligence curse” has manifested.

AI labs becoming new rentiers. The primary goal of AI labs is to develop AGI. Some labs are even adjusting their corporate structures to ensure they can maximize the commercial value of this goal. Once they develop systems that can “do it all,” will they share them for free? The answer is clearly no. They will become a horizontal foundational layer in the economy, extracting rents from all economic activities by selling AI systems to oligarchs. Initially, there may be some outsourcing services profiting by optimizing AI systems in specific verticals (this phenomenon already exists), but don’t expect this situation to last long, as the goal of AGI is to be “all-powerful.” This model will allow AI labs to occupy a significant portion of global GDP and wield economic power previously only held by states.

Transactions between businesses and between them and oligarchs. Resources existing in the real world, such as land, energy, computing resources, manufacturing centers, data centers, etc., which support the realization of goals, will become more valuable. Coffee chains or marketing companies may become irrelevant, but real estate companies and energy enterprises will be wealthier than ever. Oligarchs controlling these resources (at least in the short term, possibly under human control) will derive most of the value from these sources.

The difference between the intelligence curse and the resource curse lies in the long-term motivation for diversification. The climate issues brought by oil and the rise of renewable energy have forced oil-rich countries to seek new sources of income while also empowering their citizens. However, this influence cannot be replicated with AGI. Each new generation of models will be more powerful than the last and is likely to continue to be controlled by a few oligarchs. AGI will not “run out” like oil. Computing resources or energy may face bottlenecks, but the cost of computing continues to decline, and energy is gradually transitioning to green. We do not need to transition from AGI to something else—once it is developed, it will exist indefinitely.

Assuming oligarchs act along the default path, what fate will most ordinary people face? Tell me the motivation, and I can tell you the outcome:

• Companies will lay off employees and will no longer hire new ones. Ordinary people will no longer produce anything that companies consider valuable. In the short term, companies may rely on them as consumers of their products, but as most companies targeting ordinary people lose their economic demand base, these companies will gradually decline.

• States will drastically cut public funding. Remember, the revenue base of states will shift towards other oligarchs. They will not derive any value from the labor of ordinary people, so they will be unwilling to invest in things that can make ordinary people efficient producers. Returns on capital, power, and risk resilience will come from ensuring that AI labs can develop better models and that companies using these models can execute tasks in the real world. Additionally, taxes used to support human investment will largely come from AGI labs. Competition between states means that if any state attempts to implement a universal basic income (UBI) policy through such taxation, its AGI development may lag behind that of other countries.

• Ordinary people will lack resources to support themselves or each other. The vast majority will not have enough economic strength to make any demands. They will be unable to motivate the entities controlling resources to invest in ordinary people. This means that (in the best case) they will struggle to meet basic needs or will have to rely on the oligarchs’ charitable assistance out of benevolence.

For some time, ordinary people may still generate some value. Rentier states need some human labor to carry out resource transfers in the real world, such as extracting oil from underground. There may be a scenario where humans are compensated for physical labor while intelligent agents are confined to the virtual realm. However, as robotics technology advances[6], the demand for human physical labor will decrease. Humans will not be better, faster, cheaper, or more reliable than artificial labor-replacing AIs, thus will be unable to participate in economic activities.

For rentier and colonial states[7], value primarily comes from raw materials or physical goods, which are typically sold to foreign buyers. Only a few humans participate in the production or management of these resources, and most do not benefit. For AGI, the situation may be similar. This raises an urgent question: who is everything produced by oligarchs for?

Oligarchs have specific goals, thus production activities will revolve around achieving these goals. States wish to control territory, companies aim to generate more wealth for owners, and individuals with amassed capital may have their own ambitions. Perhaps they want to use their newly acquired power to colonize Mars or develop ocean resources. Their goals may also be less grand—many extremely wealthy individuals are content to maximize their personal enjoyment, ensuring their new social status is secure, which may require considerable power and resources. And when ordinary people are no longer in the value cycle, oligarchs have no motivation to share benefits with them.

Even those at the top of the social pyramid remain important, and emerging forces find it difficult to gain entry into this system. One reason a faction holds power is either because it possessed power before the intelligence curse began or because it was in a favorable position when the intelligence curse began.

This sounds very much like a feudal economy. Rudolf made a clever comparison:

In the worst-case scenario, AI billionaires possess almost limitless power, establishing a permanent aristocracy based on labor-replacing AI capital. The power gap between different classes may shock modern individuals, just as modern people find the status hierarchy of feudal society repugnant, but don’t worry—just as the lower classes of feudal society largely accepted their class without superhuman persuasive AI, the future lower classes will do the same.

In short, the intelligence curse will massively generate motivations similar to those of rentier states, with typical constraints absent. When humans become unimportant, oligarchs will default to not investing in humans. Without intervention, the default outcome will resemble the most unfair rentier states—only a few extremely wealthy players, while the rest exist in large-scale poverty, maintaining a stable equilibrium. A few post-AGI elites will control all oligarchs, while others will struggle to meet basic needs.

So, is anyone working to solve this problem?

6The World Awaits Your Action

Most people are not taking this issue seriously. When I discussed with several friends and top experts in the world about how to better govern AI before 2030 (https://ai2030.encodejustice.org/), our economic sector advised the government, “In the face of needing to take more drastic measures, we should consider some bold, innovative economic policy ideas.” To put these policy terms plainly, it means: “We don’t know what to do; we need some smart people to figure it out.”

We must break the collective denial culture driven by infinite optimism[8]. Unrealistic fantasies dominate the current public opinion, with some people harboring a self-righteous mindset: they believe their work is unique and can never be automated away, so they feel they do not need to care about these issues.

Here are two impressive dialogues I had:

More than a year ago, I spoke with a senior AI policy person about the possibility that automation could make people’s lives worse. He could not imagine that technology could replace jobs. Why?

We will create new jobs—perhaps everyone will work in AI policy!

I thought he was joking, but after a deeper discussion, I found he was serious. Everyone thinks their job is safe—even those in AI policy.

In a recent conversation, I mentioned the concept of the intelligence curse (at that time, I had not fully articulated this concept). The other party, a very influential figure in the AI field, agreed that the most likely outcome of AGI is technological unemployment, but he believed we would automatically move toward a utopian future. He said:

We will no longer need to work—we will be free to self-actualize, pursuing meaningful goals like writing poetry.

Simply believing that “others will figure it out” will not lead you to a utopian poetic life. You are not praying to God; you are praying to humans who are even more ignorant than you.

Those in the AI safety field believe they are immune to this; they have identified an extremely important issue—”intent alignment” (ensuring that the intentions behind AI behavior align with human true goals)—and are working hard to solve it. I completely agree that intent alignment must be resolved, and there is no other choice, but discussions in the AI safety field are often as optimistic as those predicting “poetry parties.” Aligning AGI and superintelligence does not equal utopia[9]. What you are doing is ensuring that the most powerful technology in human history is controlled by forces most susceptible to the intelligence curse. We cannot just plan how to develop AGI; we must also plan what to do the day after AGI emerges.

For those who can recognize the essence of the intelligence curse, collective denial is drowned in endless pessimism.

On the day after o3 was released, I received a text from a software engineer who refused to use the Cursor tool, expressing his disbelief that this technology could be stronger than him:

What do you think of o3? This is the first time I feel like I might be “done for.”

Endless pessimism makes us feel “done for” and hopeless. Discussions about “what is your p-doom (probability of doom)?” are more common than “what is your solution?”

If you feel powerless and despairing about the technological progress of the past year, then cheer up. One of the few who can see what will happen in the future, the world is waiting for you to take action. Hope is the cornerstone of moving toward the future.

In the next article, I will propose some methods I believe can partially break the intelligence curse, drawing partly from the experiences of countries that successfully avoided the resource curse.

I am working on the details, but I believe the solutions can be categorized into two types:

1. Governance Solutions. In healthy democratic countries, the ballot may overcome the intelligence curse. People can change the status quo through voting, but currently, our governments are not prepared for this.

2. Innovation Solutions. Technology can enhance human agency, promote human ownership of AI systems or clusters of intelligent agents, or otherwise keep humans relevant in the economic system, motivating oligarchs to continue investing in human capital.

This is not just a blog; governments around the world should start predicting the capabilities of AI now and think about solutions to the intelligence curse. Think tanks need to start formulating policies to help us prepare for a “post-employment” world. AI labs need to critically examine their motivations and establish more robust internal governance structures to overcome these issues. Aspiring young people should actively start businesses, designing technologies that allow humans to remain relevant in the economy and benefit the masses, while venture capitalists should start funding these projects. If you are in a favorable position to contribute to solving this problem, what are you waiting for?

Some issues are unsolvable, but all significant topics are worth our utmost efforts. I am more optimistic than ever; once we clarify the problem, we can find directions to solve it.

Change the motivation, and you can change the outcome. Everything starts now.

Thanks to Rudolf Laine, Josh Priest, Lysander Mawby, Jacob Pfau, Luca Gandrud, Bilal Chughtai, Nicholas Osaka, Stefan Arama, Joe Pollard, and Caleb Peppiatt for helping review this article.

Notes

[1] If you disagree with my point, I strongly recommend reading these articles:

https://situational-awareness.ai/from-gpt-4-to-agi/#Counting_the_OOMs,

https://medium.com/@richardcngo/visualizing-the-deep-learning-revolution-722098eb9c5,

https://blog.aiimpacts.org/p/2023-ai-survey-of-2778-six-things,

https://epoch.ai/trends.

And watch this video:

https://www.youtube.com/watch?v=SKBG1sqdyIU.

You should also know that the public goals of OpenAI, Meta, and Google DeepMind are all to develop AGI, and Anthropic is also working in that direction. You should also know that the primary recommendation of the 2024 U.S.-China Economic and Security Review Commission is to “establish and fund a project similar to the Manhattan Project dedicated to competing for open access to AGI capabilities.”

[2] For more, see Chapter 7 of this book: https://archive.org/details/africasince1940p0000coop/page/172/mode/1up.

[3] The reasons why a few rentier states like Oman and Norway develop into welfare states (and the implications for the intelligence curse) will be the subject of future articles. Spoiler: Oman’s model will not be a solution to the intelligence curse, but Norway’s model may be.

[4] For more evidence, see:

https://armgpublishing.com/wp-content/uploads/2021/01/8.pdf,

https://digitalcommons.iwu.edu/cgi/viewcontent.cgi?article=1512&context=uer,

https://pubs.aeaweb.org/doi/pdfplus/10.1257/jep.30.1.161.

[5] If the next question that comes to your mind is “then what is the relative advantage?”, note that this part originally had a 1500-word article refuting this viewpoint, but it was deleted due to length constraints. That article will be published soon.

[6] This content was written nine months before this article, and the running models were far inferior to today’s advanced models. Again, it is emphasized that linear progression is believable.

[7] I will write an article exploring how colonial countries share many similarities with resource monopoly countries. In both types of countries, exploitative systems create wealth for power centers, which have no motivation to care for the people within their borders. Post-colonial countries continue to suffer because these systems are no longer tools for value extraction from foreign powers, but have turned into tools for value extraction by domestic political elites.

[8] Infinite/absolute optimism/pessimism was first defined by Peter Thiel in his book “Zero to One” (https://www.goodreads.com/book/show/18050143-zero-to-one). For a summary of this concept, see: https://www.lesswrong.com/posts/7LZHS4afrXCNuuGK9/book-summary-zero-to-one.

[9] This assumption hinges on whether we either a) solve the intent alignment issue before ensuring the system aligns with human values, or b) completely abandon aligning the system with human values, with oligarchs preferring to have machines fully obey their intentions rather than opposing them based on some moral standards that conflict with their interests or beliefs.

Others are watching

• 900-page free “Generative AI and Large Models” e-book

• Top 10 AI Outlooks for 2025

• Capital, AGI, and Human Ambition

• Generative AI Reasoning Technologies, Markets, and Futures

• Brad Smith, President of Microsoft: The Golden Opportunity for AI in America

• 2024 AI Review: Investment Boom, Infrastructure Reconstruction

Let Super Product Developers Achieve “Token Freedom”

Invite friends to use SiliconCloud,giving away 20 million Tokens/personExperience now Accelerated Version DeepSeek-V2.5cloud.siliconflow.cnScan to join the user group