What Cheap Intelligence Won't Fix

If intelligence is becoming cheaper, why aren’t companies, societies, and governments producing proportionally better outcomes?

Maybe because there is an interesting problem underneath all of this.

The scenarios produced by our minds about the short term, medium term, and long term can be completely different. And the hypotheses produced by our minds can make us optimize for something that isn’t actually true.

The future is not 100% measurable. It is impossible to know exactly what will happen in a few years.

But I think there is an even harder problem: producing a good measurement of reality when your mental model is not well constructed enough to update itself when evidence and data appear in front of you.

And this made me realize something about AI.

AI can actually produce value for society.

You can maintain many competing hypotheses at the same time. You can use AI to analyze them, criticize them, search for evidence, and design experiments around them in order to advance toward something meaningful.

But there is a huge difference between having something capable of producing value and actually producing that value in the real world.

People have difficulty adapting.

People have difficulty trusting each other.

People have different incentives.

People have difficulty coordinating and executing together.

And sometimes what looks irrational from the perspective of the whole system may actually make sense from the perspective of one person inside that system.

So I want to create an image with you.

Just imagine your perfect future.

What do you see?

Maybe there is equality, and everyone has abundance.

Maybe there is agency, and everyone is capable of doing what they want to do.

Maybe there is health, everyone has access to good healthcare, and people live long lives.

Maybe there is peace. There is no war and less conflict.

And/or:

Maybe there is something particularly good about your life.

Maybe your family is completely happy.

Maybe you have the power you want to influence and impact the world.

The reason I’m using “and” and “or” here is that there are global outcomes and local outcomes.

Some people optimize for both.

Some people optimize mostly for one of them.

And those objectives don’t necessarily point in the same direction.

So what is the actual cost of coordination?

What would it cost to make everyone aligned?

And is making everyone aligned even possible when different people want different futures?

This is where I think the problem becomes much more interesting.

Imagine that AI helps us reason better.

It helps us generate hypotheses faster.

It helps us analyze enormous amounts of information.

It helps us write software, automate processes, model scenarios, and discover inefficiencies.

You might expect:

More intelligence → better decisions → better outcomes.

But reality contains another layer.

A company can know that a process is inefficient while the person responsible for that process has an incentive to preserve it.

A government can have evidence supporting a policy while different groups disagree about the costs they are willing to accept.

An employee can understand that automation would improve the company while also understanding that automation might threaten their position.

Two intelligent people can observe exactly the same evidence and still prefer different outcomes.

And even when people want the same outcome, they may disagree about which actions will actually produce it.

So intelligence alone does not solve the problem.

There is reality.

There are our models of reality.

There are our predictions about the future.

There are our individual incentives.

There are our values.

And then there is the problem of coordinating all of these things between millions or billions of people.

So what is the cost of actual coordination?

What is the cost of making intelligent agents aligned enough to produce a desirable outcome together?

Because AI alignment is not only an interesting problem with computers.

There is an older version of the alignment problem.

Humanity has been trying to align humans for a very long time.

Companies try to align employees.

Governments try to align citizens and institutions.

Markets try to coordinate people through prices and incentives.

Families try to coordinate people through relationships, obligations, and trust.

Societies build laws, norms, contracts, currencies, organizations, and institutions partly because coordination between intelligent humans is difficult.

And AI may make something uncomfortable increasingly visible:

Many problems we thought were caused primarily by insufficient intelligence may actually be caused by misaligned incentives, different objectives, bad models of reality, and failures of coordination between intelligent humans.

Maybe intelligence was only one of the bottlenecks.

And as that bottleneck becomes cheaper, we are going to see the others much more clearly.