An expected utility maximiser is a theoretical agent who considers
its actions, computes their consequences and then rates them according to a
utility function. Next, it performs the action which it thinks is likely to
produce the largest utility. Then it iterates this process.
For an example, consider a computer program that plays the game of go. Such a
program considers its possible moves, calculates their possible consequences,
and then performs the move that it thinks gives it the best chance of
Expected utility maximisation is common framework used in the context
of modelling intelligent agents and constructing synthetic intelligences.
A utility function can neatly encapsulate many concepts from economics - such
as risk aversion and temporal discounting and marginal
If the utility function is expressed as in a Turing-complete lanugage,
the framework represents a remarkably-general model of intelligent
agents - one which is capable of representing any pattern of behavioural
responses that can itself be represented computationally.
The utility function encodes all the agent's preferences - often including:
discounting refers to how an agent values utility now, compared to
utility later. Is ten dollars now better than twenty dollars tomorrow?
An agent's temporal discounting preferences specify such things.
refers to the reluctance of an agent to accept a bargain with an
uncertain payoff rather than another bargain with a more certain, but
Any computable agent may described using a utility function
Utility maximisation is a general framework which is powerful enough to
model the actions of any computable agent. The actions of any computable agent - including humans - can be expressed using a utility function. This was spelled out by Dewey in a 2011 paper titled: "Learning What to Value" - in his section about "O-Maximisers".
Some argue that humans have no utility function. However, this makes little sense: all computable agents have utility functions. The human utility function may not be easy to write down - but that doesn't mean that it doesn't exist.
Powerful expected utility maximisers can be expected to
systems. Self-improvement is one of the fundamental strategies
expected utility maximisers are likely to use to help
them attain their goals.
After a certain point, such systems tend to naturally come to share
various traits with living organisms - they will resist death,
maintain themselves, absorb resources, grow and/or reproduce,
eliminate the competition - and so on. These natural
tendencies are not necessarily benign.
Self-improving systems may wish to change their levels of temporal
discounting and risk aversion - depending on their
capabilities. One obvious way of doing that is to make these factors
depend on your percieved self-confidence.
The complex field of utility engineering deals with how to
construct utiliy functions which are useful, and don't have too many
undesirable side effects.
Pragmatic and ideal goals
Pragmatic utility functions
Self-improving systems will often make use of the concepts of
ideal goals and pragmatic goals.
An ideal goal represents what a system actually wants.
Pragmatic goals are a cut-down versions of this - which are
faster, easier or cheaper to calculate.
For example, the synthetic intelligence,
Deep Blue had a complex utility function with over 8,000
parts, which contained relative piece values, the worth of central
control vs castling, heuristics about pawn promotion - and so on.
However, its real aim was to increase IBM's stock price by
winning games of chess.
A self-improving system will normally only have one ideal
utility function - but may derive various pragmatic utility
functions from this - depending on the resource constraints it
It is not usually a good idea to encode strategies for dealing with resource
constraints into the ideal goals of a system - since resource
availability may change as time passes. Strategies such as outcome
pruning and temporal discounting normally belong in
pragmatic utility functions - and are best kept out of ideal utility
Instrumental and ultimate goals
Similarly, expected utility maximisers typically have one ultimate
goal, but may pursue various instrumental goals in service
Often long-term projects can be broken down into numerous short-term
ones. For example having a child can be decomposed into learning a
trade, getting a job, buying a house,
finding a mate - and so on. These short-term targets are known as
Instrumental and ultimate goals are sometimes referred to as primary
goals and secondary goals. The terms instrumental
values and terminal values express the same basic concept.
Instrumental goals can sometimes be used as
pragmatic goals - but these are really separate concepts.
Ultimate goals and ideal goals however are often
rather similar concepts.