Existential risk reduction
Posted: Wed Feb 18, 2009 1:05 am
We begin with an agent. Utility maximization is her ideal, goal development and implementation as the furthering of that ideal. She discounts expected future mortality from her current utility level.
As framed, the agent finds it in her interest to reduce existential risk (ER) exposure. Implicitly, her goal is an indefinite lifespan as confirmed by her best estimate of near-term future mortality at any one time (a limited metric). For continuous ER reduction over an infinite time horizon, a geometric series of conditioned probabilities exists which sum to 0< p <1 (boundary exclusive) for an agent at some time t. She cannot forecast over an infinite time horizon, but she is privy to near-term expected ER rate reductions that she manipulates to produce a limiting p-value < 0.5 of death subject to her continual maintenance across time.
In our previous thread, we discussed goals as arbitrarily derived, which may not be a precise specification for this universe. Agents not as the one above, who do not optimize towards indefinite life or do so in some persistently sub-optimal manner, will cease to perpetuate themselves in the long-run. Reproduction as means of agent-class perpetuation without cognitive inheritance will render each generation of new cognitions insufficiently equipped with respect to their ER-reducing counterparts. The reproductive class may too extinguish either of their own right, or through game-theoretic and/or information asymmetric competition with more developed ER-reducing agents. Therefore, it's conjectured here that goals will arise to further the ideal of utility maximization incorporating an existential risk component. Within the (rather large and poorly defined) space of compatible goals to that end, we may see some arbitrariness and randomization among agents as to the goals they individually pursue.
edit: typo
As framed, the agent finds it in her interest to reduce existential risk (ER) exposure. Implicitly, her goal is an indefinite lifespan as confirmed by her best estimate of near-term future mortality at any one time (a limited metric). For continuous ER reduction over an infinite time horizon, a geometric series of conditioned probabilities exists which sum to 0< p <1 (boundary exclusive) for an agent at some time t. She cannot forecast over an infinite time horizon, but she is privy to near-term expected ER rate reductions that she manipulates to produce a limiting p-value < 0.5 of death subject to her continual maintenance across time.
In our previous thread, we discussed goals as arbitrarily derived, which may not be a precise specification for this universe. Agents not as the one above, who do not optimize towards indefinite life or do so in some persistently sub-optimal manner, will cease to perpetuate themselves in the long-run. Reproduction as means of agent-class perpetuation without cognitive inheritance will render each generation of new cognitions insufficiently equipped with respect to their ER-reducing counterparts. The reproductive class may too extinguish either of their own right, or through game-theoretic and/or information asymmetric competition with more developed ER-reducing agents. Therefore, it's conjectured here that goals will arise to further the ideal of utility maximization incorporating an existential risk component. Within the (rather large and poorly defined) space of compatible goals to that end, we may see some arbitrariness and randomization among agents as to the goals they individually pursue.
edit: typo