Max Mednik
  • Home
  • About
  • Interests
    • Angel investing
    • Magic
    • Scuba Diving
  • Blog
  • Contact

Readings and musings

Notes on Superintelligence by Nick Bostrom

6/28/2018

0 Comments

 
Picture
A good friend of mine who's an engineer and entrepreneur and really into AI recommended that I check out Superintelligence: Paths, Dangers, Strategies by Nick Bostrom. I just finished reading it and thought it was pretty eye-opening and scary.

I enjoyed the historical overview of the field of AI, and the many examples of current programs and how they rank against humans in various domains.

My biggest takeaways:
-McCarthy’s dictum: When something works, it’s no longer called AI.
-Quote from Knuth: computers can now do most of the things that require thinking but very few of the things that we or animals can do without thinking.
-If/when general AI is solved, the transition to superintelligence (above-human level intelligence) will happen too fast to respond to it at the time, so we should think and plan ahead.
-It's really hard to design objective functions/values for AI. Most strategies that on first order seem ok are really bad when considering second and third order effects.
-The most likely scenario is that we will get something wrong and basically be screwed. This is quite scary.
-Approaches like whole brain emulation seem interesting but really difficult to pull off in practice.
-The indirect value loading approach ("the AI should try to maximize whatever most of us would want it to maximize had we thought about it long and hard") seems interesting and compelling (and was new to me).

I'm personally skeptical we will ever achieve general AI. I think we'll just get better and better at domain-specific applications, but I don't think we'll ever figure out how to artificially make a machine think in the way we do (or for that matter understand how we truly think). I think it's just one of those mysteries that will never be fully solved.

I kind of lost steam about 2/3 of the way through the book when the level of detailed analysis of very futuristic scenarios seemed kind of overboard to me. I thought that it was hard to really effectively reason about situations which will in all likelihood be very different from anything we can imagine right now. It's good to be cautious and try to plan ahead, but I just thought it got too much into the weeds and too fine-grained for the immense uncertainty in question.

My full notes on the book are below.

​Fable of owls and sparrows and control problem
Finding an AI vs first learning how to tame it (but how to do so without an AI first)

Preface

1 past developments and current capabilities
Growth modes and big history
Step changes in growth modes
First intelligent machine is that last invention man will ever make
Intelligence explosion
McCarthy’s dictum: when something works, it’s no longer called AI
Knuth: computers can now do most of the things that require thinking but very few of the things that animals can do without thinking
Preinstalled automatic safety technology to stop algos when they go wrong

2 Paths to superintelligence
Better than human level machine intelligence (HLMI)
Recursive self-improvement
Whole brain emulation by scanning brain into computer and simulating every electrical impulse and thought
Genetic selection and breeding to achieve superintelligence
Gene spell checking, proofreading to remove mutations 
Genetic synthesis of ideal sequences 
Brain computer interfaces
Networking brains and organizations

3 forms of superintelligence
Speed superintelligence
Just like human but faster
Collective superintelligence
Collective of minds
Quality superintelligence
Same or better speed but much better quality

4 kinetics of explosion of superintelligence
Slow takeoff over decades
Fast takeoff over days
Moderate takeoff over weeks
Recalcitrance to change
Content recalcitrance
Optimization power divided by recalcitrance

5 decisive strategic advantage
Will one ai dominate or multiple

6 cognitive superpowers 
Singleton
Methods to take over
Send von Neumann probes to other planets to inhabit other planets by AIs
Wise Singleton 

7 superintelligent will
Intelligence and motivation orthogonal
Instrumental convergence because some objectives useful for almost any goal
Self preservation
Goal content integrity
Cognitive enhancement
Resource accumulation

8 is the default outcome doom
Treacherous turn
Sandbox test won’t work because will fake it
Perverse instantiation 
Infrastructure profusion

9 control problem
Two agency problems
Capability control methods
Boxing
Physical
Informational
Threat of simulator 
Stunting
Force running on worse hardware or less data
Trip wires
Honeypots to test evil intentions 
Incentive methods
Social integration
Motivation selection methods
Domesticity
Oracle
Indirect normativity

10 oracles, genies, sovereigns, tools

11 multipolar scenarios
Multiple competing superintelligent agents
Can help handle control problem

12 acquiring values
Value loading problem really hard
Could set up system for it to guess and learn the values itself 
Philosophical problems of not knowing what we want

13 choosing what to choose
Indirect normativity
Coherent extrapolated volition 
More robust approach
AI should do that which we would want it to do if we knew more and based on what most people would want as long as most would agree 
Otherwise Hard to lay down rules without bad second order effects
Moral permissibility and moral rightness

14 strategic picture
​
15 crunch time
Philosophy with a deadline
0 Comments

Your comment will be posted after it is approved.


Leave a Reply.

    Archives

    June 2021
    May 2021
    March 2021
    February 2021
    January 2021
    December 2020
    November 2020
    October 2020
    September 2020
    August 2020
    July 2020
    April 2020
    January 2020
    December 2019
    November 2019
    October 2019
    September 2019
    August 2019
    July 2019
    May 2019
    March 2019
    January 2019
    December 2018
    November 2018
    October 2018
    September 2018
    August 2018
    July 2018
    June 2018
    May 2018
    April 2018
    February 2018
    January 2018
    November 2017
    October 2017
    September 2017
    May 2017
    April 2017
    November 2016
    October 2016
    September 2016
    August 2016
    July 2016
    June 2016
    May 2016
    December 2015
    November 2015
    October 2015
    September 2015
    August 2015
    July 2015
    June 2015
    May 2015
    April 2015
    March 2015
    February 2015
    January 2015
    December 2014
    November 2014
    October 2014
    September 2014
    August 2014
    July 2014
    June 2014
    May 2014
    April 2014
    March 2014
    February 2014
    January 2014
    December 2013
    November 2013
    October 2013
    September 2013
    August 2013
    July 2013
    June 2013
    May 2013
    April 2013
    March 2013
    February 2013
    January 2013
    December 2012
    November 2012
    October 2012
    September 2012
    August 2012
    July 2012
    June 2012
    May 2012
    April 2012
    March 2012
    February 2012
    January 2012
    December 2011
    November 2011
    October 2011
    September 2011
    August 2011
    July 2011
    June 2011
    May 2011
    April 2011
    March 2011
    February 2011
    January 2011
    December 2010
    November 2010
    October 2010
    September 2010
    August 2010
    July 2010
    June 2010
    May 2010
    April 2010
    March 2010
    February 2010

    Categories

    All
    Angel Investing
    Cacti
    Cars
    China
    Community Service
    Culture
    Design
    Djing
    Dogs
    Education
    Entertainment
    Entrepreneurship
    Family
    Finance
    Food
    Google
    Happiness
    Incentives
    Investment Banking
    Judaism
    Law
    Lighting
    Magic
    Marketing
    Medicine
    Networking
    Nolabound
    Philosophy
    Professionalism
    Psychology
    Reading
    Real Estate
    Religion
    Romance
    Sales
    Science
    Shangri-La
    Social Entrepreneurship
    Social Media
    Sports
    Teams
    Technology
    Travel
    Turtles
    Ucla
    Venture Capital
    Web Services
    Weddings
    Zen

    Subscribe

    RSS Feed

Picture
Picture
  • Home
  • About
  • Interests
    • Angel investing
    • Magic
    • Scuba Diving
  • Blog
  • Contact