A good friend of mine who's an engineer and entrepreneur and really into AI recommended that I check out Superintelligence: Paths, Dangers, Strategies by Nick Bostrom. I just finished reading it and thought it was pretty eye-opening and scary.

I enjoyed the historical overview of the field of AI, and the many examples of current programs and how they rank against humans in various domains.

My biggest takeaways:
-McCarthy’s dictum: When something works, it’s no longer called AI.
-Quote from Knuth: computers can now do most of the things that require thinking but very few of the things that we or animals can do without thinking.
-If/when general AI is solved, the transition to superintelligence (above-human level intelligence) will happen too fast to respond to it at the time, so we should think and plan ahead.
-It's really hard to design objective functions/values for AI. Most strategies that on first order seem ok are really bad when considering second and third order effects.
-The most likely scenario is that we will get something wrong and basically be screwed. This is quite scary.
-Approaches like whole brain emulation seem interesting but really difficult to pull off in practice.
-The indirect value loading approach ("the AI should try to maximize whatever most of us would want it to maximize had we thought about it long and hard") seems interesting and compelling (and was new to me).

I'm personally skeptical we will ever achieve general AI. I think we'll just get better and better at domain-specific applications, but I don't think we'll ever figure out how to artificially make a machine think in the way we do (or for that matter understand how we truly think). I think it's just one of those mysteries that will never be fully solved.

I kind of lost steam about 2/3 of the way through the book when the level of detailed analysis of very futuristic scenarios seemed kind of overboard to me. I thought that it was hard to really effectively reason about situations which will in all likelihood be very different from anything we can imagine right now. It's good to be cautious and try to plan ahead, but I just thought it got too much into the weeds and too fine-grained for the immense uncertainty in question.

My full notes on the book are below.

Toddlers love to read the same book over and over again: it helps them learn language and gives them a small sense of security, comfort, predictability, and mastery over even a small part of the confusing world around them. While helpful for them, it can drive parents crazy (or lead them to come up with ridiculous premises for blog posts).

Yesterday my toddler insisted on my reading for the fifth time (that day) the Russian folk tale "Петушок и бобовое зёрнышко," or "The Rooster and the Bean." (Click those links to read the tale in the respective language.)

The gist of the story is that this poor rooster who is always rushing chokes on a bean, and in order to save his life, a chicken needs to solve one problem after another in an endless cascade of sub-tasks which finally allow her to save the rooster's life in the end.

The story teaches (at least) three obvious lessons. First, don't rush while eating (or in general). Second, accomplishing things in life (or getting anything non-trivial like a "project" done for that matter) is going to be much more complicated and involve a lot more dependent steps completed first. And third, often many other people will be involved whose help you will need in order to get what you want done.

But for me personally, I secretly got a kick out of reading the story because it also teaches a valuable lesson about technology development. When I was at Google, there were many times I tried to do something seemingly quite simple ("just change the color over here" or "just move this piece of code from here to here"), and it required 5+ steps of dependent work (refactoring, renaming, moving, etc.) to be completed in series before the final trivial change/fix could be made. I learned from a teammate that this is generally called "yak shaving," which refers to the seemingly endless series of useless activities which, by allowing you to overcome intermediate difficulties, allow you to solve a larger problem [Sources: 1, 2, 3].

And I've experienced the same thing in life and house projects as well: "I just want to install a camera here" devolves into "I need to run power here" which devolves into "I need to rewire, order, and set up a thousand other things first."

I guess it's never too early to learn the lesson that projects are complicated, and getting a big thing done requires patience with lots of steps, which is why this Russian folk tale about the rooster seem to teach this lesson so early on.

If you want to waste any more of your precious life, you can watch the pointless video below of the Ren & Stimpy Show episode about the (pointless) "Yak Shaving Day" that apparently inspired the term "yak shaving."