Show notes

I’m pleased to release the second episode today. Here are the show notes:

  • 0:00 Intro
  • 1:57 News, announcements, and introducing show topics
  • 4:26 The Normalization of AI
  • 12:06 Investment, M&A, and the markets
  • 16:03 Self-driving cars
  • 17:20 Advancements in deep learning / neural networks
  • 18:50 Improvements in robotics
  • 21:26 Cloud Platforms & Open-source software
  • 25:05 Hardware improvements
  • 26:29 Development tools
  • 31:30 Conclusion

And here is a full transcript of the show:

Constructed Intellect Intro

You are listening to the Constructed Intellect podcast, where we cover the business and technology of artificial intelligence.

I’m your host, Ray Grieselhuber, and today we’re going to be the last twelve months in AI and machine learning developments. Additionally, I’m going to cover some of the top news items from the last week, and talk about next week’s show. I ended up missing last week’s show, the first week of the year is always a little crazy for me. But I’m back and glad to have you with me as well.

You can follow along with the show notes, which you can find for every episode at ConstructedIntellect.com. Today’s episode is episode 2.

If this is your first time listening, Thanks for joining. The Constructed Intellect podcast is produced pretty much every week. Come back often, tell your friends, and be sure to subscribe via iTunes or Stitcher.

We also post full-transcripts of each show, along with the show, on our website. You can subscribe to our RSS feed to get regular updates there as well. We’re on Twitter at @conintellect and Facebook at Facebook.com/ConstructedIntellect. All links are in the show notes.

If you’re looking for daily news on AI and machine learning, we have an email newsletter that goes out almost every day, summarizing the latest news and trends you’re most going to want to be aware of, and you can subscribe right on our website.

Alright, let’s get into the show.

News, announcements, and introducing show topics

I’m going to kick off today’s episode with some commentary on the last week’s worth of news in AI. After that, I’ll get into the main topic for today, which is a look at the most important AI / ML developments over the last 12 months. And lastly, as always, I’ll give a quick heads up for next week’s show.

First, some news.

As a reminder, I send out a daily newsletter every few days, at the moment. Once I get a few more things automated, it will go out daily. The most recent one just went out on January 4th. While preparing for it, I realized that a ton of things had happened, as I went through over 80 stories and curated the best ones for the newsletter edition.

In that newsletter, there was a really good link that summarized a lot of the intellectual innovations in AI over the last twelve months. It’s on the website now and I’ll link to it again in the show notes.

CES was this last week. I’m sure we’ll be hearing many updates this coming week of things that people saw during the event. I didn’t get a chance to make it out there myself, but I have heard that AI and robotics played a prominent role.

What I think will be interesting to come out of CES will be to see the merging of the best ideas from many industries into creating the next generation of AI and ML.

Another newsletter will be going out in the next couple of days, so stay tuned for that.

In the main topic of today’s show, I’m going to discuss what I consider to be the biggest events and changes in AI over the last 12 months. I’ve broken this into eight categories:

Believe me, there is a lot of stuff I’m not covering here. I’ll post all of the relevant links that I can find and even still it doesn’t cover everything that’s happening right now. But the CI website is always a good place to start.

  1. The normalization of AI
  2. Investment, M&A, and the markets
  3. Self-driving cars
  4. Advancements in deep learning / neural networks
  5. Improvements in robotics
  6. Cloud Platforms & Open-source software
  7. Hardware improvements
  8. Development tools

The normalization of AI

Let’s kick things off with what I call the normalization of AI.

2015 was the year that the normalcy bias against AI - and AI being a legitimate threat in the future - was breached. People who were responsible for this breach: Nick Bostrom, Elon Musk, Bill Gates, Sam Altman, and the others at the Open AI project. Many people were simply not aware of how much closer the reality of an existentially threatening AI had moved. Granted, we could still be years or even decades away, but that’s not a very long time. And the truth is, we don’t know when it will happen. We won’t know if AI is an existential risk until it’s too late. Because it will function very nicely for us, until the minute that it doesn’t.

This is the central message of Nick Bostrom’s book, Superintelligence. The timing on the release of his book, the end of 2014, was perfect. It was, in my opinion, important in 2015 not necessarily because of how well it was written - it’s a little plodding at time - but because it created a relatively complete index of the ways AI-based innovation can go very, very bad.

Most of these missteps can be characterized as what Bostrom labels “perverse instantiations.” A basic example that Bostrom gives is that an AI programmer might build an artificial intelligence system to manage a paperclip factory. The goal that the AI system has is to build as many paperclips as efficiently as possible. The problem would occur is if, somehow, that AI gained superintelligence, gained control over the entire world’s infrastructure and optimized everything on the planet, then the galaxy, to build paperclips.

It’s a silly example, of course, but illustrates very well the problem of perverse instantiations.

Perverse instantiations are a subset of the difficulty with programming and software design in general. The problem is with empathy and context. How do you program empathy? Ethics? Context? Knowing when enough is enough? It sounds relatively straightforward at first, but it’s not. For this reason alone, Bostrom’s book is worth reading.

In software development today, we depend on empathy on behalf of the programmer to get the user constraints correct. This is a source of trillions of dollars worth of bugs. Perverse instantiations will be an example lack of empathy on behalf of the AI systems. We have yet to master true empathy as humans toward one another. We can’t even achieve a universal ethic of empathy. How can we possibly hope to build AI systems that are capable of understanding what we “mean”? What do we mean? Empathy works well in the golden rule context, in individual relationships. Scaling it, so far, is impossible.

Truth be told, we are probably better off, in terms of existential risk management, of stopping all future technology development until we develop enough as a species on a level in order to solve these questions. Maybe that’s impossible. Maybe it’s impossible to develop as a species in this way until we are faced with these questions. Lots to think about.

This is the problem with life. It wants to take over everything, and can only see its own version of life as the best example of life. Fortunately, or not, the problem with life is solved by random death. The risk with a superintelligent AI is that it can’t be killed. Life has never experienced this sort of constraint removal before. The consequences could be disastrous.

There are various ways to define AI. One is just a system that learns. We have machine learning algorithm. Ok: definition has been met. But it doesn’t feel like real AI to us. Why? Because it’s not super intelligent? Or because we can kill at at any time, simply by killing the software process that runs it? Does this lead us to another, scarier definition of AI?

That something is AI only once it decides not to die, or allow itself to be killed. It’s probably already too late for humans if a piece of software ever reaches that point.

We have to look at the big picture, and this is the reason I love Bostrom’s book, Superintelligence. From the way he writes, it seems that he is aware of the end game in the evolution of intelligence. He casually defines early-stage AI as an AI that only uses the partial resources of a planet. He knows that the primary, immediate goal of a Superintelligence should be to replicate beyond the bounds of this planet as quickly as possible. He probably knows the end game. The end game is this: the heat death of the universe. If intelligent life can’t survive that, in one form or another, then there literally is no meaning at all. To anything.

It is safe to assume that a super intelligent AI will have read all the literature on preventing AI from getting out of control. This is the fundamental risk with technology. Oswald Spengler talks about this.

There was a great article series on the website Wait But Why that illustrated all of this very nicely, linked in the show notes.

In any case, the political and social consequences are going to be severe.

There was a really cool post on Pivotal Labs website titled “Abstraction, or, The Gift of Our Weak Brains.” It’s linked in the show notes. Go read it later.

Much of the conversation in AI centers around how much more powerful AI will be than human minds because of its cognitive abilities. This article is a reminder of how human’s greatest weakness, when compared with a sufficiently advanced AI, may turn out to be our greatest strength. If AIs ever develop this capability, then this whole point is moot, of course, but it remains to be seen when and if that will ever happen. Our ability to draw connections and create new ideas is truly fascinating to study. It’s one of the reasons I’m skeptical of our so-called modern, rational mindset in so much of our discourse.

Allowing people to indulge in what we would call irrational, abstract story creation may be a critical tool tool to allow humans the ability to discover fascinating new abstractions about the world. Anyway, that’s just a hypothesis. I’m sure one could easily tear it apart, but it’s interesting to think about.

So this awakening toward the existential risks in AI has been a big part of the normalization of AI over the last year. That’s part of the story. The other part of the story is that, and I touched on this in my first episode, is simply that innovation and investment in AI is back, in a big way. I’ll talk about this more through the rest of the show.

Investment, M&A, and the markets

One of the coolest things over the last year has been to watch the amount of activity that has formalized and increased in 2015 toward machine intelligence. Most of the VCs I interact with have at least one, if not more, partners and / or associates highly focused on identifying opportunities in this space.

Probably the canonical example of AI-based acquisitions over the last couple of years was Google’s acquisition of DeepMind in early 2014. We live today in the world of Artificial Narrow Intelligence (ANI). This is all of the companies out there, including Google, using machine learning to solve problems in narrow domains. DeepMind is probably the highest profile company, especially now because it is part of Google (or Alphabet, will take me some time to get used to that name), working on Artificial General Intelligence (AGI). That is, generalized machine intelligence that can carry lessons learned in one problem domain into other, completely new problem domains, and eventually be successful.

DeepMind’s CEO demonstrated this concept very well in a video I posted the other day. I’ll link to it again in the show notes. He was able to demonstrate that it’s possible to train generalized AI algorithms on video games. Plural. Not a video game. But take the same instance of intelligence from one game to another. And the system was able to take lessons it had learned in previous games into new games. Assuming it’s not some form of smoke and mirrors, this is a big leap forward. So that was a good acquisition.

Vicarious, another startup focusing on AGI, raised more money in 2015.

Toyota has committed to investing $1B in AI, in self-driving cars, of course.

$300m was put into AI companies in 2014. The numbers are not yet in for 2015, but it would be surprising indeed if that number was not close to double.

There is a new report out just today on Hizook.com that in 2015, robotics companies alone raised almost a billion dollars in VC funding. This, of course, does not include all of the other AI investments.

Shivon Zillis is an investor and analyst who covers the AI space. She did a nice writeup on TechCrunch that categorizes the various types of AI companies.

She also has put together what is probably the first infographic on the web in mapping out all of the various AI companies, broken out into the categories I just mentioned a second ago. Looking at this list, seeing the companies there, and digging into CrunchBase is probably the best way, at the moment, to get a feel for the volume in this space. If I had time, I’d do a project where I connected all of this online but am currently too swamped to do it. If someone out there is interested in doing something like this, send me a link and I’ll be sure to link to it.

Tesla is increasingly becoming an AI company, that just happens to do cool stuff with cars and batteries. Their stock price remains flat, compared with the beginning of 2015, and I believe they have some very strong competitive risks up ahead of them. But nobody can question that they, combined with Google, are leading the way in this space.

Self-driving cars

2015 was a huge year for self-driving cars.

It even ended on a fascinating note: a guy even built a self-driving car in his garage. Granted he’s a smart dude, the first person to unlock an iPhone, but the fact that this is now a problem that individuals can start to take on shows the maturity that currently exists in open-source software and the accessibility of automotive hardware to software hackers.

Cars are becoming a very interesting platform to hack on, because they combine some of the biggest challenges in AI: computer vision, robotics, sensors, real time response, learning how to make the right decisions, and more.

It’s actually quite amazing that so much has been accomplished in this arena and evidence, yet again, that software development in general is, in today’s world the most powerful mechanism to affect change. These developments didn’t come from any of the automotive companies. They came from an AI company that masquerades as a search engine.

Which leads us into the conversation around neural networks.

Deep learning / neural networks

I spent quite a bit of time talking in my last podcast episode talking about neural networks. The truth is, neural networks have some severe limitations. There have been some really good articles covering these limitations. I’ll link to these in the show notes.

Limitations of Neural Networks.

That being said, there is no question of a resurgence in interest in neural networks. There is a good article on Medium’s Backchannel that interviews DeepMind’s Demis Hassabis. During the interview, the topic of Geoff Hinton’s work and its influence on this resurgence in deep learning comes up. Hassabis confirmed that it has had an influence on his work.

Many of the new ML projects I’m personally aware of are making use of neural networks to one degree or another. The point is that a lot of this interest became more formal in 2015, anecdotally speaking. I expect it to increase in 2016, as well as an awareness of some of the inherent limitations. And it’s very possible that once people start encountering these limitations, if work-arounds aren’t found, some of the current hype around AI, particularly AGI, will dry up. ANI isn’t going anywhere though.

Hype around neural networks can be an important canary in the coal mine for measuring the level of hype and excitement around AI in general. The are an important part of the toolkit for now though.

Improvements in robotics

What can I say about robotics in 2015?

I think everybody has been impressed with the dramatic evolutions we saw. Drones became a centerpiece of many conversations, with some autonomous drones starting to emerge. There is still a lot of innovation happening around more anthropic robotics designs - humanoid style designs, large and small, two limbed and four. There were some advances there but I think we’re probably another year or two out from seeing something mind blowing there. So far, the best designs in robotic adaptive mobility are still Boston Dynamic’s designs, again now owned by Google.

One thing that has been interesting, and long overdue, has been the separation in the imagination of pretty much everybody the idea of AI from robotics. People as a whole are beginning to grasp, probably as a result of the growth of cloud technologies, that AI can and will live outside individual devices / robots.

On the flip side, robots are now already everywhere. Like many of the advances in AI, its proliferation will be gradual and under the radar, until one day it’s not. We are surrounded by drones, industrial automation, cargo bots, and more.

One of the areas that is continuing to attract a lot of hackers is computer vision. A buddy of mine put together, in just a few short months, a very sophisticated computer vision system with broad ranging applications. I can’t go into details but when you realize what small teams can accomplish with tools like web cameras and OpenCV, you realize the power we have at our disposal right now.

On a similar note, Oculus is shipping in Q1 and people are getting very excited about VR. I believe that there are going to be interesting technology integrations that make use of computer vision and VR technologies in the next year or two.

The FAA regulations about drones that occurred in 2015 were long-expected. Watch this space for sure in 2016.

Overall, 2015 was a very important year in robotics but I would characterize it as a year with mostly micro (vs. macro) innovations. The net effect of these innovations in 2015 will have a large impact on how things continue to evolve in the next couple of years.

Cloud Platforms and Open-source software

Google’s open sourcing of TensorFlow was probably the biggest announcement to come from a large technology company. If you’re listening to this podcast, you probably already know what TensorFlow is. If you’re not, I have some links and tutorials to the main TensorFlow site on the Learn AI guide.

Another quality project for Python is Theano, a compiler for symbolic expressions. Theano had a new release in the spring of 2015.

Many of the quality open source toolkits are being used to support development on neural networks-based intelligence systems.

Nathan Benaich (apologies if I’m butchering that last name) had a great article from a VCs perspective on investing in AI. He’s a partner at Playfair Capital, which focused heavily on AI investments. The biggest takeaway from his article, and the general consensus of most experts I talk to is that there are still a lot of problems around data, having quality data sets to learn from.

We’re living in a world that is exploding with data, so this seems counter-intuitive, but as someone coming from the analytics world, I can attest that having too much data can be just as much of a problem as not having enough data, if quality is a problem. Defining quality in the first place is challenging and doing so in a way that scales across data sets with records in the billions or trillions is almost impossible for many projects. The problem with this is that if you can’t create quality training data, your learning system won’t have anything good to learn from.

The Shivon Zillis article, Machine Intelligence in the Real World, I mentioned earlier touches on this as well. This is where her categorization of AI startups becomes really useful. Her categorizations are largely based on how startups solve the data problem. For example, looking at some of her categories. Panopticons collect broad datasets, utilizing the principle of “in the land of the blind, the one eyed-man is king” principle. Another category, the opposite of Panopticons are Lasers. Lasers are working on collecting focused datasets for highly vertical problem domains. A third category, Alchemists, take existing data and turn that data into “gold.” She mentions 8 categories overall and these will surely evolve. At any rate, it’s a great article. Go check it out.

This whole discussion made a lot of sense to me personally coming from the cloud / analytics world. Quality of data and the difficulty in collecting that data is THE problem we face every day.

On this note, I’ll point to another really good article, written by the CEO of Metabase, on the last twelve months in analytics. He covers some of the really exciting technologies that are gaining traction in startups and the corporate world. All of these are data / analytics tools that feed directly into my previous point, about the difficulty of data management. I believe that we’re going to continue to see a lot of convergence between cloud-based data management and analytics systems with AI and machine-based intelligence.

Some of the cooler projects he mentions are Druid, which I was surprised to see gaining so much traction but I’ve always thought was a great idea, Spark, TensorFlow (again), as well as some interesting news and updates about public and private markets in the space. Definitely worth a read.

Hardware improvements

Ok, hardware.

Facebook open-sourced the hardware designs of a server architecture it calls Big Sur. Big Sur includes eight GPU boards and is remarkable energy efficient. Facebook uses the Big Sur architecture to continue to push the performance of training its neural networks on the variety of data it’s producing, including photos, posts, and so forth.

As a reminder, Facebook was behind the initial release of the Open Compute Project, which is an organization that has the goal of collaborating on hardware design for machine intelligence.

Architectures like Big Sur, and GPU-based computing more generally, are going to continue to have a major impact on the resurgence of deep learning technologies over the coming years.

As more demands are placed on chip architecture to deal with the computing needs for machine intelligence, we’re going to see a lot of innovation in this space. In 2015, Nvidia did a good job of capturing some of the marketing / branding around GPU-based computing. Both Intel and AMD signaled that they were continuing to invest heavily, not just in GPUs, but in competing architectures as well.

Development tools

Finally, let’s talk about development tools. It’s such a huge topic that I won’t be able to cover it all but I am noticing some trends.

2015 was a great year for tools. The early days of 2016 are as well.

Have you ever had that feeling, where you’re reading somebody’s post about the tools they use? It rarely comes along these days, at most, about once a year. But I’ve learned to recognize it and, every time it happens, I know that we’re at the beginning of another technology revolution. I had it in the early days of the internet. I had it when blogs were becoming a thing. I had it in the early days of Twitter, especially when FriendFeed launched. I had it the first time I read a Paul Graham essay. It’s a visceral feeling. I used to get this same feeling walking through a library. It feels like options. Opportunity. Like a whole new world is out there that you didn’t know about.

I am feeling that a lot these days. I feel it every time I read some new catalog of machine learning tools, or find a new class on Coursera, or read one of Fogus’s annual best things lists. His last one was great by the way.

Surprisingly, it’s always the list of slightly unconventional, maybe even primitive tools that seem to give me this feeling the most. It’s not those tools in particular. It’s the mind of the person who selected those tools.

In terms of programming languages, I’ve personally paid the most attention to Java and Python over the last year. If there are other I’m originally a Java guy but got so burned out on it by 2006 / 2007, that I switched entirely into dynamic languages, Python first and then Ruby. As we’ve scaled things further at my first company, GinzaMetrics, I’m finding that we’re migrating more of our performance-sensitive systems back to Java (the main part of the app is Ruby on Rails).

I coded mostly in Python for about 2 years, then switched almost entirely to the Rails stack when I first started my company. More and more, especially as I get back into machine learning / data analysis projects, I’m leaning much more back to Python and Java in most cases. It’s probably no accident that these two languages have so much support in the AI world, given that they are both used heavily and supported by Google.

Each language has its own set of tools and libraries surrounding. I’d probably personally start with Python for new projects, and consider Java if I knew I would need certain things in terms of either performance or library support.

Other languages, languages that I consider to be more geared toward data science and analysis work are also quite popular. These include R and Matlab. I continue to hear good things about Julia. Lisp, of course, especially for programmers who like functional programming has a revered place but I don’t see or hear of two many new Lisp projects these days.

In the Python world, I’ve kind of been fascinated with IPython and Jupyter notebooks. Most of my old man friends look a little sideways at this project, but it’s cool to see the way code snippets can be easily shared and run from something that feels like a Hackpad document (the best, real time, shared rich text editor, IMO, acquired by Dropbox a couple of years ago).

I’ve noticed a lot of really interesting notebooks being shared and these notebooks seem to be a particularly good way to share tutorials and setup guides.

Apache Spark is something I touched on a little earlier but, for AI / ML in the cloud at least, it is quickly becoming a standard in data science and machine learning applications with high performance requirements. A preview of version 1.6 was released in November and the final version was just released this week.

Another important tool that has really blossomed in 2015 has been the sheer amount of online training available through courses and even companies sharing. It’s almost a perfect environment, at least in terms of volume of information available, for programmers to learn new skills and get involved with AI. I expect more of this to continue. As always, when you have so much information and so many options, the next challenge is then to know which technologies are going to achieve critical mass, because we are still in the early days of a lot of this.

Conclusion

Well, that wraps up today’s show. We covered a lot of material and I hope it was useful for you. Feel free to send more suggestions of things I may have missed over the last year and I’ll put the really good ones up on the site and / or send them out in the newsletter.

Next week’s show will cover what I see happening in AI and machine learning in 2016.

Thanks for listening and enjoy the rest of your week.