#51 FRANCOIS CHOLLET - Intelligence and Generalisation

Video Description

In today's show we are joined by Francois Chollet, I have been inspired by Francois ever since I read his Deep Learning with Python book and started using the Keras library which he invented many, many years ago. Francois has a clarity of thought that I've never seen in any other human being! He has extremely interesting views on intelligence as generalisation, abstraction and an information conversation ratio. He wrote on the measure of intelligence at the end of 2019 and it had a huge impact on my thinking. He thinks that NNs can only model continuous problems, which have a smooth learnable manifold and that many "type 2" problems which involve reasoning and/or planning are not suitable for NNs. He thinks that many problems have type 1 and type 2 enmeshed together. He thinks that the future of AI must include program synthesis to allow us to generalise broadly from a few examples, but the search could be guided by neural networks because the search space is interpolative to some extent. Panel; me, Yannic and Keith Tim Intro [00:00:00] Manifold hypothesis and interpolation [00:06:15] Yann LeCun skit [00:07:58] Discrete vs continuous [00:11:12] NNs are not turing machines [00:14:18] Main show kick-off [00:16:19] DNN models are locally sensitive hash tables and only efficiently encode some kinds of data well [00:18:17] Why do natural data have manifolds? [00:22:11] Finite NNs are not "turing complete" [00:25:44] The dichotomy of continuous vs discrete problems, and abusing DL to perform the former [00:27:07] Reality really annoys a lot of people, and ...GPT-3 [00:35:55] There are type one problems and type 2 problems, but...they are enmeshed [00:39:14] Chollet's definition of intelligence and how to construct analogy [00:41:45] How are we going to combine type 1 and type 2 programs? [00:47:28] Will topological analogies be robust and escape the curse of brittleness? [00:52:04] Is type 1 and 2 two different physical systems? Is there a continuum? [00:54:26] Building blocks and the ARC Challenge [00:59:05] Solve ARC == intelligent? [01:01:31] Measure of intelligence formalism -- it's a whitebox method [01:03:50] Generalization difficulty [01:10:04] Lets create a marketplace of generated intelligent ARC agents! [01:11:54] Mapping ARC to psychometrics [01:16:01] Keras [01:16:45] New backends for Keras? JAX? [01:20:38] Intelligence Explosion [01:25:07] Bottlenecks in large organizations [01:34:29] Summing up the intelligence explosion [01:36:11] Post-show debrief [01:40:45] Pod version: https://anchor.fm/machinelearningstreettalk/episodes/51-Francois-Chollet---Intelligence-and-Generalisation-ev1i79 Tim's Whimsical notes; https://whimsical.com/chollet-show-QQ2atZUoRR3yFDsxKVzCbj NeurIPS workshop on reasoning and abstraction; https://slideslive.com/38935790/abstraction-reasoning-in-ai-systems-modern-perspectives Rob Lange's article on the measure of intelligence (shown in 3d in intro): https://roberttlange.github.io/posts/2020/02/on-the-measure-of-intelligence/ Francois cited in the show; LSTM digits multiplication code example: https://keras.io/examples/nlp/addition_rnn/ ARC-related psychology paper from NYU: https://cims.nyu.edu/~brenden/papers/2103.05823.pdf This is the AAAI symposium Francois mentioned, that he co-organized; there were 2 presentations of psychology research on ARC (including an earlier version of the preprint above): https://aaai.org/Symposia/Fall/fss20symposia.php#fs04 https://fchollet.com/ https://twitter.com/fchollet https://www.linkedin.com/in/fchollet/ #deeplearning #machinelearning #artificialintelligence