Jason Yosinski

Hi there!

I'm a scientist working on understanding and controlling large neural networks, these days at Open AI. I also serve as president of the ML Collective research group, where we work on fun problems and try to make ML research more accessible to all by collaborating across traditional industrial and academic lab boundaries. If you're interested in learning more, stop by our open reading group on Friday!

I completed my Ph.D. at Cornell, where at various times I worked with Hod Lipson (at the Creative Machines Lab), Yoshua Bengio (at U. Montreal's MILA), Thomas Fuchs (at Caltech JPL), and Google DeepMind. I was fortunate to be supported by a NASA Space Technology Research Fellowship, which gave me the opportunity to trek around and work with all these great folks.

Afterward, I helped start Uber AI after Uber acquired our startup, Geometric Intelligence, and subsequently founded a company, Windscape AI, to bring more efficient wind energy to the world. Over the years I've invested in and advised for companies including Recursion Pharmaceuticals, Mosaic ML, and a few others.

Find me on Bluesky, Twitter, or Github if you'd like.

Research

👇 Below are some older projects. 👉 More recent ones are over at ML Collective.

Nevermind all this, just show me the videos!
Or, if you prefer, here's a slightly outdated CV.

Selected Papersmore »

1. 👁 first page of paper
   
   (pdf)
   Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, Rosanne Liu. Plug and Play Language Models: a Simple Approach to Controlled Text Generation. International Conference on Learning Representations (ICLR) 2020. 26 April 2020.
   abstract▾ | bib▾

   Large transformer-based language models (LMs) trained on huge text corpora have shown unparalleled generation capabilities. However, controlling attributes of the generated language (e.g. switching topic or sentiment) is difficult without modifying the model architecture or fine-tuning on attribute-specific data and entailing the significant cost of retraining. We propose a simple alternative: the Plug and Play Language Model (PPLM) for controllable language generation, which combines a pretrained LM with one or more simple attribute classifiers that guide text generation without any further training of the LM. In the canonical scenario we present, the attribute models are simple classifiers consisting of a user-specified bag of words or a single learned layer with 100,000 times fewer parameters than the LM. Sampling entails a forward and backward pass in which gradients from the attribute model push the LM's hidden activations and thus guide the generation. Model samples demonstrate control over a range of topics and sentiment styles, and extensive automated and human annotated evaluations show attribute alignment and fluency. PPLMs are flexible in that any combination of differentiable attribute models may be used to steer text generation, which will allow for diverse and creative applications beyond the examples given in this paper.

   @article{dathathri-2020-ICLR-plug-play-language-models,
   	Author = {Sumanth Dathathri and Andrea Madotto and Janice Lan and Jane Hung and Eric Frank and Piero Molino and Jason Yosinski and Rosanne Liu},
   	Journal = {International Conference on Learning Representations (ICLR)},
   	Month = {April},
   	Title = {{Plug and Play Language Models: a Simple Approach to Controlled Text Generation}},
   	Year = {2020}}
   

2. 👁 first page of paper
   
   (pdf)
   Sam Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian Neural Networks. Advances in Neural Information Processing Systems (NeurIPS), 2019. 4 June 2019. arXiv page.
   abstract▾ | bib▾

   Even though neural networks enjoy widespread use, they still struggle to learn the basic laws of physics. How might we endow them with better inductive biases? In this paper, we draw inspiration from Hamiltonian mechanics to train models that learn and respect exact conservation laws in an unsupervised manner. We evaluate our models on problems where conservation of energy is important, including the two-body problem and pixel observations of a pendulum. Our model trains faster and generalizes better than a regular neural network. An interesting side effect is that our model is perfectly reversible in time.

   @article{greydanus-2019-arXiv-hamiltonian-neural-networks,
   	Author = {{Greydanus}, Sam and {Dzamba}, Misko and {Yosinski}, Jason},
   	Journal = {arXiv e-prints},
   	Month = {Jun},
   	Pages = {arXiv:1906.01563},
   	Title = {{Hamiltonian Neural Networks}},
   	Year = {2019}}
   

3. 👁 first page of paper
   
   (pdf)
   Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks?. Advances in Neural Information Processing Systems 27 (NIPS '14), pages 3320 - 3328. 8 December 2014.
   See also: earlier arXiv version. Oral presentation (1.2%).
   abstract▾ | bib▾

   Many deep neural networks trained on natural images exhibit a curious phenomenon in common: on the first layer they learn features similar to Gabor filters and color blobs. Such first-layer features appear not to be specific to a particular dataset or task, but general in that they are applicable to many datasets and tasks. Features must eventually transition from general to specific by the last layer of the network, but this transition has not been studied extensively. In this paper we experimentally quantify the generality versus specificity of neurons in each layer of a deep convolutional neural network and report a few surprising results. Transferability is negatively affected by two distinct issues: (1) the specialization of higher layer neurons to their original task at the expense of performance on the target task, which was expected, and (2) optimization difficulties related to split- ting networks between co-adapted neurons, which was not expected. In an example network trained on ImageNet, we demonstrate that either of these two issues may dominate, depending on whether features are transferred from the bottom, middle, or top of the network. We also document that the transferability of features decreases as the distance between the base task and target task increases, but that transferring features even from distant tasks can be better than using random features. A final surprising result is that initializing a network with transferred features from almost any number of layers can produce a boost to generalization that lingers even after fine-tuning to the target dataset.

   @inproceedings{yosinski_2014_NIPS
    title={How transferable are features in deep neural networks?},
    author={Yosinski, Jason and Clune, Jeff and Bengio, Yoshua and Lipson, Hod},
    booktitle={Advances in Neural Information Processing Systems 27 (NIPS '14)},
    editor = {Z. Ghahramani and M. Welling and C. Cortes and N.D. Lawrence and K.Q. Weinberger},
    publisher = {Curran Associates, Inc.},
    pages = {3320--3328},
    year={2014}
   }
   

4. 👁 first page of paper
   
   (pdf)
   Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 8 June 2015. Oral presentation (3.3%), CVPR 2015 Community Top Paper Award.
   See also: arXiv page.
   abstract▾ | bib▾

   Deep neural networks (DNNs) have recently been achieving state-of-the-art performance on a variety of pattern-recognition tasks, most notably visual classification problems. Given that DNNs are now able to classify objects in images with near-human-level performance, questions naturally arise as to what differences remain between computer and human vision. A recent study [30] revealed that changing an image (e.g. of a lion) in a way imperceptible to humans can cause a DNN to label the image as something else entirely (e.g. mislabeling a lion a library). Here we show a related result: it is easy to produce images that are completely unrecognizable to humans, but that state-of-the-art DNNs believe to be recognizable objects with 99.99% confidence (e.g. labeling with certainty that white noise static is a lion). Specifically, we take convolutional neural networks trained to perform well on either the ImageNet or MNIST datasets and then find images with evolutionary algorithms or gradient ascent that DNNs label with high confidence as belonging to each dataset class. It is possible to produce images totally unrecognizable to human eyes that DNNs believe with near certainty are familiar objects, which we call "fooling images" (more generally, fooling examples). Our results shed light on interesting differences between human vision and current DNNs, and raise questions about the generality of DNN computer vision.

   @article{nguyen-2015-CVPR-deep-neural-networks,
   	Author = {Nguyen, Anh and Yosinski, Jason and Clune, Jeff},
   	Journal = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
   	Month = {June},
   	Title = {Deep Neural Networks Are Easily Fooled: High Confidence Predictions for Unrecognizable Images},
   	Year = {2015}}
   
   

5. 👁 first page of paper
   
   (pdf)
   Yoshua Bengio, Éric Thibodeau-Laufer, Guillaume Alain, Jason Yosinski. Deep Generative Stochastic Networks Trainable by Backprop. Proceedings of the International Conference on Machine Learning. 21 June 2014.
   See also: supplemental section, earlier and later arXiv versions.
   abstract▾ | bib▾

   We introduce a novel training principle for probabilistic models that is an alternative to maximum likelihood. The proposed Generative Stochastic Networks (GSN) framework is based on learning the transition operator of a Markov chain whose stationary distribution estimates the data distribution. The transition distribution of the Markov chain is conditional on the previous state, generally involving a small move, so this conditional distribution has fewer dominant modes, being unimodal in the limit of small moves. Thus, it is easier to learn because it is easier to approximate its partition function, more like learning to perform supervised function approximation, with gradients that can be obtained by backprop. We provide theorems that generalize recent work on the probabilistic interpretation of denoising autoencoders and obtain along the way an interesting justification for dependency networks and generalized pseudolikelihood, along with a definition of an appropriate joint distribution and sampling mechanism even when the conditionals are not consistent. GSNs can be used with missing inputs and can be used to sample subsets of variables given the rest. We validate these theoretical results with experiments on two image datasets using an architecture that mimics the Deep Boltzmann Machine Gibbs sampler but allows training to proceed with simple backprop, without the need for layerwise pretraining.

   @inproceedings{bengio_icml_2014_deep-generative-stochastic-networks,
    title={Deep Generative Stochastic Networks Trainable by Backprop},
    author={Bengio, Yoshua and Thibodeau-Laufer, Eric and Alain, Guillaume and Yosinski, Jason},
    booktitle={Proceedings of the International Conference on Machine Learning},
    year={2014}
   }
   
   

6. 👁 first page of paper
   
   (pdf)
   Jeff Clune, Jason Yosinski, Eugene Doan, and Hod Lipson. EndlessForms.com: Collaboratively Evolving Objects and 3D Printing Them. Proceedings of the 13th International Conference on the Synthesis and Simulation of Living Systems. 21 July 2012.
   Winner of Best Poster award.
   abstract▾ | bib▾

   This abstract introduces EndlessForms.com, the first website to allow users to interactively evolve three-dimensional (3D) shapes online. Visitors are able to evolve objects that resemble natural organisms and engineered designs because the site utilizes a relatively new generative encoding inspired by concepts from developmental biology (Figure 1). This Compositional Pattern Producing Network (CPPN) encoding abstracts how natural organisms grow from a single cell to complex morphologies. Once evolved, visitors can click a button and have their evolved design 3D printed in materials ranging from plastic to silver. The site takes advantage of a recently released Web technology technology called WebGL that enables the visualization of 3D objects in Internet browsers. EndlessForms.com thus brings together recent innovations in evolutionary computation, Web technologies, and 3D printing to create a powerful collaborative interactive evolution experience that was not possible as recently as a year ago. In the first year of its release, nearly 3 million objects have been evaluated on the site during 190,000 generations of interactive evolution. A sizable community of citizen scientist participated: there were over 40,000 unique visitors from 150 countries and all 50 US states. In addition to its scientific mission of fueling intuitions regarding generative encodings for evolutionary algorithms, EndlessForms serves an educational outreach goal: visitors learn about evolution and developmental biology in a fun virtual setting and can transfer the 3D objects they create to the physical world (Figure 2). View a video tour of EndlessForms at http://goo.gl/YvoBw

   @inproceedings{clune2012endlessforms.com:-collaboratively-evolving-objects,
   	Author = {Jeff Clune and Jason Yosinski and Eugene Doan and Hod Lipson},
   	Booktitle = {Proceedings of the 13th International Conference on the Synthesis and Simulation of Living Systems},
   	Title = {EndlessForms.com: Collaboratively Evolving Objects and 3D Printing Them},
   	Year = {2012}
   }
   
   

7. 👁 first page of paper
   
   (pdf)
   Jason Yosinski, Jeff Clune, Diana Hidalgo, Sarah Nguyen, Juan Cristobal Zagal, and Hod Lipson. Evolving Robot Gaits in Hardware: the HyperNEAT Generative Encoding Vs. Parameter Optimization. Proceedings of the 20th European Conference on Artificial Life, Paris, France. pp 890-897. 8 August 2011.
   abstract▾ | bib▾

   Creating gaits for legged robots is an important task to enable robots to access rugged terrain, yet designing such gaits by hand is a challenging and time-consuming process. In this paper we investigate various algorithms for automating the creation of quadruped gaits. Because many robots do not have accurate simulators, we test gait-learning algorithms entirely on a physical robot. We compare the performance of two classes of gait-learning algorithms: locally searching parameterized motion models and evolving artificial neural networks with the HyperNEAT generative encoding. Specifically, we test six different parameterized learning strategies: uniform and Gaussian random hill climbing, policy gradient reinforcement learning, Nelder-Mead simplex, a random baseline, and a new method that builds a model of the fitness landscape with linear regression to guide further exploration. While all parameter search methods outperform a manually-designed gait, only the linear regression and Nelder-Mead simplex strategies outperform a random baseline strategy. Gaits evolved with HyperNEAT perform considerably better than all parameterized local search methods and produce gaits nearly 9 times faster than a hand-designed gait. The best HyperNEAT gaits exhibit complex motion patterns that contain multiple frequencies, yet are regular in that the leg movements are coordinated.

   @InProceedings{Yosinski2011EvolvedGaits,
    author = {Jason Yosinski and Jeff Clune and Diana Hidalgo and Sarah Nguyen and Juan Cristobal Zagal and Hod Lipson},
    title = {Evolving Robot Gaits in Hardware: the HyperNEAT Generative Encoding Vs. Parameter Optimization},
    booktitle = {Proceedings of the 20th European Conference on Artificial Life},
    year = {2011},
    month = {August},
    numpages = {8},
    location = {Paris, France},
   }
   

Google scholar | see all 41 papers and posters »

Selected Pressmore »

see more press »

Miscellaneous