Prasanth Thangavel

Ilya Sutskever: "Sequence to Sequence Learning with Neural Networks: What a Decade"

2 min read

SpeakersIlya Sutskever
SourcesYoutube, Twitter

The overarching message is that while the original approach was revolutionary and led to tremendous progress, the field must evolve beyond current methods (e.g., pre-training) to achieve next-level AI capabilities (superintelligence, self-awareness, etc.).

Interestingly, some of these points are corroborating with Situational Awareness written by Leopold Aschenbrenner

Main Points

1. Original Success Formula (2014)

  • What we got right? Early scaling hypothesis that gave us immense progress
    • Large neural network/ Better models/algos
      • Autoregressive model trained on text
    • Large dataset
    • Bigger compute
  • What we go wrong? - The LSTM - Pipelining
  • Where are we now? - Neural networks can mimic human cognitive functions for tasks like translation, and perform on-par or even better in certain evals though being unreliable at times.

2. Evolution of Pre-training

  • Led to breakthrough models like GPT-2, GPT-3
  • Drove major AI progress over the decade
  • However, pre-training era will eventually end due to data limitations, despite compute growing

3. Data Limitation Crisis

  • We only have “one internet” worth of data
  • Data is becoming AI’s “fossil fuel”
  • This forces the field to find new approaches

Key Conclusions:

1. Future Directions (Near term)

  • Need to move beyond pure pre-training
  • Potential solutions include:
    • Agent-based approaches
    • Synthetic data
    • Better inference-time compute
  • Brain and body size relationships in evolution
    • Biology figured out how to scale somehow
    • An interesting and promising outlook for future of AI that we will figure out.

2. Path to Superintelligence (Long term)

  • Current systems will evolve to be:
    • Truly agentic (versus current limited agency)
    • Capable of real reasoning
    • More unpredictable future
    • Self-aware
  • This transition will create fundamentally different AI systems from what we have today

3. Historical Perspective

  • The field has made incredible progress in 10 years
  • Many original insights were correct, but some approaches (like pipelining) proved suboptimal
  • We’re still in early stages of what’s possible with AI

© Credits to the Youtube comment for summarising this talk effectively!

Graph View