Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Loading in …3
1 of 4




Download to read offline

AI Progress on Open Leaderboards - Benchmark Roadmap

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all


  1. 1. Leaderboards Framework AI Progress on Open Leaderboards - Benchmark Roadmap Perceive World Develop Cognition Build Relationships Fill Roles Pattern recognition Video understanding Memory Reasoning Social interactions Fluent conversation Assistant & Collaborator Coach & Mediator Speech Actions Declarative Deduction Scripts Speech Acts Tasks Institutions Chime Thumos SQuAD SAT ROC Story ConvAI Images Context Episodic Induction Plans Intentions Summarization Values ImageNet VQA DSTC RALI General-AI Translation Narration Dynamic Abductive Goals Cultures Debate Negotiation WMT DeepVideo Alexa Prize ICCMA AT Learning from Labeled Training Data and Searching (Optimization) Learning by Watching and Reading (Education) Learning by Doing and being Responsible (Exploration) 2015 2018 2021 2024 2027 2030 2033 2036 10/17/2017 (c) IBM 2017, Cognitive Opentech Group 1 Which experts would be really surprised if it takes less time… and which experts really surprised if it takes longer? Approx. Year Human Level ->
  2. 2. AI Trends 10/17/2017 © IBM Cognitive Opentech Group (COG) 2 Dota 2 “Deep Learning” for “AI Pattern Recognition” depends on massive amounts of “labeled data” and computing power available since ~2012; Labeled data is simply input and output pairs, such as a sound and word, or image and word, or English sentence and French sentence, or road scene and car control settings – labeled data means having both input and output data in massive quantities. For example, 100K images of skin, half with skin cancer and half without to learn to recognize presence of skin cancer.
  3. 3. Every 20 years, compute costs are down by 1000x • Cost of Digital Workers • Moore’s Law can be thought of as lowering costs by a factor of a… • Thousand times lower in 20 years • Million times lower in 40 years • Billion times lower in 60 years • Smarter Tools (Terascale) • Terascale (2017) = $3K • Terascale (2020) = ~$1K • Narrow Worker (Petascale) • Recognition (Fast) • Petascale (2040) = ~$1K • Broad Worker (Exascale) • Reasoning (Slow) • Exascale (2060) = ~$1K 310/17/2017 (c) IBM 2017, Cognitive Opentech Group 2080204020001960 $1K $1M $1B $1T 206020201980 +/- 10 years $1 Person Average Annual Salary (Living Income) Super Computer Cost Mainframe Cost Smartphone Cost T P E T P E AI Progress on Open Leaderboards Benchmark Roadmap to solve AI/IA
  4. 4. GPD/Employee 10/17/2017 (c) IBM 2017, Cognitive Opentech Group 4 (Source) Lower compute costs translate into increasing productivity and GDP/employees for nations Increasing productivity and GDP/employees should translate into wealthier citizens AI Progress on Open Leaderboards Benchmark Roadmap to solve AI/IA

Editor's Notes

  • Expert predictions on HMLI: URL

    2015 Pattern Recognition Speech: URL:
    2015 Pattern Recognition Images: URL:
    2015 Patten Recognition Translation: URL:

    2018 Video Understanding Actions: URL:
    > Also UCF101
    2018 Video Understanding Context: URL:
    2018 Video Understanding DeepVideo: URL:

    2021 Memory Declarative: URL:
    Also Allen AI Kaggle Science Challenge

    2024 Reasoning Deduction: URL:

    2027: Social Interaction Scripts: URL:

    2030: Fluent Conversation Speech Acts: URL:
    2030: Fluent Conversation Intentions: URL:
    2030: Fluent Conversation Alexa Prize: URL:

    2033: Assistant & Collaborator Summarization: URL:
    2033: Assistant & Collaborator Debate: URL:

    2036: Coach & Mediator General AI: URL:
    2036: Coach & Mediator Negotiation: URL:

  • 1950 Nathaniel Rochester (IBM) 701 first commercial computer that did super-human levels of numeric calculations routinely. He worked at MIT on arithmetic unit of WhirlWind I programmable computer.

    Dota 2 is most recent August 11, 2017 as a super-human game player in Valve Dota 2 competition – Elon Musk’s OpenAI result.

    Miles Bundage tracks gaming progress:


  • What is beyond Exascale? Zetta (21), Yotta (24)

    Time dimension (x-axis) is plus or minus 10 years….

    Daniel Pakkala (VTT)

    Dan Gruhl:
    In 1983 10 GF was ~10 million.
    That's 24.55 million in today's dollars.
    or 2.4 billion for 1 TF in 1983
    Today 1 TF is about $3k
  • Source:
  • ×