Benchmarks News

In 2029, will any AI be able to construct "reasonably" bug-free code of >= 10k LOC from a natural language specification? (Gary Marcus benchmark #4)

Sep 16, 8:43 PMJan 1, 8:00 AM

67.44%chance

776134

OptionVotes

YES

1008

945

In 2029, will any AI be able to work as a competent cook in an arbitrary kitchen? (Gary Marcus benchmark #3)

Sep 16, 8:37 PMJan 1, 8:00 AM

35.28%chance

848381

OptionVotes

YES

1784

1317

By 2029, will any AI be able to read a novel and reliably answer questions about it? (Gary Marcus benchmark #2)

Sep 16, 8:34 PMJan 1, 8:00 AM

92%chance

10442592

OptionVotes

YES

2568

1472

In 2029, will any AI be able to take an arbitrary proof in the mathematical literature and translate it into a form suitable for symbolic verification? (Gary Marcus benchmark #5)

Sep 16, 8:47 PMJan 1, 8:00 AM

38.95%chance

625520

OptionVotes

YES

1528

726

By 2029 will any AI be able to watch a movie and accurately tell you what is going on? (Gary Marcus benchmark #1)

Sep 16, 8:33 PMJan 1, 8:00 AM

92.83%chance

7424516

OptionVotes

YES

3220

964

Will at least 3 of the Gary Marcus benchmark questions resolve YES?

Sep 16, 9:19 PMJan 1, 8:00 AM

81.63%chance

281969

OptionVotes

YES

1093

943

Will it cost less than 100k USD to train and run a language model that outperforms GPT-3 175B on all benchmarks by the end 2024?

Apr 8, 6:21 PMJan 1, 4:59 AM

84.64%chance

546105

OptionVotes

YES

2396

810

Will more than 40 teams participate in the first round of the AI Forecasting Benchmark Series?

Jun 21, 12:39 AMJul 8, 6:59 AM

45.39%chance

2100

OptionVotes

YES

1097

912

Benchmark Gap #3: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use?

Apr 8, 6:06 PMJan 1, 4:59 AM

70.25%chance

12323

OptionVotes

YES

1045

975

Will OpenAI's next-gen math-focused model score at least 95% on the MATH benchmark?

Nov 28, 10:01 AMJan 1, 2:59 PM

71.21%chance

25967

OptionVotes

YES

1063

961

Will >50% of the tasks in the WebArena benchmark be solved by EOY 2024?

Jul 27, 2:24 AMJan 1, 4:59 AM

62.05%chance

14351

OptionVotes

YES

1028

918

Will openAI have the most accurate LLM across most benchmarks by EOY 2024?

Nov 20, 5:17 AMJan 1, 4:59 AM

39%chance

191281

OptionVotes

YES

1007

952

Latest stories

Linux Foundation Plans 'High Performance Software Foundation' Utilizing AI; AMD Launches Ryzen Threadripper 7000 Processors
Authors
6
7 months
AI
Tech
Alan Wake 2 Released to Mixed Reviews, Features New Difficulty Settings and Alternative Narrative
Authors
12
8 months
Tech

Benchmarks News

Prediction markets for Benchmarks

Prediction markets for Benchmarks

Prediction markets for Benchmarks

In 2029, will any AI be able to construct "reasonably" bug-free code of >= 10k LOC from a natural language specification? (Gary Marcus benchmark #4)

In 2029, will any AI be able to work as a competent cook in an arbitrary kitchen? (Gary Marcus benchmark #3)

By 2029, will any AI be able to read a novel and reliably answer questions about it? (Gary Marcus benchmark #2)

In 2029, will any AI be able to take an arbitrary proof in the mathematical literature and translate it into a form suitable for symbolic verification? (Gary Marcus benchmark #5)

By 2029 will any AI be able to watch a movie and accurately tell you what is going on? (Gary Marcus benchmark #1)

Will at least 3 of the Gary Marcus benchmark questions resolve YES?

Will it cost less than 100k USD to train and run a language model that outperforms GPT-3 175B on all benchmarks by the end 2024?

Will more than 40 teams participate in the first round of the AI Forecasting Benchmark Series?

Benchmark Gap #3: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use?

Will OpenAI's next-gen math-focused model score at least 95% on the MATH benchmark?

Will >50% of the tasks in the WebArena benchmark be solved by EOY 2024?

Will openAI have the most accurate LLM across most benchmarks by EOY 2024?

Latest stories

Prediction markets for Benchmarks

Prediction markets for Benchmarks

Prediction markets for Benchmarks

In 2029, will any AI be able to construct "reasonably" bug-free code of >= 10k LOC from a natural language specification? (Gary Marcus benchmark #4)

In 2029, will any AI be able to work as a competent cook in an arbitrary kitchen? (Gary Marcus benchmark #3)

By 2029, will any AI be able to read a novel and reliably answer questions about it? (Gary Marcus benchmark #2)

In 2029, will any AI be able to take an arbitrary proof in the mathematical literature and translate it into a form suitable for symbolic verification? (Gary Marcus benchmark #5)

By 2029 will any AI be able to watch a movie and accurately tell you what is going on? (Gary Marcus benchmark #1)

Will at least 3 of the Gary Marcus benchmark questions resolve YES?

Will it cost less than 100k USD to train and run a language model that outperforms GPT-3 175B on all benchmarks by the end 2024?