When a company releases a new AI video generator, it’s not long before someone uses it to make a video of actor Will Smith eating spaghetti.
It’s become something of a meme as well as a benchmark: Seeing whether a new video generator can realistically render Smith slurping down a bowl of noodles. Smith himself parodied the trend in an Instagram post in February.
Google Veo 2 has done it.
We are now eating spaghett at last. pic.twitter.com/AZO81w8JC0
— Jerrod Lew (@jerrod_lew) December 17, 2024
Will Smith and pasta is but one of several bizarre “unofficial” benchmarks to take the AI community by storm in 2024. A 16-year-old developer built an app that gives AI control over Minecraft and tests its ability to design structures. Elsewhere, a British programmer created a platform where AI plays games like Pictionary and Connect 4 against each other.
It’s not like there aren’t more academic tests of an AI’s performance. So why did the weirder ones blow up?
For one, many of the industry-standard AI benchmarks don’t tell the average person very much. Companies often cite their AI’s ability to answer questions on Math Olympiad exams, or figure out plausible solutions to Ph.D.-level problems. Yet most people — yours truly included — use chatbots for things like responding to emails and basic research.
Crowdsourced industry measures aren’t necessarily better or more informative.
Take, for example, Chatbot Arena, a public benchmark many AI enthusiasts and developers follow obsessively. Chatbot Arena lets anyone on the web rate how well AI performs on particular tasks, like creating a web app or generating an image. But raters tend not to be representative — most come from AI and tech industry circles — and cast their votes based on personal, hard-to-pin-down preferences.
Ethan Mollick, a professor of management at Wharton, recently pointed out in a post on X another problem with many AI industry benchmarks: they don’t compare a system’s performance to that of the average person.
“The fact that there are not 30 different benchmarks from different organizations in medicine, in law, in advice quality, and so on is a real shame, as people are using systems for these things, regardless,” Mollick wrote.
Weird AI benchmarks like Connect 4, Minecraft, and Will Smith eating spaghetti are most certainly not empirical — or even all that generalizable. Just because an AI nails the Will Smith test doesn’t mean it’ll generate, say, a burger well.
One expert I spoke to about AI benchmarks suggested that the AI community focus on the downstream impacts of AI instead of its ability in narrow domains. That’s sensible. But I have a feeling that weird benchmarks aren’t going away anytime soon. Not only are they entertaining — who doesn’t like watching AI build Minecraft castles? — but they’re easy to understand. And as my colleague Max Zeff wrote about recently, the industry continues to grapple with distilling a technology as complex as AI into digestible marketing.
The only question in my mind is, which odd new benchmarks will go viral in 2025?
Having read this I believed it was extremely enlightening.
I appreciate you spending some time and effort to put this content
together. I once again find myself personally spending a significant amount of time both reading and posting comments.
But so what, it was still worth it!
You made some really good points there. I looked on the net for additional information about the issue and found most individuals will go along
with your views on this website.
Awesome blog! Do you have any recommendations for aspiring writers?
I’m hoping to start my own website soon but I’m a little lost on everything.
Would you advise starting with a free platform like
Wordpress or go for a paid option? There are so many choices out there that I’m completely confused ..
Any tips? Thank you!
It’s in point of fact a great and useful piece of info.
I’m satisfied that you simply shared this useful information with us.
Please stay us informed like this. Thank you for sharing.
Polska apteka w UK vitarosashop.com stanowi istotny
element polskiej społeczności mieszkającej UK.
Ta właśnie sklep online specjalizuje się na
oferowaniu bogatej gamy produktów farmaceutycznych,
suplementów diety oraz artykułów pierwszej potrzeby,
dopasowanych do wymagań polskich mieszkańców zamieszkujących w UK.
Polska apteka w UK vitarosashop.com zapewnia konkurencyjne
ceny na szeroki wybór, takie jak leki bez recepty, suplementy diety oraz kosmetyki.
Dzięki temu użytkownicy są w stanie korzystać z wysokiej jakości produktów
bez konieczności odwiedzania fizycznej apteki.
Jednym z głównych atutów Polskiej apteki w UK jest dostępność do lekarstw od polskich producentów, co gwarantuje
autentyczność produktów. Polska apteka UK vitarosashop.com zapewnia pewne zakupy online z szybką i dyskretną wysyłką,
co jest niezwykle ważne dla klientów poszukujących łatwego dostępu do leków.
Polska apteka UK Vitarosa Shop zajmuje się o zadowolenie klientów poprzez szeroki wybór produktów zdrowotnych, konkurencyjne
ceny oraz sprawną obsługę. Dzięki temu Polska apteka UK tworzy popularny wybór dla klientów polskiego pochodzenia w UK.
Zakupując w Polskiej apteki UK, użytkownicy mogą liczyć na szeroki wybór spełniające ich wymagania, konkurencyjne ceny
oraz szybką dostawę. Polska apteka w UK vitarosashop.com
jest idealnym rozwiązaniem dla klientów polonijnych poszukujących łatwego dostępu do leków w Wielkiej Brytanii.
Reasumując, Polska apteka UK Vitarosa Shop oferuje kompleksową ofertę produktów zdrowotnych,
konkurencyjne ceny oraz efektywną i dyskretną dostawę, co sprawia, że jest idealnym wyborem dla klientów polskiego pochodzenia żyjących
w Wielkiej Brytanii.
Its like you read my mind! You seem to know a lot about this, like you wrote the book in it
or something. I think that you could do with some pics to drive the message
home a bit, but instead of that, this is wonderful blog.
A fantastic read. I’ll definitely be back.
Great beat ! I wish to apprentice even as you amend your web site, how
could i subscribe for a blog website? The account helped
me a acceptable deal. I were a little bit acquainted of this your broadcast
offered shiny transparent idea
This site was… how do you say it? Relevant!! Finally I have found
something which helped me. Kudos!
Today, I went to the beachfront with my kids. I found a sea shell
and gave it to my 4 year old daughter and said “You can hear the ocean if you put this to your ear.” She placed the shell to her ear and screamed.
There was a hermit crab inside and it pinched her ear.
She never wants to go back! LoL I know this is entirely off topic but I had to tell
someone!
I’m not sure where you’re getting your information,
but great topic. I needs to spend some time learning much more or
understanding more. Thanks for magnificent info I was looking for this info for my
mission.
I am sure this post has touched all the internet people, its really really fastidious post on building up
new blog.