Evaluation

Natural Language Processing Large Language Models Deep Learning Evaluation Commonsense Reasoning Italian

GITA4CALAMITA - Evaluating the Physical Commonsense Understanding of Italian LLMs in a Multi-layered Approach: A CALAMITA Challenge

In the context of the CALAMITA Challenge, we investigate the physical commonsense reasoning capabilities of large language models (LLMs) and introduce a methodology to assess their …

Giulia Pensa

Ekhi Azurmendi

Julen Etxaniz

Begoña Altuna

Itziar Gonzalez-Dios

• Dec 6, 2024 • 1 min read

PDF Code Dataset

Natural Language Processing Large Language Models Deep Learning Evaluation Multilinguality Culture Basque

BertaQA: How Much Do Language Models Know About Local Culture?

Large Language Models (LLMs) exhibit extensive knowledge about the world, but most evaluations have been limited to global or anglocentric subjects. This raises the question of how …

Julen Etxaniz

Gorka Azkune

Aitor Soroa

Oier Lopez de Lacalle

Mikel Artetxe

• Jun 11, 2024 • 1 min read

PDF Code Dataset arXiv

Natural Language Processing Large Language Models Deep Learning Evaluation Reproducibility

Lessons from the Trenches on Reproducible Evaluation of Language Models

Effective evaluation of language models remains an open challenge in NLP. Researchers and engineers face methodological issues such as the sensitivity of models to evaluation …

Stella Biderman

Hailey Schoelkopf

Lintang Sutawika

Leo Gao

Jonathan Tow

Baber Abbasi

Alham Fikri Aji

Pawan Sasanka Ammanamanchi

Sidney Black

Jordan Clive

Anthony DiPofi

Julen Etxaniz

Benjamin Fattori

Jessica Zosa Forde

Charles Foster

Jeffrey Hsu

Mimansa Jaiswal

Wilson Y. Lee

Haonan Li

Charles Lovering

Niklas Muennighoff

Ellie Pavlick

Jason Phang

Aviya Skowron

Samson Tan

Xiangru Tang

Kevin A. Wang

Genta Indra Winata

François Yvon

Andy Zou

• May 23, 2024 • 1 min read

PDF Code arXiv

Natural Language Processing Large Language Models Evaluation Data Contamination Deep Learning

NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark

In this position paper, we argue that the classical evaluation on Natural Language Processing (NLP) tasks using annotated benchmarks is in trouble. The worst kind of data …

Oscar Sainz

Jon Ander Campos

Iker García-Ferrero

Julen Etxaniz

Oier Lopez de Lacalle

Eneko Agirre

• Oct 27, 2023 • 1 min read

PDF arXiv

No results found

Evaluation

GITA4CALAMITA - Evaluating the Physical Commonsense Understanding of Italian LLMs in a Multi-layered Approach: A CALAMITA Challenge

BertaQA: How Much Do Language Models Know About Local Culture?

Lessons from the Trenches on Reproducible Evaluation of Language Models

NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark