LLM GameLab: An Interactive Platform for Testing Large Language Models in Board Games

Paulina Morillo; Alex Terreros; Cèsar Ferri; José Hernández-Orallo

doi:10.1007/978-3-032-06129-4_36

LLM GameLab: An Interactive Platform for Testing Large Language Models in Board Games

Paulina Morillo
, Alex Terreros
, Cèsar Ferri
, José Hernández-Orallo

Research Group on Spatial Data Infrastructure Artificial Intelligence Geoportals and Applied Computing (IDE IA GEO CA)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

While large language models are constantly evaluated in various skills, such as math, general knowledge, and coding, their ability to understand and follow game rules has not yet been deeply explored. The latter is especially important as it allows testing whether LLMs can operate within predefined limits without deviating or making illogical mistakes. Therefore, this demo paper presents a tool for interacting with LLMs in board games. The tool allows the creation of players with different large language models pitted against each other or to play in human vs. LLM mode. The platform includes rules predefined in prompts for four simple games based on Tic-Tac-Toe and Connect Four. Each player can be evaluated to account for their illegal movements, wins, draws, losses, and response times. The application also allows for the creation of new games, opening up the possibility of examining LLM behavior in situations they have not previously encountered.

Original language	English
Title of host publication	Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track and Demo Track - European Conference, ECML PKDD 2025, Proceedings
Editors	Inês Dutra, Alípio M. Jorge, Carlos Soares, João Gama, Mykola Pechenizkiy, Paulo Cortez, Sepideh Pashami, Arian Pasquali, Nuno Moniz, Pedro H. Abreu
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	486-490
Number of pages	5
ISBN (Print)	9783032061287
DOIs	https://doi.org/10.1007/978-3-032-06129-4_36
State	Published - 2026
Event	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2025 - Porto, Portugal Duration: 15 Sep 2025 → 19 Sep 2025

Publication series

Name	Lecture Notes in Computer Science
Volume	16022
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2025
Country/Territory	Portugal
City	Porto
Period	15/09/25 → 19/09/25

Bibliographical note

Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.

Keywords

decision-making
General Game Playing
LLM evaluation

Access to Document

10.1007/978-3-032-06129-4_36

Cite this

Morillo, P., Terreros, A., Ferri, C., & Hernández-Orallo, J. (2026). LLM GameLab: An Interactive Platform for Testing Large Language Models in Board Games. In I. Dutra, A. M. Jorge, C. Soares, J. Gama, M. Pechenizkiy, P. Cortez, S. Pashami, A. Pasquali, N. Moniz, & P. H. Abreu (Eds.), Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track and Demo Track - European Conference, ECML PKDD 2025, Proceedings (pp. 486-490). (Lecture Notes in Computer Science; Vol. 16022). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-032-06129-4_36

Morillo, Paulina ; Terreros, Alex ; Ferri, Cèsar et al. / LLM GameLab : An Interactive Platform for Testing Large Language Models in Board Games. Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track and Demo Track - European Conference, ECML PKDD 2025, Proceedings. editor / Inês Dutra ; Alípio M. Jorge ; Carlos Soares ; João Gama ; Mykola Pechenizkiy ; Paulo Cortez ; Sepideh Pashami ; Arian Pasquali ; Nuno Moniz ; Pedro H. Abreu. Springer Science and Business Media Deutschland GmbH, 2026. pp. 486-490 (Lecture Notes in Computer Science).

@inproceedings{4e028f9b80df4c2f8c119d68c9dac3f7,

title = "LLM GameLab: An Interactive Platform for Testing Large Language Models in Board Games",

abstract = "While large language models are constantly evaluated in various skills, such as math, general knowledge, and coding, their ability to understand and follow game rules has not yet been deeply explored. The latter is especially important as it allows testing whether LLMs can operate within predefined limits without deviating or making illogical mistakes. Therefore, this demo paper presents a tool for interacting with LLMs in board games. The tool allows the creation of players with different large language models pitted against each other or to play in human vs. LLM mode. The platform includes rules predefined in prompts for four simple games based on Tic-Tac-Toe and Connect Four. Each player can be evaluated to account for their illegal movements, wins, draws, losses, and response times. The application also allows for the creation of new games, opening up the possibility of examining LLM behavior in situations they have not previously encountered.",

keywords = "decision-making, General Game Playing, LLM evaluation",

author = "Paulina Morillo and Alex Terreros and C{\`e}sar Ferri and Jos{\'e} Hern{\'a}ndez-Orallo",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.; European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2025 ; Conference date: 15-09-2025 Through 19-09-2025",

year = "2026",

doi = "10.1007/978-3-032-06129-4\_36",

language = "English",

isbn = "9783032061287",

series = "Lecture Notes in Computer Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "486--490",

editor = "In{\^e}s Dutra and Jorge, \{Al{\'i}pio M.\} and Carlos Soares and Jo{\~a}o Gama and Mykola Pechenizkiy and Paulo Cortez and Sepideh Pashami and Arian Pasquali and Nuno Moniz and Abreu, \{Pedro H.\}",

booktitle = "Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track and Demo Track - European Conference, ECML PKDD 2025, Proceedings",

address = "Germany",

}

Morillo, P, Terreros, A, Ferri, C & Hernández-Orallo, J 2026, LLM GameLab: An Interactive Platform for Testing Large Language Models in Board Games. in I Dutra, AM Jorge, C Soares, J Gama, M Pechenizkiy, P Cortez, S Pashami, A Pasquali, N Moniz & PH Abreu (eds), Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track and Demo Track - European Conference, ECML PKDD 2025, Proceedings. Lecture Notes in Computer Science, vol. 16022, Springer Science and Business Media Deutschland GmbH, pp. 486-490, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2025, Porto, Portugal, 15/09/25. https://doi.org/10.1007/978-3-032-06129-4_36

LLM GameLab: An Interactive Platform for Testing Large Language Models in Board Games. / Morillo, Paulina; Terreros, Alex; Ferri, Cèsar et al.
Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track and Demo Track - European Conference, ECML PKDD 2025, Proceedings. ed. / Inês Dutra; Alípio M. Jorge; Carlos Soares; João Gama; Mykola Pechenizkiy; Paulo Cortez; Sepideh Pashami; Arian Pasquali; Nuno Moniz; Pedro H. Abreu. Springer Science and Business Media Deutschland GmbH, 2026. p. 486-490 (Lecture Notes in Computer Science; Vol. 16022).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - LLM GameLab

T2 - European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2025

AU - Morillo, Paulina

AU - Terreros, Alex

AU - Ferri, Cèsar

AU - Hernández-Orallo, José

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.

PY - 2026

Y1 - 2026

N2 - While large language models are constantly evaluated in various skills, such as math, general knowledge, and coding, their ability to understand and follow game rules has not yet been deeply explored. The latter is especially important as it allows testing whether LLMs can operate within predefined limits without deviating or making illogical mistakes. Therefore, this demo paper presents a tool for interacting with LLMs in board games. The tool allows the creation of players with different large language models pitted against each other or to play in human vs. LLM mode. The platform includes rules predefined in prompts for four simple games based on Tic-Tac-Toe and Connect Four. Each player can be evaluated to account for their illegal movements, wins, draws, losses, and response times. The application also allows for the creation of new games, opening up the possibility of examining LLM behavior in situations they have not previously encountered.

AB - While large language models are constantly evaluated in various skills, such as math, general knowledge, and coding, their ability to understand and follow game rules has not yet been deeply explored. The latter is especially important as it allows testing whether LLMs can operate within predefined limits without deviating or making illogical mistakes. Therefore, this demo paper presents a tool for interacting with LLMs in board games. The tool allows the creation of players with different large language models pitted against each other or to play in human vs. LLM mode. The platform includes rules predefined in prompts for four simple games based on Tic-Tac-Toe and Connect Four. Each player can be evaluated to account for their illegal movements, wins, draws, losses, and response times. The application also allows for the creation of new games, opening up the possibility of examining LLM behavior in situations they have not previously encountered.

KW - decision-making

KW - General Game Playing

KW - LLM evaluation

UR - https://www.scopus.com/pages/publications/105020020385

U2 - 10.1007/978-3-032-06129-4_36

DO - 10.1007/978-3-032-06129-4_36

M3 - Conference contribution

AN - SCOPUS:105020020385

SN - 9783032061287

T3 - Lecture Notes in Computer Science

SP - 486

EP - 490

BT - Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track and Demo Track - European Conference, ECML PKDD 2025, Proceedings

A2 - Dutra, Inês

A2 - Jorge, Alípio M.

A2 - Soares, Carlos

A2 - Gama, João

A2 - Pechenizkiy, Mykola

A2 - Cortez, Paulo

A2 - Pashami, Sepideh

A2 - Pasquali, Arian

A2 - Moniz, Nuno

A2 - Abreu, Pedro H.

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 15 September 2025 through 19 September 2025

ER -

Morillo P, Terreros A, Ferri C, Hernández-Orallo J. LLM GameLab: An Interactive Platform for Testing Large Language Models in Board Games. In Dutra I, Jorge AM, Soares C, Gama J, Pechenizkiy M, Cortez P, Pashami S, Pasquali A, Moniz N, Abreu PH, editors, Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track and Demo Track - European Conference, ECML PKDD 2025, Proceedings. Springer Science and Business Media Deutschland GmbH. 2026. p. 486-490. (Lecture Notes in Computer Science). doi: 10.1007/978-3-032-06129-4_36