Evaluating CxG Generalisation in LLMs via Construction-Based NLI Fine Tuning
By: Tom Mackintosh, Harish Tayyar Madabushi, Claire Bonial
Potential Business Impact:
Helps computers understand sentence structure better.
We probe large language models' ability to learn deep form-meaning mappings as defined by construction grammars. We introduce the ConTest-NLI benchmark of 80k sentences covering eight English constructions from highly lexicalized to highly schematic. Our pipeline generates diverse synthetic NLI triples via templating and the application of a model-in-the-loop filter. This provides aspects of human validation to ensure challenge and label reliability. Zero-shot tests on leading LLMs reveal a 24% drop in accuracy between naturalistic (88%) and adversarial data (64%), with schematic patterns proving hardest. Fine-tuning on a subset of ConTest-NLI yields up to 9% improvement, yet our results highlight persistent abstraction gaps in current LLMs and offer a scalable framework for evaluating construction-informed learning.
Similar Papers
NeurIPS 2023 LLM Efficiency Fine-tuning Competition
Computation and Language
Makes AI smarter by cleaning its learning data.
MERGE: Minimal Expression-Replacement GEneralization Test for Natural Language Inference
Computation and Language
Makes AI understand sentences better, even when words change.
Enhancing NLP Robustness and Generalization through LLM-Generated Contrast Sets: A Scalable Framework for Systematic Evaluation and Adversarial Training
Computation and Language
Makes AI understand language better, even tricky parts.