CUHK
News Centre

The Chinese University of Hong Kong (CUHK) today (30 October) announced the launch of CLEVA-Cantonese, the world’s first dynamic evaluation platform and ecosystem dedicated to the Cantonese language. Cantonese is a vital language for communities in Hong Kong, Guangdong and other Cantonese-speaking regions. This pioneering platform delivers fair, dynamic, informative benchmarking that reveals how well various large language models (LLMs) support Cantonese. It provides researchers and developers with meaningful insights to accelerate the improvement and real-world application of Cantonese-capable LLMs.

This project is a collaboration between CUHK’s InnoHK Centre for Perceptual and Interactive Intelligence (CPII) and the CUHK Language and Vision (LaVi) Lab. It is co-led by Professor Helen Meng Mei-ling, Patrick Huen Wing Ming Professor of Systems Engineering and Engineering Management and Director of CPII, and Professor Wang Liwei, Assistant Professor in the Department of Computer Science and Engineering at CUHK, Leader of the LaVi Lab and CLEVA project leader.

An evolving ecosystem for Cantonese LLM evaluation

CLEVA (Chinese Language Models EVAluation Platform), developed by CUHK’s LaVi Lab, is widely recognised as one of the largest and most comprehensive evaluation benchmarks for Mandarin Chinese LLMs. Building upon this foundation, CLEVA-Cantonese establishes the world’s first evolving ecosystem for Cantonese LLM evaluation. It integrates a collaborative, automated workflow that cycles through four key phases: data import and filtering, language model understanding, evaluation, and feedback. This continuous process provides timely insights to guide LLM innovation, improves services for Cantonese-speaking populations and generates research outcomes that can assist in the evaluation of other low-resource languages.

Cantonese evaluation for LLMs is crucial, as it provides clear performance signals that pinpoint model strengths and areas for improvement, thereby accelerating their development. It also enables scalable, timely assessment that keeps pace with rapid model iteration cycles, while ensuring trustworthy comparisons through standardised tasks, prompts and multi-metric evaluations.

CLEVA-Cantonese is built to meet the special challenges of creating a high-quality Cantonese benchmark:

• It is capable to evaluate written vernacular Cantonese (粵語白話文) – the written form of everyday spoken Cantonese – capturing unique linguistic traits such as colloquial expressions and slang, code-switching with English and Mandarin, and romanisation in the form of Jyutping (粵拼).
• CLEVA-Cantonese standardises the end-to-end workflow for evaluation, including constructing representative tasks with up-to-date data, evaluating LLMs using consistent prompts and selecting a suite of informative metrics.
• Through collaboration with data providers such as Phoenix TV, CLEVA-Cantonese continuously adopts the latest data, which naturally reflects emerging language trends in Cantonese and mitigates data contamination.

Professor Wang said: “We utilise natural language understanding technology based on LLMs to assist in constructing a series of multidimensional evaluation tasks. These tasks are designed around linguistic features, ensuring the benchmark faithfully reflects the language’s structural and knowledge-based characteristics. CLEVA-Cantonese marks the beginning of an ecosystem that brings together academic research, data contributors and state-of-the-art model developers to drive LLM advancement across languages, with immediate benefits for Cantonese-speaking communities.”

Early findings and the continuous improvement loop

The CLEVA-Cantonese team has completed an initial round of evaluation with a range of international and domestic LLMs, spanning open-source and proprietary models. The findings show that even the latest models still struggle to fully capture the nuances of Cantonese, leaving substantial room for improvement in grammar, pronunciation and vocabulary. These insights will guide the next generation of LLMs, enhancing their alignment with Cantonese and performance in related tasks. As stronger models emerge, CLEVA-Cantonese will iteratively refine its evaluation criteria – completing the continuous cycle of data import, language model understanding, evaluation and feedback.

Professor Meng concluded: “Building upon CUHK’s interdisciplinary expertise, we will continuously refresh the benchmark through expanded data partnerships, develop an open evaluation platform for researchers, developers and institutions, extend CLEVA-Cantonese to support more languages, tasks and spoken Cantonese, and provide shared tools to advance collaborative research across linguistics, education, culture and related domains. CLEVA-Cantonese elevates evaluation to a systematic process. It makes gaps for improvement visible, guides research and product roadmaps, and helps ensure Cantonese is well supported across areas such as education, healthcare, public services and cultural life.”