{"id":20423,"date":"2025-02-26T15:04:03","date_gmt":"2025-02-26T14:04:03","guid":{"rendered":"https:\/\/telecomkh.info\/?p=20423"},"modified":"2025-02-26T15:04:03","modified_gmt":"2025-02-26T14:04:03","slug":"gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms","status":"publish","type":"post","link":"https:\/\/telecomkh.info\/?p=20423","title":{"rendered":"GSMA Open-Telco LLM Benchmarks launches to advance AI in telecoms"},"content":{"rendered":"

New community provides an open-source framework to assess large language models for capability, energy efficiency and safety<\/strong><\/p>\n

The GSMA Foundry, the GSMA\u2019s innovation hub, announced the launch of GSMA Open-Telco LLM Benchmarks, an open-source community aimed at improving the performance of large language models (LLMs) for telecom-specific applications. The community provides an industry-first framework for evaluating AI models in real-world telecom use cases and is supported at launch by Hugging Face, Khalifa University, The Linux Foundation and a host of leading mobile network operators and vendors.
\nAs AI adoption in telecoms accelerates, LLMs have demonstrated significant shortcomings in handling technical telecom knowledge, regulatory compliance and network troubleshooting. In recent tests, GPT4[1] scored less than 75% on TeleQnA[2] [3], a comprehensive dataset tailored to assess the knowledge of LLMs in the field of telecommunications, and less than 40% on 3GPPTdocs Classification[4], a dataset based on 3GPP standards documentation. Microsoft\u2019s Phi2[5], a much smaller model, scored only 10% on MATH500[6] [7], a benchmark of 500 general maths questions.
\nThese results highlight the current limitations of AI models in addressing telecom-specific queries. GSMA Open-Telco LLM Benchmarks will address these gaps by providing transparent, open evaluations of AI models across capabilities, energy efficiency and safety.
\n\u201cToday\u2019s AI models struggle with telecom-specific queries, often producing inaccurate, misleading or impractical recommendations,\u201d said Louis Powell, Head of AI Initiatives, GSMA. \u201cBy creating an industry-wide set of benchmarks, we\u2019re not only improving model performance but also ensuring AI in telecoms is safe, reliable and aligned with real-world operational needs.\u201d
\nThe mobile network operators supporting the launch of GSMA Open-Telco LLM Benchmarks include Deutsche Telekom, LG Uplus, SK Telecom and Turkcell and technology vendor, Huawei.
\nThe GSMA Open-Telco LLM Benchmarks community enables mobile network operators, AI researchers and developers to submit use cases, datasets and models for evaluation. A standardised benchmarking framework ensures that all AI models are evaluated against real-world challenges in areas such as telecoms domain knowledge, mathematical reasoning, energy consumption and safety. The resulting benchmarks will be hosted on Hugging Face to ensure transparency and encourage community engagement.<\/p>\n

Quotes from partners:<\/strong><\/p>\n

Hugging Face<\/strong>
\nJeff Boudier, Head of Product and Growth, Hugging Face, said: \u201cHugging Face is the leading open platform for AI builders, and we\u2019re thrilled to support and host the GSMA Open-Telco LLM Benchmarks to advance telecoms AI adoption and innovation.\u201d<\/p>\n

Khalifa University
\n\u201cAcademia plays a crucial role in advancing AI for telecommunications by ensuring rigorous benchmarking and scientific integrity. At Khalifa University, we are proud to support the GSMA Open-Telco LLM Benchmarks initiative. This effort will drive innovation and enhance the reliability of AI models in real-world telecom applications,\u201d said Prof. Merouane Debbah, Director, 6G Research Center, Khalifa University.<\/p>\n

LG Uplus<\/strong>
\nSangyeob Lee, Chief Technology Officer, LG Uplus, said: \u201cWe stand at a turning point, heading for human and AI agent coexistence, and the telcos will play a vital role in establishing safe, autonomous connections between them. LG Uplus is committed to this AI agent innovation through LLM advancement and welcomes GSMA Open-Telco LLM Benchmarks as our guiding light towards the assured intelligence services that we pursue.\u201d<\/p>\n

The Linux Foundation<\/strong>
\n\u201cThe launch of GSMA\u2019s Open-Telco LLM Benchmarks marks a significant milestone in advancing AI adoption across the telecom industry,\u201d said Arpit Josphipra, General Manager, Networking, Edge and IoT, The Linux Foundation. \u201cBy establishing open, standardised benchmarks, this initiative brings much-needed transparency and performance insights, enabling operators and ecosystem partners to deploy domain-specific AI with confidence. The Linux Foundation supports this effort, as it aligns with our vision of open collaboration to drive innovation and efficiency in telecom networks worldwide.\u201d<\/p>\n

SK Telecom<\/strong>
\nEric Davis, Head of AI Tech Collaboration Office, SK Telecom said: \u201cThe introduction of GSMA Open-Telco LLM Benchmarks marks a pivotal milestone for the telecommunications industry in its pursuit of tangible AI benefits. By establishing a standardised evaluation framework, we\u2019re simultaneously driving innovation and ensuring AI solutions deliver the robustness, reliability and precision that our rapidly evolving sector demands.\u201d<\/p>\n

The launch follows last year\u2019s industry-wide commitment to exploring telco AI use cases ethically and sustainably, central to which was the GSMA\u2019s Responsible AI Maturity Roadmap, which helps MNOs ensure best-practice principles are applied from inception through evolution.<\/p>\n

AI at MWC25 Barcelona<\/strong>
\nThe \u2018Gen AI Summit: Experimentation to Transformation\u2018 at MWC25 Barcelona will feature a range of sessions aimed at exploring the practical applications and transformative potential of generative AI within the telecommunications sector. Key sessions will include discussions on AI-driven network optimisation, personalised customer experiences and the integration of generative AI in 5G and beyond. Renowned speakers such as Harry Singh, Chief Digital Officer at BT; Harrison Lung, Group Chief Strategy Officer at e&; Kaniz Mahdi, Director Technology AWS Industries at Amazon Web Services; and Laurent Leboucher, CTO at Orange; will share their insights and experiences on leveraging AI for industry advancement.
\nOther AI speaker highlights at MWC25 Barcelona include keynote 7, \u2018Tech Game Changers\u2018, where Arthur Mensch, a leading AI researcher and CEO of Mistral AI, will take the stage to discuss the latest developments and the real-world applications that are set to revolutionise the telecoms industry. In addition, keynote 10, \u2018Why AI Agents will Change Everything\u2019, will see Bret Taylor, CEO and co-founder of Sierra and Board member of OpenAI, discuss how AI agents are poised to transform businesses and enterprises.<\/p>\n

Above, Louis Powell, Head of AI Initiatives, GSMA \/ image credited to the GSMA<\/em><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

New community provides an open-source framework to assess large language models for capability, energy efficiency and safety The GSMA Foundry, the GSMA\u2019s innovation hub, announced the launch of GSMA Open-Telco LLM Benchmarks, an open-source community aimed at improving the performance of large language models (LLMs) for telecom-specific applications. The community provides an industry-first framework for … <\/p>\n

Continue reading \u00abGSMA Open-Telco LLM Benchmarks launches to advance AI in telecoms\u00bb<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":20424,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[9,60],"tags":[],"_links":{"self":[{"href":"https:\/\/telecomkh.info\/index.php?rest_route=\/wp\/v2\/posts\/20423"}],"collection":[{"href":"https:\/\/telecomkh.info\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/telecomkh.info\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/telecomkh.info\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/telecomkh.info\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=20423"}],"version-history":[{"count":1,"href":"https:\/\/telecomkh.info\/index.php?rest_route=\/wp\/v2\/posts\/20423\/revisions"}],"predecessor-version":[{"id":20425,"href":"https:\/\/telecomkh.info\/index.php?rest_route=\/wp\/v2\/posts\/20423\/revisions\/20425"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/telecomkh.info\/index.php?rest_route=\/wp\/v2\/media\/20424"}],"wp:attachment":[{"href":"https:\/\/telecomkh.info\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=20423"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/telecomkh.info\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=20423"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/telecomkh.info\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=20423"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}