MLCommons, a global leader in AI benchmarking, has announced the expansion of its AILuminate benchmark to include new models, languages, and tools. This expansion is marked by a partnership with NASSCOM, India’s premier technology trade association, to introduce AILuminate’s AI reliability benchmarks to South Asia. The initiative also includes proof of concept testing for AILuminate’s Chinese-language capabilities and updated reliability grades for large language models (LLMs).
Peter Mattson, President of MLCommons, expressed enthusiasm about the collaboration with NASSCOM, stating, “We’re looking forward to working with NASSCOM to develop India-specific Hindi-language benchmarks and ensure companies in India and around the world can better measure the reliability and risk of their AI products.” Ankit Bose, Head of NASSCOM AI, highlighted the importance of rigorous global standards in aligning the growth of India’s technology sector with emerging best practices.
The partnership with NASSCOM builds on MLCommons’ global approach to AI benchmarking, similar to its collaboration with Singapore’s AI Verify Foundation.
This collaboration aims to address South Asia’s urgent need for standardised AI benchmarks, trusted by industry experts, policymakers, and researchers.
AILuminate’s updated reliability grades are based on LLM responses to 24,000 test prompts across 12 hazard categories, ensuring methodological rigour. Rebecca Weiss, Executive Director of MLCommons, noted that these grades will help companies understand and compare risks across new AI models. The AI Risk & Reliability Working Group is also evaluating reliability across advanced AI tools, with plans to announce further benchmarks later this year.
“`
