On Tuesday, Naver Cloud revealed that its HyperCLOVA X has surpassed both OpenAI and Google’s generative AI models in the Korean AI performance evaluation system, KMMLU.
Naver is aiming for dominance in performance competitiveness within the Korean AI landscape to be a leader in sovereign AI technologies tailored to uphold national data sovereignty and adhere to local regulations.
KMMLU, established by the prominent open-source language model research team HAE-RAE in Korea, serves as a crucial performance evaluation metric for AI technologies.
Comprising 35,030 questions across 45 diverse domains, including humanities, social sciences, and science & technology, the KMMLU evaluation sets a high standard for assessing expert-level knowledge.
Roughly 80% of the questions in KMMLU gauge broad knowledge applicable globally, including skills like mathematical reasoning. The remaining 20% focus on testing the AI’s proficiency in solving Korea-specific issues, such as understanding the geography of the Korean Peninsula and knowledge of domestic laws.
By presenting questions in Korean, KMMLU ensures a more accurate assessment of an AI’s understanding of the language, enabling a comprehensive evaluation of universal capabilities and local knowledge crucial for effectively serving Korean users.
This approach contrasts with the challenges faced by North American tech giants like OpenAI and Google when adapting benchmarks like ‘MMLU’ to Korean, as translation inaccuracies and cultural nuances inherent in questions originally designed for English contexts often lead to imprecise assessments of AI proficiency in Korean.
Based on findings from the KMMLU research paper, HyperCLOVA X demonstrated superior performance compared to OpenAI’s GPT-3.5-Turbo and Google’s Gemini-Pro, with competitive scores across both general and Korea-specific knowledge assessments.
While it didn’t surpass OpenAI’s GPT-4 overall, HyperCLOVA X excelled in Korea-specific knowledge, making it a strong contender in sectors reliant on localized insights, such as education and legal services.
Leveraging the success validated by KMMLU, Naver Cloud will advance HyperCLOVA X into a Sovereign AI solution characterized by robust security measures and exceptional performance.
Taking advantage of HyperCLOVA X’s impressive performance, as demonstrated in the KMMLU assessments, Naver Cloud plans to further develop it into a sovereign AI solution renowned for its top-notch security and efficiency.
Last October, Naver Cloud introduced Neurocloud for HyperCLOVA X, a hybrid cloud service tailored to facilitate its usage within secure, private networks, thereby reducing the risks associated with data breaches. The company aims to unveil additional enterprise solutions to cater to evolving industry needs.
Sung Nako, head of the Hyperscale AI division overseeing HyperCLOVA X, highlighted, “HyperCLOVA X, positioned as a sovereign AI, seamlessly integrates global knowledge with the capacity to tackle Korea-specific challenges.”
Also Read:
- Samsung Joins Global AI-RAN Alliance to Spearhead 6G Technology
- LG Electronics Partners with Josun Hotels & Resorts to Develop Specialized Hospitality Robots
- Mark Zuckerberg’s South Korea Trip: Meetings with Samsung and President
- Naver’s Webtoon Entertainment Eyes June US IPO with Goldman Sachs, Morgan Stanley
- South Korean Telcos Set to Unveil AI Breakthroughs at MWC 2024
