China AI Big Model Industrial Application Index Release

Today (4th), the 2024 World Artificial Intelligence Conference and High Level Conference on Artificial Intelligence Global Governance opened at the Shanghai World Expo Center. At the "AI Empowers Industry, Digital Intelligence Enhances the Future" Artificial Intelligence Empowers New Industrialization Theme Forum, the "China AI Big Model Industrial Application Index (2024)" was released.

With the rapid iterative development of artificial intelligence technology, the industrial application scenarios of AI large models continue to enrich. The "China AI Big Model Industrial Application Index" aims to timely track the development trend of AI big model technology at home and abroad, explore potential industrial application scenarios of big models, and systematically evaluate the progress and effectiveness of the integration of general artificial intelligence and industry.

The index system consists of basic application capabilities of large models and industry application capabilities. The basic application capabilities of large models include accuracy and stability in fields such as text and graphics; The industry application capability of large models includes the effectiveness of their application in specific scenarios such as research and development, design, production and manufacturing, and operation and maintenance management in key industries such as civil explosives, electricity, petrochemicals, steel, and pharmaceuticals.

After multiple rounds of evaluation and expert discussions, in terms of basic application capabilities, China has taken the lead in releasing the accuracy and stability index of Chinese AI large model text generation. The accuracy index reflects the accuracy of the first domestic and foreign AI large models in the industrial field text generation. Overall, the accuracy index of domestic large models is 78, while the accuracy index of international large models is 76. The accuracy of top domestic large models in text generation in the industrial field is already competitive; At the level of industry knowledge, China has a certain leading advantage in knowledge reserves in multiple industries such as electricity, building materials, and textiles; At the level of segmentation ability, domestic models are superior to foreign models in terms of sentence ability and professional knowledge mastery, but there is still room for improvement in mathematical and physical abilities. The stability index represents the ability of the model to continuously provide correct answers under certain external disturbances, reflecting the robustness of the model in complex environments. Overall, the stability index of domestic large models is 71 points, lower than the international score of 74 points; At the level of industry knowledge, the knowledge reserves in industries such as power and mining need to be improved; At the level of subdivision ability, domestic large models need to be further strengthened in areas such as sequential transformation and logical perturbation.

In terms of industry application capabilities, we are the first to release the China AI Big Model Civil Explosive Industry Application Index, and conduct application evaluations for big models in safety and control scenarios in the civil explosive industry. Overall, the application index of the domestic large model industry is 84 points, while the international score is 93 points. Foreign large models still have a leading advantage; In terms of scene segmentation ability, there is a certain gap between domestic models and foreign models in personnel behavior monitoring, accident monitoring, and dangerous vehicle monitoring segmentation scenarios, and there is still significant room for improvement in domestic models.