OpenAI shares playbook for trustworthy AI evaluations

Key Takeaways

OpenAI shares playbook for trustworthy AI evaluations

OpenAI has shared a playbook for trustworthy third-party evaluations of frontier AI models. These evaluations assess the safety and capabilities of advanced AI models. OpenAI suggests that effective evaluation reports should clearly state the claim being tested and provide evidence that the results are valid. The goal is to help inform emerging standards for AI evaluations.

OpenAI's AI evaluation guidelines

OpenAI has outlined guidelines for evaluating AI models. The guidelines emphasize the importance of a 'harness' - the environment and setup that facilitates an AI's actions. The harness significantly influences how a model performs. OpenAI suggests that evaluation reports should clearly articulate the specific claim being tested and provide evidence supporting the validity of the results.

Meet mKernel for faster AI communication

Researchers have developed mKernel, a library of persistent CUDA kernels for faster GPU-driven communication. mKernel fuses intra-node NVLink communication, inter-node RDMA, and compute into a single kernel. This can improve the performance of AI workloads by reducing communication overhead.

ByteDance developing AI chips like Nvidia partner Groq

ByteDance, the owner of TikTok, is developing a new AI chip similar to those from Nvidia partner Groq. The chip is intended to run AI models and reduce ByteDance's reliance on foreign chip suppliers. The move is part of a broader effort by Chinese tech companies to develop their own AI capabilities.

Illinois to regulate AI with new safety bill

The governor of Illinois has vowed to sign a bill regulating AI. The Artificial Intelligence Safety Measures Act requires large frontier developers to create a framework addressing risks and governance. It also mandates transparency reports, annual third-party audits, and critical safety incident reporting.

CODA warns tech companies about AI copyright infringement

The Content Overseas Distribution Association (CODA) has warned tech companies about AI copyright infringement. CODA argues that current AI services may constitute infringement of copyrights by producing imagery and video similar to existing Japanese content.

AI chatbots may not deliver accurate health responses

A study found that AI chatbots do not consistently deliver accurate health responses. The study evaluated the responses of AI chatbots to everyday medical questions and found that nearly 76% of responses were accurate. However, significant differences were found across medical specialties.

OpenAI Agents SDK for building AI agents

OpenAI has introduced the Agents SDK, a tool for building AI agents. The SDK provides a model-native harness for enhanced functionality and allows developers to create more capable and adaptable AI agents.

Jensen and Dario debate AI chip exports

Anthropic's CEO Dario Amodei and Nvidia's CEO Jensen Huang have differing opinions on AI chip exports. Dario argues that US export controls could give American companies a 12-24 month lead in frontier AI capabilities. Jensen counters that Chinese chips are already competitive and that selling Nvidia chips to China would accelerate their capabilities.

UN seeks global AI governance

The United Nations is seeking to govern AI globally. The UN's Global Dialogue on AI Governance aims to shape the global governance landscape and address issues of legitimacy and inclusivity in AI governance.

Sources

• A shared playbook for trustworthy third party evaluations
• OpenAI's Playbook for AI Evaluation
• Meet mKernel: A Multi-GPU, Multi-Node Fused Kernel Library for GPU-Driven Communication
• China’s ByteDance Developing New AI Chips Like Those from Nvidia Partner Groq
• Illinois Governor Vows to Sign AI Safety Bill
• Use of Japanese anime and manga for AI training is a "serious issue that undermines creators' rights," CODA warns tech companies
• AI Chatbots Do Not Consistently Deliver Accurate Health Responses
• OpenAI Agents SDK: Building with Model-Native Harnesses
• Jensen And Dario Square Off On AI Chip Exports
• The World Is Trying to Govern AI. The UN Wants In. | Council on Foreign Relations