Nvidia's GH200 chip outperforms competitors with 7,167 tok/s

Key Takeaways

• Cumulus Labs' Ion inference engine achieves 7,167 tok/s on Nvidia's GH200 chip.
• Amazon SageMaker introduces detailed metrics and Insights dashboard for AI inference monitoring.
• Anthropic faces backlash over US restrictions on its AI models.
• OpenAI's beneficial trait training makes AI models safer and harder to manipulate.
• BlackFog expands ADX Vision shadow AI security to macOS.
• Salesforce offers CodeGen for generating, validating, and reranking Python functions.
• Rights groups criticize UK Home Office's AI-powered facial age estimation for asylum seekers.
• Students praise AI classroom experience, while Cal State faculty push back against AI replacement.
• Operators in the apartment industry focus on AI operations, control, and governance.
• US restrictions on frontier AI spark global debate.

Cumulus Labs boosts AI inference with custom CUDA kernels

Cumulus Labs has developed Ion, a C++ inference engine with custom CUDA kernels for NVIDIA's GH200 Grace Hopper architecture. This engine achieves 7,167 tok/s on a single GH200 chip and 12.5-second cold starts, outperforming competitors. The technology uses coherent CUDA graphs and eager KV writeback for efficient performance.

SageMaker adds detailed metrics for AI inference monitoring

Amazon SageMaker now offers detailed metrics and an Insights dashboard on CloudWatch for monitoring generative AI inference endpoints. The feature helps diagnose issues like GPU memory pressure, KV cache saturation, and traffic imbalance across Availability Zones.

AI accountability in apartment industry

Operators in the apartment industry are shifting focus from AI automation to operations, control, and governance. Structured data, processes, and ownership of tools are crucial for effective AI adoption. Experts stress the importance of training, guardrails, and regular data review.

Rights groups criticize UK Home Office's AI age estimation for asylum seekers

Over 60 rights groups urge the UK government to scrap AI-powered facial age estimation for asylum seekers, citing bias, inaccuracy, and potential unlawfulness. The technology's performance varies by ethnicity and skin tone, raising concerns about its reliability.

US ban on Anthropic AI sparks global backlash

The US restrictions on Anthropic's AI models spark debate on frontier AI versus smaller models in cybersecurity. ISMG editors discuss the impact on the global AI landscape and emerging themes from upcoming roundtables.

OpenAI's beneficial trait training makes AI safer and more helpful

OpenAI researchers show that small doses of beneficial trait training make AI models broadly safer and harder to manipulate. This approach differs from Anthropic's constitutional method, relying on empirically measurable behavioral traits.

BlackFog expands ADX Vision shadow AI security to macOS

BlackFog launches ADX Vision for macOS, extending its shadow AI security product to Apple devices. The software helps prevent sensitive information from leaving a business through unsanctioned large language models.

Students praise AI classroom experience

Students in Professor Scott Dyreng's AI class at Fuqua share their positive experience with the AI classroom, citing improved teamwork and engagement. They disagree with criticisms of surveillance and psychological safety concerns.

Cal State faculty push back against AI replacement

The California State University faculty union is contesting the use of AI tools to replace professors, citing concerns about job security and academic integrity. The union is backing legislation to prevent AI from replacing faculty.

Salesforce CodeGen tutorial: Generate, Validate, and Rerank Python Functions

This tutorial demonstrates how to use Salesforce CodeGen to generate, validate, and rerank Python functions with unit tests and safety checks. The guide covers loading the model, preparing it for code generation, and using it for real-world applications.

Sources

• Claude's Corner: Cumulus Labs — When the Inference Market Gets Outclassed by CUDA Kernels
• Monitor and debug generative AI inference with SageMaker detailed metrics and Insights dashboard on CloudWatch
• Apartmentalize Special Report: Why AI Adoption Needs Accountability
• Rights groups brand Home Office's AI age guesser for asylum-seekers as biased and inaccurate
• ISMG Editors: Cyber Backlash Over the US Ban on Anthropic AI
• OpenAI researchers show small doses of "beneficial trait" training make AI models broadly safer and harder to manipulate
• BlackFog expands ADX Vision shadow AI security to macOS
• Scott Dyreng’s AI classroom: Actually, we liked it.
• Cal State faculty push to prevent AI tools from replacing them as schools and staff experiment
• Salesforce CodeGen Tutorial: Generate, Validate, and Rerank Python Functions With Unit Tests and Safety Checks