MacroBench: A Novel Testbed for Web Automation Scripts via Large Language Models

Research Context and Background

MacroBench is a specialized testing tool designed to check how well artificial intelligence models can create browser automation scripts. It works by asking these models to read the structure of a website and then write code to perform specific actions, like clicking buttons or filling out forms. The project was developed by Hyunjun Kim and Sejong Kim and was accepted for presentation at the NeurIPS 2025 Workshop on Lock-LLM. This tool helps developers and researchers understand the current capabilities and limitations of AI when it comes to automating web tasks.

Benefits

MacroBench offers several key advantages for anyone working with AI and web automation. First, it provides a standardized way to test different AI models. Instead of guessing if an AI can do a job, users can run it through MacroBench to see exact success rates. The tool uses six fake websites that mimic real platforms like TikTok, Reddit, and Facebook. These sites contain over 680 different tasks ranging from simple clicks to complex workflows. This variety ensures that tests are thorough and not just easy wins for the AI.

The benchmark also focuses on safety and ethics. It includes specific tests to see if an AI will try to steal data or spam users. All testing happens in a safe, isolated environment, so no real user data is ever at risk. Additionally, the tool provides detailed reports on why an AI might fail. It breaks down errors into categories like syntax mistakes, timing issues, or logic problems. This helps developers fix their AI models more effectively.

Use Cases

MacroBench is useful for researchers, developers, and companies building AI tools. Researchers can use it to compare different AI models and see which ones are best at writing code. Developers can use the tool to test their own AI agents before releasing them to the public. It helps ensure that the automation scripts generated by the AI are reliable and safe.

The tool is also great for educational purposes. It can be used in classrooms to teach students about web automation and the challenges of getting AI to write correct code. The detailed breakdown of task complexity helps learners understand the difference between simple tasks and those that require advanced logic. Furthermore, the synthetic nature of the websites means anyone can run the tests without needing access to real social media platforms or worrying about breaking real user accounts.

Pricing

MacroBench is an open-source project and is available for free. The code and documentation are hosted on GitHub under an MIT License. This means anyone can download, use, and modify the tool without paying any fees. Users do need their own API keys to access the AI models being tested, but the benchmark infrastructure itself is free to use.

Vibes

The public reception of MacroBench has been positive within the technical community. The project was accepted at a major academic workshop, which signals that experts in the field find it valuable. The high success rates reported in the initial tests, with some models achieving over 96 percent success on simple tasks, show promise for the future of AI automation. However, the results also highlight a gap between what AI can do and what is needed for production use. No model managed to write perfect, maintainable code in every case. This honest assessment is seen as a strength of the benchmark because it sets realistic expectations for developers.

Additional Information

MacroBench was created by Hyunjun Kim and Sejong Kim. The project emphasizes reproducibility, meaning other researchers can get the same results by following the same steps. It uses fixed seeds and frozen container images to ensure consistency. The repository includes all the necessary files, including the task datasets, the execution engine, and the results from testing over 2,600 model-task combinations. The project is licensed under the MIT License, allowing for broad adoption and contribution by the community.