Cli Modelarium

Cli Modelarium is a command-line tool designed to help developers compare different Large Language Models. It allows users to test multiple AI models at the same time from a single terminal window. This tool is built for people who want to evaluate which AI model works best for their specific needs without getting lost in complex setup steps. It supports major cloud providers like OpenAI and Anthropic as well as local models running on your own computer.

Benefits

Cli Modelarium offers several key advantages for anyone working with AI models. It provides live parallel streaming so users can see all models generate responses simultaneously. This feature includes real-time tracking of speed and cost to help users make informed decisions quickly. The tool also includes statistical analysis features that run multiple tests to ensure results are reliable and not just luck. Users can use LLM-as-a-judge to let one AI model score the quality of outputs from other models. It supports deterministic assertions which means users can set specific rules to pass or fail a test automatically. This is useful for building automated quality checks into software development pipelines. The tool handles security well by storing API keys in the operating system keychain rather than saving them as plain text files.

Use Cases

This tool is ideal for developers, data scientists, and product managers who need to benchmark AI models. It works well for comparing different cloud providers to find the best price and performance combination. Teams can use it to test how different temperature settings affect the creativity or accuracy of AI responses. It is also useful for A/B testing different system prompts to see which one produces better results. Organizations can run batch evaluations to test many prompts against many models at once. The tool integrates easily into continuous integration and continuous deployment systems to ensure AI features meet quality standards before they go live. It is also helpful for comparing local models against cloud services to understand performance gaps.

Pricing

Cli Modelarium is open source and available under the Apache 2.0 license. This means it is free to download and use for personal or commercial projects. Users do not need to pay a subscription fee to use the tool itself. However, using the tool with cloud providers like OpenAI or Anthropic will incur costs based on the API usage of those services. The tool itself does not charge for the comparison process.

Vibes

The tool is designed by a developer who focuses on terminal-first workflows, which suggests a preference for simplicity and efficiency. The documentation indicates that the project welcomes issues and pull requests from the community, showing an active development approach. Users who value command-line tools and statistical rigor will likely appreciate the detailed features for reproducibility and significance testing. The tool is described as polished and capable of handling complex comparisons from a single command.

Additional Information

Cli Modelarium requires Python 3.11 or higher to run. It supports macOS, Windows, and Linux systems. The tool stores API keys securely using the native keychain system on each operating platform. It supports eight major cloud providers including OpenAI, Anthropic, Google, xAI, DeepSeek, Mistral, Groq, and OpenRouter. Users can also connect to local models through Ollama, LM Studio, vLLM, or llama.cpp. The tool was verified to support specific models as of May 25, 2026. It is not intended for production-scale routing or load balancing but is perfect for evaluation and comparison tasks. The project is hosted on GitHub and the developer is available for collaboration through LinkedIn.