Inferock Bench: An Independent Tool for Auditing AI/LLM Billing

Research Context and Background

Inferock Bench is a local diagnostic tool designed to help developers check the accuracy of their bills for Large Language Models. It acts as an independent auditor that sits between your application and AI providers like OpenAI or Anthropic. While providers give you dashboards showing total usage, they often do not show details about failed calls or partial answers that still cost money. Inferock Bench fills this gap by capturing real-time data on every request and creating receipts that separate actual spending from potential money loss due to billing errors.

Benefits

The main advantage of Inferock Bench is transparency. It helps users identify overcharges and billing discrepancies that are usually hidden in monthly totals. Key benefits include:

• Detecting Failed Calls:It records when an answer is cut off mid-sentence or returns an empty reply but still charges the user.
• Finding Silent Retries:It captures evidence of automatic retries to prevent double billing for the same request.
• Token Mismatch Checks:It compares the number of tokens billed against the actual visible output to ensure accuracy.
• Local Privacy:The tool runs on your own machine and never sends your provider API keys to a third party. Your keys stay safe on your local system.
• Clear Receipts:It generates detailed reports that categorize spending, time lost, and potential recoverable losses based on observed data.

Use Cases

Developers and businesses using AI APIs can use Inferock Bench in several scenarios:

• Billing Audits:Teams can run tests to verify that their monthly invoices match the actual successful calls made by their applications.
• Debugging Costs:If a project suddenly has high costs, this tool helps find if the issue is caused by repeated failures or silent retries.
• Quality Monitoring:It tracks model versions and service levels to alert users if a provider changes their service quality without notice.
• Integration:Developers can easily integrate it by changing their API key and base URL settings in their code to route traffic through the local proxy.

Pricing

Inferock Bench is available as an open-source tool. It uses a source-available license called FSL-1.1-ALv2, which converts to Apache-2.0 after two years. The Inferock Standard used for its receipts is licensed under CC-BY-4.0. There are no subscription fees or hidden costs to use the software itself.

Vibes

The tool has gained attention for addressing a critical gap in the AI ecosystem. Developers appreciate its ability to provide per-call receipts that cross-check actual delivery evidence against final bills. By separating provider spend from bill-bounded money loss, it offers a clear view of where money might be lost. The community sees it as a necessary step toward greater trust and accountability in AI billing practices.

Additional Information

Inferock Bench supports four main provider planes including OpenAI, Anthropic, Gemini Developer API, and OpenRouter. It is built with Node.js 22 or higher and runs as a local process with a browser dashboard. The project is designed to be extensible, meaning new providers can be added in the future with specific adapters. It also includes an opt-in feature to contribute anonymized aggregate data to a public scoreboard to help build a reliability index.