Datu AI analyst open-source

Datu AI Analyst Open-Source

Datu Core is an AI-powered tool that makes it easy for anyone to analyze and visualize data using plain English. You don't need to be a data expert to use it. Datu Core connects to various data platforms and performs tasks that usually require data analysts or data scientists. It helps identify data quality issues, models data based on user requests, and visualizes data to understand Key Performance Indicators (KPIs).

Benefits

• Easy to Use:No technical expertise needed. Just use plain English to get insights from your data.
• Quick Setup:Install and run Datu Core in minutes.
• Connects to Multiple Data Sources:Works with various data platforms.
• Identifies Data Quality Issues:Helps ensure your data is accurate and reliable.
• Visualizes Data:Creates charts and graphs to help you understand your data better.

Use Cases

• Business Analysis:Understand your business performance with KPI visualizations.
• Data Quality Checks:Ensure your data is clean and accurate.
• Data Transformation:Convert raw data into useful formats.
• Data Exploration:Explore your data to find insights.
• Reporting:Create dashboards to share insights with stakeholders.

Installation

To install Datu Core, you need Python 3.11 or later. Follow these steps:1. Create and activate a virtual environment:

python -m venv .venvsource .venv/bin/activate # On Windows use: .venv\\Scripts\\activate

2. Install Datu Core:

pip install 'datu-core[all]'

Running the Application

To run the application, simply type:

datu

Connect to Data Source

Datu Core fetches the schema listed in theprofiles.ymlfile to avoid fetching the schema every time. It works on the target data source that is selected. Here is an example structure ofprofiles.yml:

datu_demo:target: dev-postgres # Target is used to select the datasource that is currently active. Change this if you would like to use a different datasource.outputs:dev-postgres:type: postgreshost: "{{ env_var('DB_HOST', 'localhost') }}" # if a environment variable is supplied that gets priority. This is useful for not hardcoding.port: 5432user: postgrespassword: postgresdbname: my_sap_bronz

Features

Datu Core offers several advanced features:*Dynamic Schema Discovery & Caching:Automatically introspects the target database schema and caches the discovered metadata.*LLM Integration for SQL Generation:Uses OpenAI's API (e.g., GPT-4o-mini) to generate SQL queries that transform raw (Silver) data into a Gold layer format. The system prompt includes a concise summary of the schema to help the LLM generate valid queries.*Transformation Preview:The generated SQL is previewed by executing a sample query (with a LIMIT) and displaying the result in a formatted HTML table.*Persistent View Creation:Users can review the transformation preview and then create a view in the Gold layer. This view automatically reflects updates from the underlying Bronze data.*CSV Download:Users can download the full result of the transformation as a CSV file.*User-Friendly Chat Interface:The frontend features a ChatGPT-like interface with persistent conversation state, syntax highlighting for code blocks, and copy-to-clipboard functionality.*CSV Upload:Upload data as CSV files, in addition to or instead of connecting to a database.*Visualizations:Create bar, line, area, scatter, pie, or KPI visualizations to explore your data.*Data Catalog:View automatically generated business definitions for your fields.*Dashboards:Build dashboards with multiple KPIs to share insights with stakeholders.

Documentation

For detailed guidance and examples, explore the official documentation.

Contributing

We welcome contributions! See the contributing guidelines for details on:* Reporting bugs and features* Development setup* Contributing via Pull Requests* Code of Conduct* Reporting of security issues

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.

About

Datu Core AI Analyst is an open-source project designed to simplify data analysis and visualization. It leverages advanced AI technologies to provide users with powerful tools to understand and act on their data efficiently.