Do We Need ERP Like Before?D365, SAP, or Oracle — Why Building Your Own Intelligent Tool Is the Smarter Move
Your D365 implementation has 100GB of data and reports take hours to run. What if you could ask a question in plain English and get the answer in seconds — with charts? Here's the technology that makes it possible.
The Conversation That Started It All
A client running Dynamics 365 Finance & Operations came to us with a familiar pain: 100GB of transactional data and reports that take hours to generate. Their finance team was spending half their day waiting for reports to load.
Our proposal was straightforward: "We can build you a better product — with just the features you actually use — powered by machine learning and chat-based reporting."
Their question back: "What technology can handle a database that size and still be fast?"
Here's our answer.
The Core Problem
Why ERP Reports Are Slow
OLTP (Transactional)
D365's Azure SQL is optimized for transactional writes — inserting orders, updating inventory, processing payments. Row-by-row operations.
- Reads entire rows when you only need 2 columns
- Aggregations scan millions of rows sequentially
- Complex joins across normalized tables
- Reports compete with live transactions for resources
OLAP (Analytical)
Columnar databases are optimized for analytical reads — aggregating, filtering, and grouping large datasets. Column-by-column operations.
- Reads only the columns you need
- Vectorized processing on compressed data
- Denormalized schema = fewer joins
- Dedicated analytical workload, no contention
The performance difference is not incremental — it's architectural.
Hours→Seconds
Same data. Different database design. 100-1000x improvement.
Database Layer
What to Use for 100GB+ Analytics
You need a columnar/analytical database that can query 100GB+ in seconds, not hours.
ClickHouse
RecommendedOpen-source, columnar, purpose-built for analytical queries. Scans billions of rows in under a second. Handles 100GB trivially, scales to petabytes.
DuckDB
LightweightEmbedded columnar database — no server needed. Think "SQLite for analytics." Perfect for single-tenant or desktop deployments.
Apache Druid / Pinot
Real-timeDesigned for real-time ingestion alongside fast queries. Best when you need live data from the ERP flowing into dashboards.
PostgreSQL + Extensions
FamiliarStay in the Postgres ecosystem with columnar extensions like pg_analytics or Citus. Reasonable performance at 100GB with the familiarity of Postgres.
Architecture
The Full Stack — End to End
From data sources to the end user asking a question in plain English. Here's every layer.
End user asks:
"Show me revenue by region last quarter"
Chat Interface
Outputs:
Python / FastAPI
Semantic Layer
Business terms → tables/columns. Maps "revenue" to the right SQL join. The secret sauce that makes AI queries accurate.
Intelligent Query Engine
Claude API — natural language → SQL → results. With schema context, it generates precise analytical queries.
ClickHouse (Columnar)
100GB+ in seconds — denormalized, query-optimized, partitioned by date
ETL / Sync Engine
Airbyte + Python CDC — incremental sync, schema mapping, change tracking
D365 F&O
OData / SQL
SAP / Oracle
APIs / JDBC
CSV / Excel
File uploads
Custom DB
Postgres / MySQL
Our Pick
The Stack We'd Build On
ClickHouse Cloud
Sweet spot for 100GB-1TB range
Claude API
Natural language → SQL translation
React + Chat UI
Conversational interface with charts
Python (FastAPI)
Orchestrator & semantic layer
Airbyte
ERP → ClickHouse pipeline
Python (Pandas/Polars)
The master of data search
The Real Question
Why Not Just Optimize D365's Database?
You could add better indexing to D365's Azure SQL. You could tune queries. You could throw more compute at it. But you'd be fighting its OLTP design forever.
A dedicated analytical store gives you a 100-1000x query speed improvement and lets you offer a fundamentally different experience — seconds instead of hours. It's not optimization. It's a paradigm shift.
The Case for Custom
Why Building Your Own Tool Is the Move
Today, building a custom solution is faster, cheaper, and better than forcing an ERP to do what it was never designed for.
Speed to Market
With Python, React, and modern databases, you can build the same features D365 offers — in weeks, not months. No vendor lock-in, no licensing maze.
Python Is a Search Master
Python excels at data manipulation, search, and analysis. Pandas, Polars, SQLAlchemy — these tools were built for exactly this kind of work. Your custom tool will search faster than any ERP ever could.
Intelligent Computing from Day One
D365 bolts intelligence on as an afterthought. When you build your own, matrices computing is the core — chat-based queries, intelligent suggestions, anomaly detection built into every interaction.
Better UX, Less Training
ERP interfaces are designed by committee. Your tool is designed for your users. Custom fields, workflows, and dashboards that match how your team actually works — no 200-page training manuals.
100-1000x Faster Reports
Moving from OLTP (Azure SQL) to a columnar analytical database is not an incremental improvement — it is a fundamental architecture change. Reports that took hours now take seconds.
Fraction of the Cost
D365 F&O licenses start at $180/user/month. A custom solution on ClickHouse Cloud + your own frontend costs a fraction of that — and you own the code forever.
Market Gap
Where This Fits
Products like ThoughtSpot, Qlik Sense, and Power BI are in this space, but none offer a true conversational intelligence-first experience tightly integrated with ERP data.
Your edge would be the domain-specific semantic layer — you know D365's schema deeply — combined with a chat-first UX. No dashboards to configure. No report builder to learn. Just ask a question and get the answer.
$180+
D365 F&O per user/month
Hours
Typical report generation
Seconds
With custom ML stack
Ready to Replace Hours with Seconds?
We build custom intelligent reporting tools that sit on top of your ERP data. No rip-and-replace — just a faster, smarter way to get answers.