3.1. Parquet & Delta Tables
Pricing teams are already familiar with SQL for querying relational databases, aggregating metrics, and producing reports. For most insurance pricing datasets - often a few million rows at most - SQL is fast, reliable, and entirely sufficient for analysis.
While Parquet and Delta tables offer modern features like columnar storage, versioning, and ACID transactions, in practice many pricing teams benefit most from keeping data in their existing SQL databases.
SQL in Pricing Teams
SQL is the backbone of pricing analytics. Typical workflows include:
- Extracting and joining data from multiple source tables.
- Aggregating metrics such as premiums, losses, and exposures.
- Feeding reports or models with clean, structured data.
Because datasets are generally manageable in size, queries run quickly, and analyses can be easily shared or reused. Most teams already have robust SQL databases, so storing derived datasets or aggregated features back into SQL is often simpler and easier to manage than adopting a new storage format.
Parquet and Delta Tables
Parquet is a columnar file format optimized for analytics. Its main advantages include efficient storage, compression, and fast reads for analytical workloads.
Delta tables extend Parquet with features like ACID transactions, time travel, and schema evolution. They are useful when multiple teams need to collaborate on the same dataset, or when you need reliable versioning and updates.
In practice for pricing teams:
- If your workflow involves multiple analysts updating or merging datasets programmatically, Delta tables can add safety and reproducibility.
- For read-only analytical datasets or intermediate results, Parquet can offer faster I/O and smaller file sizes.
- For most workflows, however, the simplest and most familiar approach is to keep data in SQL - query it, update it, and store intermediate results in the same database.
When to Use SQL, Parquet, or Delta
| Use Case | SQL | Parquet | Delta Table |
|---|---|---|---|
| Ad-hoc queries | ✅ | ✅ (via external tools) | ✅ (via external tools) |
| Aggregations / feature prep | ✅ | ✅ | ✅ |
| Updates / inserts | ✅ | ❌ | ✅ |
| Multiple analysts / versioning | ⚠️ (manual processes) | ⚠️ | ✅ (transactional + versioned) |
| Large-scale analytics | ⚠️ (>100M rows) | ✅ | ✅ |
Rule of thumb:
- Use SQL for most day-to-day analysis and feature preparation.
- Consider Parquet for large analytical pipelines or to share datasets outside SQL.
- Use Delta tables only if you need versioning, collaborative updates, or transactional guarantees.
Summary
SQL remains the primary tool for pricing teams because it is fast, reliable, and already integrated into existing workflows. Parquet and Delta tables can be useful in specific scenarios, but for most insurance pricing datasets, simply querying data from and storing results back into the SQL database is the simplest and most practical approach.