ffastdb_analytics

Powerful analytics for your ffastdb database — group, rank, pivot, and compute statistics directly inside your Dart or Flutter app, with no server, no network, and no SQL required.

---

What can I do with it?

I want to…	Use
Sum / average / count records by category	groupBy
Keep, sort and limit groups (top 5 categories with ≥ 10 orders)	groupBy(having:, orderBy:, limit:)
Group by several fields, by date buckets or by a computed value	groupByKeys
Chart totals per day / week / month / quarter / year	timeSeries
Get count, mean, stddev, min/max and quartiles in one call	describe
Find the top-N items by a value	topN
Rank items with tie support	rank
Find the few items that make most of the total (80/20, ABC)	pareto
Compute moving averages, sums, min/max, stddev or medians	rolling
Build a running total	cumulativeSum
Measure growth period over period	change
Generate a spreadsheet-style pivot table with totals	pivot / pivotBy
Calculate a percentile (e.g. p95 latency)	percentile
Measure spread with standard deviation	stddev
See how values are distributed	histogram / valueCounts
Check whether two fields move together / fit a trend line	correlation / regression
Spot unusual values	outliers
Keep a dashboard live as data changes	watch
Narrow down results before analysing	where
Read nested fields	'customer.country', 'items.0.price'
Average a price weighted by quantity	aggWeightedAvg
Add margins, ratios or each group's share of the total	groupBy(derived:)
Aggregate "the last 7 days" instead of "the last 7 rows"	rolling(period:)
Keep the UI smooth with big collections	Automatic — nothing to do (details)

Contents: Installation · Quick start · Aggregations · Time series · Ranking · Window functions · Pivot tables · Statistics · Live dashboards · Performance · Full example · Behavior and edge cases · How it works · Roadmap

---

Installation

Add to your pubspec.yaml:

dependencies:
  ffastdb_analytics: ^0.2.0

Then run:

dart pub get

---

Quick start

import 'package:ffastdb/ffastdb.dart';
import 'package:ffastdb_analytics/ffastdb_analytics.dart';

final db = await FfastDb.init(MemoryStorageStrategy());

// Insert some documents
await db.insertMany([
  {'category': 'Food',  'amount': 120.0, 'status': 'active'},
  {'category': 'Food',  'amount':  80.0, 'status': 'active'},
  {'category': 'Tech',  'amount': 500.0, 'status': 'inactive'},
]);

// Analyse the whole collection
final summary = await db.analytics.all.groupBy('category', {
  'revenue':    aggSum('amount'),
  'orders':     aggCount(),
  'avg_ticket': aggAvg('amount'),
});
// → [{key: Food, revenue: 200.0, orders: 2, avg_ticket: 100.0}, ...]

// Or filter first, then analyse
final activeOnly = await db.analytics
    .where((q) => q.where('status').equals('active').findIds())
    .groupBy('category', {'total': aggSum('amount')});

Tip: Every analytical method works the same way whether you use .all or .where(...) — just pick the scope you need.

---

Aggregations

Use groupBy to split documents into groups and compute values for each group. Pass any combination of aggregation expressions:

Expression	Description
aggSum('field')	Total of a numeric field
aggAvg('field')	Average of a numeric field
aggCount()	Number of documents in the group
aggMin('field')	Smallest value
aggMax('field')	Largest value
aggVariance('field')	Population variance
aggStdDev('field')	Population standard deviation
aggMedian('field')	Median value
aggPercentile('field', 0.9)	Value at a given rank
aggCountDistinct('field')	Number of distinct values
aggFirst('field') / aggLast('field')	First / last value in storage order
aggMode('field')	Most frequent value
aggWeightedAvg('price', weight: 'qty')	sum(price × qty) / sum(qty)
aggCustom((docs) => ...)	Your own function over the group's documents

aggVariance and aggStdDev accept sample: true for the sample statistic (divides by n − 1).

Every field parameter — in aggregations, keys, windows, pivots and statistics — accepts a dotted path into nested maps or lists: 'customer.country', 'items.0.price'.

aggCount() counts documents, aggSum returns 0 when a group has no numeric values, and the other aggregations return null in that case. Mixed value types never throw.

final result = await db.analytics.all.groupBy('department', {
  'headcount': aggCount(),
  'total_salary': aggSum('salary'),
  'avg_salary': aggAvg('salary'),
  'min_salary': aggMin('salary'),
  'max_salary': aggMax('salary'),
});

Conditional aggregations

Restrict any aggregation to some documents with .filter(...) — like SQL FILTER (WHERE ...). One pass, several slices:

final perMonth = await db.analytics.all.groupBy('month', {
  'income':   aggSum('amount').filter((d) => d['type'] == 'INCOME'),
  'expense':  aggSum('amount').filter((d) => d['type'] == 'EXPENSE'),
  'refunds':  aggCount().filter((d) => d['refunded'] == true),
});

Filter, sort and limit groups

// Top 5 categories by revenue, only those with at least 10 orders
final top = await db.analytics.all.groupBy(
  'category',
  {'revenue': aggSum('amount'), 'orders': aggCount()},
  having: (g) => g.count >= 10,
  orderBy: 'revenue',      // an aggregation name, 'key' or 'count'
  descending: true,
  limit: 5,
);

Derived metrics

Compute metrics from each group's aggregations — margins, ratios, shares of the total. Each derived metric receives the group and the same aggregations over every document in scope, and can be used by having, orderBy and limit:

final categories = await db.analytics.all.groupBy(
  'category',
  {'revenue': aggSum('amount'), 'profit': aggSum('profit')},
  derived: {
    'margin': metricRatio('profit', 'revenue'),   // profit / revenue
    'share':  metricShare('revenue'),             // revenue / total revenue
    'vsAvg':  (g, total) => g['revenue'] - total['revenue'] / 12,
  },
  orderBy: 'margin',
  descending: true,
);
print(categories.first['margin']);   // g['name'] reads any aggregation

Several keys, dates and computed keys

groupByKeys takes a list of GroupKeys. g.key is the value for a single key or a list for several; g.keys maps each key name to its value.

final groups = await db.analytics.all.groupByKeys(
  [
    GroupKey.date('date', DateBucket.month),   // hour | day | week | month | quarter | year
    GroupKey.field('category'),
    GroupKey.computed('size', (d) => d['amount'] > 100 ? 'big' : 'small'),
  ],
  {'total': aggSum('amount')},
  orderBy: 'key',
);

for (final g in groups) {
  print('${DateBucket.month.label(g.keys['date'])} '   // 2026-03
        '${g.keys['category']} ${g.keys['size']}: ${g.aggregations['total']}');
}

Date fields may hold a DateTime, an ISO-8601 string or milliseconds since the epoch. Weeks start on Monday (ISO 8601).

Time series

timeSeries returns one row per bucket in chronological order and fills the gaps, so charts never skip a month:

final monthly = await db.analytics.all.timeSeries(
  'date',
  DateBucket.month,
  {'revenue': aggSum('amount'), 'orders': aggCount()},
  from: DateTime(2026, 1, 1),   // optional
  to:   DateTime(2026, 12, 31), // optional
);
// Empty months → count 0, sum 0, other aggregations null.

---

Ranking

Top-N

Get the highest (or lowest) N documents by any field:

// Top 5 products by revenue
final top5 = await db.analytics.all.topN('revenue', n: 5);

// Bottom 3 (ascending)
final bottom3 = await db.analytics.all.topN('revenue', n: 3, ascending: true);

Rank

Assign a rank to every document (standard competition ranking). Ties share the same rank, and results come back in storage order:

final ranked = await db.analytics.all.rank('score');
// scores [100, 100, 80] → ranks [1, 1, 3]

for (final r in ranked) {
  print('#${r.rank}  ${r.document['name']}  score=${r.value}');
}

Pareto / ABC analysis

Which products make 80% of revenue?

final abc = await db.analytics.all.pareto('product', 'revenue');
for (final p in abc) {
  print('${p.key}: ${(p.share * 100).toStringAsFixed(1)}% '
        '(cumulative ${(p.cumulativeShare * 100).toStringAsFixed(1)}%) → ${p.abcClass}');
}

---

Window functions

Window functions compute a value for each document based on a sliding window over an ordered sequence — like Excel's moving-average formula or SQL's OVER (...).

Rolling average

// 7-day moving average of daily revenue
final rolling = await db.analytics.all
    .rollingAvg('amount', window: 7, orderBy: 'date');

for (final p in rolling) {
  print('day ${p.index}: value=${p.value}  7-day avg=${p.rollingValue}');
}

Cumulative sum

// Year-to-date running total for income entries
final ytd = await db.analytics
    .where((q) => q.where('type').equals('INCOME').findIds())
    .cumulativeSum('amount', orderBy: 'date');

print('YTD total: \$${ytd.last.cumSum}');

Other rolling aggregates

rolling computes any RollingAgg — sum, avg, min, max, variance, stddev, median — and there is a shorthand for each (rollingSum, rollingMin, rollingMax, rollingVariance, rollingStdDev, rollingMedian):

// 30-day volatility of daily prices
final vol = await db.analytics.all
    .rollingStdDev('price', window: 30, orderBy: 'date');

Time-based windows

window: 7 means "the last 7 documents". When data is irregular — weekends, missing days, several sales per day — use period for "the last 7 days" instead. orderBy must then be a date field (DateTime, ISO-8601 string or epoch milliseconds):

final weekly = await db.analytics.all.rollingSum(
  'amount',
  period: const Duration(days: 7),   // documents dated within (date − 7 days, date]
  orderBy: 'date',
);

Every rolling method accepts either window or period.

Partitions

Every window function accepts partitionBy to run an independent window per group — like SQL OVER (PARTITION BY ...). Each point carries its partition and orderValue:

// Running total per account
final balances = await db.analytics.all
    .cumulativeSum('amount', orderBy: 'date', partitionBy: 'account');

Period-over-period change

final growth = await db.analytics.all.change('total', orderBy: 'month');
for (final p in growth) {
  print('${p.orderValue}: ${p.delta} (${p.percentChange})');  // 0.12 = +12%
}
// offset: 12 compares with the same month last year.

---

Pivot tables

Turn rows into a spreadsheet-style grid — perfect for comparing values across two dimensions:

final table = await db.analytics.all.pivot(
  rowField:    'department',
  colField:    'quarter',
  valueField:  'budget',
  aggregation: PivotAgg.sum,   // sum | avg | count | min | max | variance | stddev | median
);

// Read a single cell
final q1Eng = table.rows['Engineering']?['Q1']; // num?

// Print the whole grid
print(['', ...table.columnKeys].join('\t'));
for (final entry in table.rows.entries) {
  final cells = table.columnKeys
      .map((c) => table.rows[entry.key]?[c]?.toString() ?? '-')
      .join('\t');
  print('${entry.key}\t$cells');
}

// Totals are computed from the documents, so they are right for averages too
print(table.rowTotals['Engineering']);
print(table.columnTotals['Q1']);
print(table.grandTotal);

Pass sortKeys: true to sort rows and columns. For date columns, computed keys or any aggregation, use pivotBy:

final table = await db.analytics.all.pivotBy(
  rows:    GroupKey.field('category'),
  columns: GroupKey.date('date', DateBucket.month),
  value:   aggSum('amount'),
  sortKeys: true,
);

---

Statistics

Describe

All the usual descriptive statistics in one call:

final s = await db.analytics.all.describe('amount');
print('n=${s.count} mean=${s.mean} sd=${s.stddev} '
      'min=${s.min} p25=${s.p25} median=${s.median} p75=${s.p75} max=${s.max}');

Percentile

Find the value below which a given percentage of data falls — great for SLA thresholds:

final p95latency = await db.analytics.all.percentile('latency_ms', 0.95);
print('95th percentile latency: ${p95latency}ms');

// Linear interpolation between ranks (Excel PERCENTILE.INC / NumPy default)
final median = await db.analytics.all
    .percentile('latency_ms', 0.5, interpolate: true);

By default the nearest-rank value is returned, which is always an actual value of the field.

Standard deviation

Measure how spread out values are — useful for detecting outliers:

final sigma = await db.analytics.all.stddev('amount');               // population
final s     = await db.analytics.all.stddev('amount', sample: true); // sample (n − 1)

Histogram

Understand the shape of your data by splitting it into equal-width buckets:

final hist = await db.analytics.all.histogram('price', bins: 10);
for (final bin in hist) {
  print('[${bin.low.toStringAsFixed(2)} – ${bin.high.toStringAsFixed(2)}): '
        '${bin.count} items');
}

Value counts

final countries = await db.analytics.all.valueCounts('country', limit: 5);
// → [ValueCount(MX: 120), ValueCount(AR: 80), ...] each with a share

Correlation and regression

final r   = await db.analytics.all.correlation('ad_spend', 'revenue'); // -1..1
final fit = await db.analytics.all.regression('ad_spend', 'revenue');
print('revenue ≈ ${fit?.slope} × ad_spend + ${fit?.intercept} (r² ${fit?.r2})');
print(fit?.predict(5000));

Outliers

final unusual = await db.analytics.all.outliers('amount');            // Tukey's fences (IQR, k = 1.5)
final extreme = await db.analytics.all
    .outliers('amount', method: OutlierMethod.zScore, k: 3);           // > 3σ from the mean

---

Live dashboards

watch re-runs any analysis whenever a document containing the watched field changes:

db.analytics.all
    .watch('amount', (q) => q.timeSeries('date', DateBucket.month, {
          'total': aggSum('amount'),
        }))
    .listen((series) => setState(() => _series = series));

watchGroupBy and watchHistogram remain available as shortcuts.

---

Performance: responsive by default

You don't need to do anything to keep your UI smooth. Every analysis runs cooperatively:

• Batched reads. Documents are read from the database in small batches instead of all at once.
• Time slices. Work runs in slices of at most ~8 ms. Between slices the event loop gets a turn, so Flutter keeps painting frames and handling taps while the analysis runs.
• Cached date bucketing. Local date strings are bucketed without parsing, and each distinct bucket is built once.
• Heavy final steps off the UI thread. Sorting, windows and medians over large inputs run in an isolate on native platforms. Only compact values are copied there, never documents.

Results are identical either way. Here is what that means for the longest UI freeze, measured on 50,000 documents (Dart VM, desktop):

Analysis	Longest UI freeze before	Longest UI freeze now	Total time now
timeSeries per month	2179 ms	8 ms	135 ms
groupBy with 8 aggregations	285 ms	15 ms	263 ms
describe	266 ms	26 ms	223 ms
rollingMedian (window 30)	675 ms	26 ms	674 ms
pivotBy category × month	1997 ms	33 ms	242 ms

At 60 fps a frame lasts 16 ms, so the remaining pauses cost one or two frames at most.

Because documents are read across several event-loop turns, a write made while an analysis runs may or may not be included. Run the analysis again (or use watch) to see it.

Explicit background execution (inBackground)

.inBackground() moves the whole computation to another thread: a Dart isolate on mobile, desktop and server, or a Web Worker on the web.

final summary = await db.analytics.all.inBackground().describe('amount');

You rarely need it. Doing this means copying every document to the other thread, and that copy happens on the UI thread. On 50,000 documents the copy alone freezes the UI for 120–220 ms, which is worse than the default above. It pays off only when the per-group work itself is heavy, for example an aggCustom that runs an expensive function over each group's documents.

Platform	Runs in	Dart functions (aggCustom, .filter, GroupKey.computed)
Android, iOS, desktop, server	A new Isolate (Isolate.run)	Supported
Web	A dedicated Web Worker, talking JSON	Can't be sent to a worker: those computations run on the main thread

derived, having, orderBy and limit always run on the calling isolate after the heavy work, so they can use any Dart code.

Web setup

Web Workers run a separate script, so compile one next to your app (once, and again when you upgrade the package):

1. Create web/ffastdb_analytics_worker.dart:

   import 'package:ffastdb_analytics/worker.dart';
   
   void main() => runAnalyticsWorker();

2. Compile it to JavaScript in the same folder:

   dart compile js -O2 -o web/ffastdb_analytics_worker.js web/ffastdb_analytics_worker.dart

Flutter web serves everything in web/ next to the app, so the default URL ffastdb_analytics_worker.js just works. If you put the script elsewhere, tell the executor:

AnalyticsExecutor.shared = AnalyticsExecutor.background(
  webWorkerUrl: 'workers/ffastdb_analytics_worker.js',
);

If the script is missing, the computation fails with a StateError explaining how to compile it.

Options

final executor = AnalyticsExecutor.background(
  minDocuments: 5000,   // smaller scopes stay on the calling isolate (default 1000)
);
final q = db.analytics.all.inBackground(executor);

await executor.close();  // terminates the web worker; it restarts on demand

• Lazy streams stay inline. rollingStream and cumulativeSumStream keep emitting as documents are read.
• Testing worker compatibility on the VM. AnalyticsExecutor.simulatedWorker() runs the exact JSON round trip of the web worker on the current isolate.

---

Full example — Accounting ledger

// 1. Balance grouped by account type
final balance = await db.analytics.all.groupBy('type', {
  'total':      aggSum('amount'),
  'entries':    aggCount(),
  'avg_amount': aggAvg('amount'),
});

// 2. Revenue vs expenses per quarter
final revenue = await db.analytics
    .where((q) => q.where('type').equals('INCOME').findIds())
    .groupBy('quarter', {'total': aggSum('amount')});

final expenses = await db.analytics
    .where((q) => q.where('type').equals('EXPENSE').findIds())
    .groupBy('quarter', {'total': aggSum('amount')});

// 3. Expense breakdown as a pivot (category × quarter)
final breakdown = await db.analytics
    .where((q) => q.where('type').equals('EXPENSE').findIds())
    .pivot(
      rowField:    'category',
      colField:    'quarter',
      valueField:  'amount',
      aggregation: PivotAgg.sum,
    );

// 4. Monthly income vs expenses in a single pass, gaps filled
final monthly = await db.analytics.all.timeSeries('date', DateBucket.month, {
  'income':  aggSum('amount').filter((d) => d['type'] == 'INCOME'),
  'expense': aggSum('amount').filter((d) => d['type'] == 'EXPENSE'),
});

// 5. Month-over-month expense growth
final growth = await db.analytics
    .where((q) => q.where('type').equals('EXPENSE').findIds())
    .change('amount', orderBy: 'date');

// 6. Flag unusually large transactions
final unusual = await db.analytics.all.outliers('amount');

---

Behavior and edge cases

Real-world documents are messy. The rules are the same across every method:

Situation	What happens
A group has no numeric values	aggCount counts documents, aggSum returns 0, every other aggregation returns null
A document lacks the group field	It is grouped under the null key (filter it out with having: (g) => g.key != null)
A pivot cell has no documents	The cell is null
A field mixes types (int and String)	Nothing throws: values are ordered by kind, num < String < DateTime
A document lacks the orderBy field	It goes last, keeping its original order (sorting is always stable)
A date field can't be parsed	GroupKey.date gives null; timeSeries skips the document
Invalid arguments (p outside 0–1, bins <= 0, window <= 0, negative n or limit)	ArgumentError, in release builds too
A dotted path hits a missing key, a non-map or an out-of-range index	The value is null (a key that literally contains the dots wins)
A time window's orderBy date can't be read	The document is skipped
Invalid window options (both or neither of window / period, period without orderBy)	ArgumentError
A write happens while an analysis runs	It may or may not be included (documents are read across several event-loop turns)
Returned documents (topN, rank, outliers)	They are copies: modifying them never touches the database

---

How it works

Every analytics operation follows the same two-step pattern:

1. Scope — decide which documents to include using .all or .where(filter).
2. Analyse — call any analytical method on that scope.

db.analytics
  .where(...)       ← optional filter (any ffastdb query)
  .inBackground()   ← optional, rarely needed: compute in an isolate / web worker
  .groupBy(...)     ← or any other analytical method

Documents are read from storage on each call, so results are always fresh. Every analysis is a single-pass computation. It consumes documents in batches and time slices as they are read, and aggregations (groupBy, timeSeries, pivot, stddev, topN…) keep only accumulators in memory, not the documents. See Performance.

---

Roadmap

Ideas under consideration. Feedback and pull requests are welcome.

Analysis

• Exponential moving average (RollingAgg.ema).
• Period comparison for KPI cards: this month vs the previous one or the same month last year.
• Subtotals (rollup) across grouping levels.
• Equal-frequency bins (quantiles / ntile).
• Cohort retention and funnel analysis.
• Simple forecasting: linear trend and Holt-Winters over a timeSeries.
• Concentration metrics (Gini coefficient, entropy).

Performance and integration

• Letting a background thread read the documents itself, so inBackground never copies them through the main thread.
• Incrementally maintained aggregates for watch, instead of recomputing on every write.
• Approximate distinct counts and percentiles (HyperLogLog, t-digest) for very large collections.
• Export helpers: CSV / JSON, and chart-ready series for packages such as fl_chart.

---

License

MIT