ECHO Data Package: formats by size
Every table in a package is stored in data/ as Apache Parquet. Whether it also gets a CSV copy, a JSON Lines copy or graph files depends on its size, as below. Sizes are measured on the table itself: its number of rows and its uncompressed size. A table falls in the first band whose limits it is within.
| Band | Up to (rows) | Up to (size) | Parquet | CSV | JSON Lines | Graph TSV | GraphML |
|---|---|---|---|---|---|---|---|
| small | 1 million | 2 GB | one file | yes | text and nested tables | yes | yes |
| medium | 100 million | 50 GB | one file | yes | text and nested tables | yes | no |
| large | 1 billion | 500 GB | partitioned | partitioned | no | yes | no |
| very large | no limit | no limit | partitioned | no | no | no | no |
Graph TSV and GraphML apply to network tables (edges and nodes) only. GraphML describes a single static graph, so networks recorded as several snapshots or layers get the TSV files, which keep the snapshot and layer columns, and no GraphML.
Partitioned tables
A partitioned table is a folder of Parquet files of about 512 MB each, split by the first rule that applies:
- table has created_at: by month of created_at
- edges has layer: by layer
- table has snapshot_id: by snapshot_id
- otherwise: consecutive rows, in parts of about the target size
Parquet settings
Compression zstd (level 9), row groups of 1,000,000 rows.
When these rules change
These rules are versioned separately from the standard (this is rule set 2026.11.0). When they change, every package whose tables fall in a different band is rebuilt with the new formats as a PATCH version; the data in data/ does not change. Each package records the rule set it was built with in datapackage.json (echo:policy_version).