ECHO Data Package 1.0.0

A common layout for every dataset ECHO distributes: the original files, unchanged, plus the same data converted to a small set of standard tables with documented columns, in open formats, with machine-readable metadata, licence and credit.

This page describes version 1.0.0 of the standard (draft).

Standards used

Layout

A package lives at /download/<slug>/. /download// redirects to the latest version folder. Each version has its own folder, /download/<slug>/v<MAJOR.MINOR.PATCH>/, containing:

Rules:

• A published version folder never changes. Corrections are a new version with a CHANGELOG entry. One exception: while ECHO's datasets are first being converted to this format, every package is released as 1.0.0, and a correction during that period rebuilds 1.0.0 in its place (its manifest-sha256.txt changes with it). Once the conversion is complete, this exception ends.
• data/ is the contract. csv/, jsonl/ and graph/ are generated from it and never edited by hand.
• Every table can be downloaded, or queried over HTTP, on its own.
• original/ holds the upstream files byte for byte; nothing in it is ever modified.
• Operating-system junk (__MACOSX, .DS_Store, ._*, Zone.Identifier, Thumbs.db, .ipynb_checkpoints) and saved error pages are never packaged.
• Every package records the recipe that built it: datapackage.json gives the recipe's commit in ECHO's recipe repository and the SHA-256 of its text, so a package can be rebuilt and checked.
• A partitioned table's parts may be the same files as the tables of other packages: a year of a monthly series, for example, is made of the months' files. They carry the same checksums; nothing is copied or changed.
• Licence: the package carries the licence the original publisher states. Where the publisher states none, the licence is NOASSERTION and LICENSE.md says so: ECHO assigns no licence and redistributes the publicly released files for research and teaching, with credit. Terms that are not a standard licence (for example research use only) are quoted verbatim under a LicenseRef identifier. A licence that a re-uploader declared on content they did not create is reported in the note but not applied. A lossless change of format is a technical modification, not an adaptation, so NoDerivatives licences are respected.

Conventions

Names. Tables and columns are snake_case. A platform-specific table or column keeps its meaning under an x_ prefix (x_score, x_gilded, x_votes), so it can never collide with a standard one. x_ stands for extension; it has nothing to do with the platform X (Twitter).

Identifiers.

• Every ID is a string, exactly as upstream wrote it. IDs are never converted to numbers, because large integers lose digits as floating-point numbers.
• Where upstream gives no ID, ECHO mints one as echo::; the column's schema marks it minted, and minted IDs are stable only within one version.
• Where two kinds of node share a numbering (bipartite and heterogeneous graphs), node IDs are prefixed with their type: user:17, item:17.
• IDs that were damaged upstream (rounded by a spreadsheet, written in scientific notation) are kept as found and flagged in the column's quality note. They are never repaired, because a repaired ID would point to a different record.

Time.

• Absolute times are ISO 8601 timestamps in UTC (created_at), kept to the microsecond. The upstream time zone or offset, where known, is recorded in the column's schema; a source time finer than a microsecond is kept as a string.
• Where upstream gives no time zone and the data give no evidence of one, the time is kept as text exactly as written, not presented as UTC; the column note says so.
• Where upstream recorded only a date, the column is a date, not a timestamp at midnight.
• Relative or ordinal times (seconds since the start of a study, "26 days ago", survey waves) go in a column t with a declared time_unit and time_origin. They are never presented as dates.

Text.

• Text is UTF-8, kept exactly as the source wrote it: no Unicode normalisation, lower-casing, stripping, HTML unescaping or spelling correction. Text from different sources may therefore use different Unicode forms for the same characters; normalise (for example to NFC) before comparing or matching text.
• Where upstream text is mis-encoded (mojibake), it is kept as found and the column carries a quality note; the original bytes remain in original/.
• Where upstream text is not valid UTF-8, the column holds the text decoded with replacement characters and an x__bytes column holds the original bytes (as hexadecimal text) for exactly those rows, so nothing is lost.
• Where upstream text is not valid UTF-8 and the package also holds a clean copy of the same records (for example a Kaggle copy of a CrowdFlower file), the text column holds the clean copy's text for every record that could be matched to exactly one record of the clean copy on fields other than the text itself. The upstream text is kept as found in x_ for every row, with its original bytes in x_bytes, and x_source names the file each value of the text column came from. The README notes give the matching fields and say how many rows take their text from which file; a record that cannot be matched without doubt keeps the upstream text.

Values.

• Booleans are true booleans, never the strings "True"/"False" or 0/1 codes.
• Missing values are null. Upstream sentinels (-1, NaN, "", "N/A") become null only where upstream documents them as missing; otherwise they are kept and noted.
• Lists, vectors and nested records are JSON text columns (type json) in EDP 1.0; their structure is described in the column note.
• Numbers keep their upstream precision; counts are integers.
• Placeholder values: a value that stands in for a missing, hidden or anonymous entity rather than naming a real one (an author written as None, as 1234567? or as [deleted]) is kept exactly as found, never turned into null, because upstream does not document it as missing. The recipe declares such values for a string column, and the builder counts them in the built table and writes them to the column's field metadata in datapackage.json, as echo:placeholders: a list of entries {value or pattern, count, distinct, about}, where value is an exact string, pattern is a regular expression (RE2 syntax) that must match the whole value, count is the number of rows of this package that hold such a value, distinct the number of different values among them, and about says what the value stands for. A consumer that wants real entities only (users, authors) excludes these values; a column without echo:placeholders declares none.

Lossless conversion.

• Every conversion is checked against the source: row counts per table, and a value-by-value comparison of a sample of rows. A conversion that loses or changes information fails the build.
• Where a source value cannot be represented in its standard column without loss, it is kept as a string in an x_ column and noted.

Standard tables

A package contains the tables its data supports. Columns marked required are present in every copy of that table; the others appear where the data has them. A column specific to one platform keeps its meaning under an x_ prefix. Types: string, integer, number, boolean, timestamp (UTC), date, list and json.

posts

Pieces of content people wrote: posts, comments, replies, tweets, messages, reviews.

Primary key: post_id.

threads

Conversations or discussion threads.

Primary key: thread_id.

users

Accounts, users or authors.

Primary key: user_id.

communities

Places where content is posted: subreddits, boards, groups, channels, instances.

Primary key: community_id.

memberships

Who belongs to which group.

nodes

Nodes of a network, with their attributes.

Primary key: node_id.

edges

Links of a network. One row per link as upstream recorded it: repeated links and events are never merged.

interactions

A user acting on an item: ratings, likes, check-ins, plays, bookmarks.

items

Things users act on or post about: videos, products, images, articles, venues.

Primary key: item_id.

annotations

Labels attached to posts, users, threads, links or text spans, by people, models or rules.

media

Images, video or audio attached to posts.

Primary key: media_id.

ids

Identifiers only, for collections whose terms allow sharing IDs but not content.

snapshots

The crawls or snapshots in a repeated collection.

Primary key: snapshot_id.

splits

Alternative or k-fold splits.

timeseries

Aggregate measurements over time, in long format.

features

Precomputed features or embeddings, with a feature dictionary in the schema.

Structure properties

Each package is tagged with the structural properties of its data. They are computed from the tables themselves, and can be used to find datasets on ECHO (Browse → Structure).

Collections, views and derived packages

• view of: This package is a filtered view of a collection package: {package, version, filter}.
• derived from: This package is a slice or an enrichment of another package: {package, version, how}.
• members: This package is a collection; its member packages are listed with their filters.

Metadata

Contributors. Everyone credited, with their role: the creators of the data, the original publisher, every redistributor in between (for example SNAP → KONECT → Network Repository), and ECHO as packager. Taken from upstream documentation, never guessed. Roles: creator, contributor, publisher, distributor, packager.

Privacy review. What personal information the package contains, what was withheld and why. A withheld column stays in the schema, marked withheld, so users can see that it exists.

Quality notes. Per-column notes on known upstream problems (damaged IDs, mis-encoded text, sentinel values), so no user mistakes them for ECHO's conversion.

Fields for software and AI tools. Fields that let APIs, ML pipelines and language-model tools use a package without reading its README.

• text_columns: The columns holding natural-language text, with their language where known.
• id_columns: The columns holding identifiers, and what they join to.
• row_license: Where the licence varies by row, the column that holds it.
• embedding_columns: Vector columns, with the model that produced them.
• languages: The package-level languages list in datapackage.json: the languages of the dataset's text as recorded in ECHO's catalog when the package metadata was written, as lower-case ISO 639 codes such as en, de or fil, sorted. It describes the dataset as a whole, not individual columns or rows, and is absent where the catalog records no language. Packages published before this property was added do not carry it; the catalog is the source for them.
• placeholders: The echo:placeholders list of a column's schema field: placeholder values (see Values above) with their counts, so that a tool can tell them from real entities without reading the README.

The README

Every package's README.md has these sections, in this order: