Handling Schema Changes

Tables and columns in Data Exports can change over time. Build your integration so these changes don’t interrupt your data loads.

Inspect the schema before building a full integration

Before you build pipelines, warehouse tables, or reports on top of Data Exports, download a recent full export and inspect the schema of each Parquet file you plan to use. Check the column names, data types, and which columns can contain null values. Parquet files store their schema, so most Parquet readers (for example, PyArrow, pandas, DuckDB, or Spark) can print it without loading the full file.

Base your integration on the columns the files actually contain, not on assumptions from table names, the descriptions in this documentation, or other ShipHero APIs. A field available in the GraphQL API may be named differently in an export, or may not be exported at all.

Expect tables and columns to change

The columns in a table can change between export runs. Columns can be added, removed, or renamed, and a column’s data type can change. New tables and modules can also appear in the response. Each change is listed in the Data Exports changelog. Review the schema again when an entry affects a table you use.

Read the files tolerantly

Avoid strict schema validation that rejects a file when it doesn’t match an exact expected definition. Instead:

• Select the columns you need by name, never by position.
• Ignore columns you don’t recognize instead of treating them as an error.
• Treat a missing column as null or a default value, and log that it was missing.
• Ignore tables and modules your integration doesn’t use.
• When you write to your own database, add new columns as they appear, or store the raw records alongside the columns you map explicitly.

Don’t let a schema change fail your load

A change in one table shouldn’t stop your whole pipeline. When a file doesn’t match what your integration expects:

1. Load the data you can read and continue processing the remaining tables and files in the run.
2. Log a warning or send an alert with the run_id, the module, the table, and the columns that differed.
3. Update your integration for the new schema, then query the affected period again to reprocess that run. Signed URLs expire after 24 hours, so run the query again to get new URLs instead of reusing old ones.

This keeps your data flowing when an export changes, and gives you the information you need to update your integration.