Building reliable data pipelines at scale

Pathways handles over 240 integrations, and each one comes with its own API quirks, rate limits, authentication edge cases, and failure modes. Making that reliable at scale means you cannot treat every integration as a special case. You need a sync architecture that handles the common failure patterns generically, and a layer on top that lets you configure integration-specific behavior without forking the core logic. That is the architecture we settled on after about six months of production learnings.

Getting retry logic right

Retry logic is where most pipeline systems get into trouble. Naive retry on failure creates duplicate data when the original request succeeded but the acknowledgment failed. Pathways uses idempotency keys scoped to each sync event, so retries are safe to run without checking for duplicates downstream. Conflict resolution is configurable per integration: last-write-wins is the default, but teams with strict data integrity requirements can define field-level rules that control which source wins when two systems disagree on the same value.

Full pipeline observability

Full pipeline observability was something we added later than we should have. Every sync event now generates a structured log entry that captures source, destination, record count, duration, conflict count, and retry count. Those logs are queryable through the Strand interface and exportable for teams that want to pipe them into their own monitoring stack. If a pipeline fails or slows down, you can see exactly where it happened and why, without digging through API logs on the source side.