Order syncing data flow

Overview

Order syncing is the process that closes the loop between sending postcards and measuring their impact. Every 5 minutes, a background job (process_orders) fetches new orders from each organization's integration provider (Klaviyo, Ometria, or Shopify) using checkpoint-based incremental syncing — it picks up where the last sync left off.

For each order, the system attempts to attribute it to a campaign using a cascading match hierarchy:

1. Email match — does the order email match a sent recipient's email?
2. Physical address match — does the shipping address match a recipient's address? (skips if the customer had an active subscription before the mail was sent)
3. Discount code match — does the order include a campaign or recipient-specific discount code? (the only method that works for unsent recipients)
4. Holdout match — if no experimental match was found, does the order match a holdout recipient? (uses created_at instead of sent_at since holdouts never receive mail)

The first method that matches wins. Attributed orders are saved to the database with an unprocessed attribution record, which a separate background job later validates against timing windows and business rules.

For the full attribution logic — timing constraints, holdout analysis, edge cases — see docs/attribution_methodology.md.

Flowchart

  graph TD
      A[Background Scheduler] -->|Every 5 minutes| B[process_orders]
      B --> C[get_organizations]
      C --> D{For each organization}
      D --> E{sync_orders enabled?}
      E -->|No| F[Skip organization]
      E -->|Yes| G{organization enabled?}
      G -->|No| F
      G -->|Yes| H[sync_organization_page_order]

      H --> I[get_mail_integration]
      I --> J{Integration exists?}
      J -->|No| K[Return - no integration]
      J -->|Yes| L[fetch_integration_order_events]

      L --> M{Integration Provider?}
      M -->|Klaviyo| N[get_klaviyo_client]
      M -->|Ometria| O[OmetriaClient]
      M -->|Other| P[Log warning - unsupported]

      N --> Q[find_placed_order_metric]
      Q --> R[find_events_for_metric]
      R --> S[Convert to OrderEvent]

      O --> T[list_orders since checkpoint]
      T --> U[Convert to OrderEvent]

      S --> V[Process Events]
      U --> V

      V --> W{Events found?}
      W -->|No| X[Log no orders]
      W -->|Yes| Y[get_campaigns_by_organization]
      Y --> Z[For each order_event]

      Z --> AA[process_order]
      AA --> BB{Valid org domain?}
      BB -->|Company email| CC[Skip order]
      BB -->|Valid| DD[attribute_order]

      DD --> EE[check_for_matched_emails]
      EE --> FF{Email match found?}
      FF -->|Yes| GG[Set attributed_campaign]
      FF -->|No| HH[check_for_physical_address_match]

      HH --> II{Address match found?}
      II -->|Yes| GG
      II -->|No| JJ[Check discount codes]

      JJ --> KK{Discount code match?}
      KK -->|Yes| GG
      KK -->|No| LL[check_for_holdout_order]

      LL --> MM{Holdout match?}
      MM -->|Yes| NN[Create holdout attribution]
      MM -->|No| OO[No attribution]

      GG --> PP[check_for_blacklist]
      PP --> QQ{Attributed campaign > 0?}
      QQ -->|Yes| RR[upsert_order_by_provider_id]
      QQ -->|No| SS[Skip saving order]

      RR --> TT[insert_unprocessed_attribution]
      TT --> UU[Log OrderAttributed event]

      NN --> VV[Log OrderHoldoutAttributed]
      VV --> XX[upsert_attribution_by_provider_id]

      UU --> ZZZ[Continue to next event]
      SS --> ZZZ
      XX --> ZZZ
      OO --> ZZZ

      ZZZ --> YYY{More events?}
      YYY -->|Yes| Z
      YYY -->|No| YY[Sort events by datetime]
      YY --> ZZ[update_organization_order_checkpoint]

      subgraph "Attribution Methods"
          AAA[Email Match]
          BBB[Physical Address Match]
          CCC[Discount Code Match]
          DDD[Holdout Attribution]
      end

      subgraph "Data Storage"
          EEE[Redis Cache]
          FFF[Database Orders]
          GGG[Database Attributions]
          HHH[Analytics Logs]
      end

      subgraph "External Providers"
          III[Klaviyo API]
          JJJ[Ometria API]
          KKK[Future: Shopify API]
      end