Engineering case study
icloud-to-gphotos
A Python pipeline that moves iCloud photo libraries to Google Photos in scheduled batches, preserves media metadata, and checks upload confirmation before removing the source asset.
Context
Moving a photo library involves more than copying files. A Live Photo has multiple components, edited images have additional versions, and missing capture dates can scramble the destination timeline. I built the migration around explicit resource state so interruptions and partial uploads can be handled deliberately.
System design
- PlanRead oldest first; assign resources and a batch budget.
- PrepareStream originals to disk and backfill missing metadata.
- ConfirmUpload media and record each remote confirmation.
- RetireCheck every resource and the grace period before deletion.
Key decisions
Make deletion conditional
An upload attempt is insufficient evidence to delete a source. The pipeline requires a returned media key or a confirmed remote duplicate for each resource, checks a configurable age grace period, and rechecks the ledger immediately before deletion. Both components of a Live Photo must satisfy that requirement.
Keep progress independent of the process
SQLite records assets, individual resources, and run outcomes. A later run skips resources already confirmed remotely. Failures have a finite retry budget; exhausted items are reported and retained in iCloud so one troublesome file does not hold up the remaining library.
Fit the work to the machine
Batches use item and byte budgets adjusted for available disk headroom. Downloads stream to temporary files, check their expected size, and rename atomically after completion. Staging is cleared between batches. The byte budget is a planning limit, with individual assets kept together.
Preserve meaning as well as media
Metadata backfill fills missing tags without replacing existing ones. EXIF wall-clock timestamps and UTC QuickTime timestamps need different treatment. Live Photo components share a filename stem for pairing, while edited renders upload in a separate pass to avoid accidental pairing.
Make unattended work inspectable
The CLI includes dependency and credential checks, progress inspection, and JSON run reports. Reauthentication alerts make expired iCloud sessions visible. Dry-run and no-delete modes let an operator inspect the migration policy before enabling source removal.
Operation & limits
Linux deployment uses a dedicated systemd service account, restricted filesystem access, journal logging, and a persistent timer with explicit Asia/Kolkata scheduling. A randomized delay spreads the nightly start. Windows Task Scheduler support provides the same pipeline on a desktop host.
References
Source code and documentation for a closer look.