← All projects
Research tooling · Earth Observatory of Singapore

Ultimate Downloader

Getting terabytes of ALOS-2 scenes onto a cluster is nobody's research. This turns it into one command: parallel FTP and SFTP transfers, optional PBS or Slurm submission, extraction into the layout the processors expect, and registration with the scene catalogue.

HPCPythonRemote SensingSAR
Ultimate Downloader

Ultimate Downloader

SAR processing starts with a large, dull, failure-prone download. This tool makes that step unattended.

Transfers

FTP and SFTP, with password or SSH-key auth and saved credential profiles for the servers used often. Transfers run in parallel threads with live progress, speed and ETA, whole directories can be pulled recursively, and an interrupted run resumes by comparing local against remote sizes rather than starting over.

Then the part everyone forgets

A finished download is not a usable scene. Archives are extracted and organised into the P{path}/F{frame} tree the downstream processors expect, and each extracted scene is registered with the satellite scene catalogue API — so what arrives on disk is already discoverable, not something a person has to file.

Where it runs

On a cluster it submits itself: the same download can be dispatched as a PBS or Slurm job instead of held open in a terminal, with Slack notifications on status and completion. For exploring an unfamiliar archive there is an interactive mode with a tree view and tab completion.