Interbank Payment File Transfer

Summary

A production batch-transfer service moving settlement files from a national interbank clearing system into an enterprise ERP over SFTP, on a fixed daily schedule, replacing a manual operations task. Runs on Azure Functions with durable transfer state in blob storage.

Architecture

• The SFTP prefetch window is capped at 64 outstanding requests rather than left at the library default. The comment records the diagnosis: the remote server silently drops prefetch requests past roughly 529 outstanding, stalling every download at exactly 17,334,272 bytes, which is 529 multiplied by the 32768 byte block size. A reproduced failure with a measured boundary and a targeted fix.
• Archiving is attached to a finally block on the upload rather than the success path, so it also runs when the ERP upload fails. The stated intent is that the archive mirrors what the clearing system sent, not what the ERP accepted, which makes it usable as evidence during a dispute.
• Uploads write to a dot prefixed temporary name and then rename, so a directory poller on the receiving side never observes a partial file. The leading dot also keeps the in flight file out of naive directory listings, matching the convention used to skip hidden files on the read side.
• Retry backoff uses full jitter rather than fixed exponential delay, and the archive path is given a shallower budget than the transfer paths because the storage SDK already applies its own transport level retries. The archive decorator is also the only one with an explicit non retryable set, so a missing container fails fast instead of consuming the run budget.

Highlights

• Safely re-runnable via organization-scoped idempotency keys: re-execution after partial failure never re-delivers a file.
• Diagnosed and fixed a reproducible stall on large downloads by capping prefetch concurrency to a measured bound, preserving throughput.
• Atomic delivery: uploads to a temporary remote path and renames on completion, so the ERP never sees a partial file.
• Per-file failure isolation with a loud overall failure contract, plus layered guardrails (size validation, record caps, wall-clock budget).
• 100% line coverage across 148 tests, strict typing, security scanning in CI, operator CLI for reconciliation.

Technologies

• Python
• Azure Functions
• SFTP (paramiko)
• Azure Blob Storage
• Application Insights