Multi-Source Document Assembly

Summary

An asynchronous service that collects every document related to a commercial record from three source systems (an AWS S3 archive, a customer portal on Azure Blob, and a third-party quoting platform’s API), merges them into one PDF, and files everything into SharePoint. Attachments are deduplicated by content hash before processing, and a page-by-page fallback salvages readable pages from partially corrupt PDFs so one bad attachment can never fail a request.

Architecture

• Deduplication hashes the full content in fixed size chunks and restores the stream position both before and after reading, so hashing never consumes the caller stream. It also runs before the mergeable extension filter, which means duplicate non mergeable attachments are suppressed from the pass through uploads as well as from the merge.
• The merged document is handed to the uploader as a memory view rather than a byte string, and each upload chunk is a slice of that view, so a large PDF is not copied again for every chunk. The view is explicitly released afterwards, which is required before the underlying buffer can be closed.
• Every document is re-saved through a normalization pass that rebuilds the cross reference table and drops orphaned objects before merging. That pass is what recovers most malformed files; the page by page loop behind it is the second attempt, not the primary mechanism.
• The repository carries hand written type stubs for three untyped dependencies, wired in through the type checker path and containing only the handful of symbols the application actually uses. This keeps the project under a strict type checking configuration without scattering suppression comments.

Highlights

• Content-hash deduplication before processing, since the same attachment legitimately appears in multiple systems.
• Running size budget during the merge, demoting oversized documents to separate delivery instead of discarding them.
• Page-by-page fallback salvages readable pages from partially corrupt PDFs; one bad attachment can never fail a request.
• Business rule enforced at the fetch boundary: unsigned drafts never enter the archive.

Technologies

• Python
• asyncio
• PyMuPDF
• AWS S3
• Azure Blob Storage
• Microsoft Graph