HERON (GEK TERNA Group)

Consumption File Splitter & Reconciliation

Summary

A queue-triggered Azure Function that splits very large consumption files from the national electricity distribution operator into ERP-ingestible segments, with a reconciliation report proving the split is lossless. Splitting is domain-aware (records are grouped by delivery point and calendar month so no logical unit is fragmented), and reconciliation uses exact-arithmetic, arbitrary-precision decimals so spreadsheet rounding cannot mask a discrepancy.

Architecture

A queue message points at a consumption file in blob storage, which is parsed and validated, grouped by delivery point and calendar month, then packed into segments; totals are computed independently from the source and from the segments and compared in a reconciliation report, and both the segments and the report are filed to a SharePoint library.
A queue triggered split whose segment totals are recomputed and compared against the source.
  • Groups are only ever appended whole to a segment; no code path slices one. That makes the delivery point and month unit a hard guarantee, and it makes the segment row cap the soft constraint instead. A group larger than the cap produces an oversized segment rather than a fragmented one.
  • Reconciliation totals are accumulated with arbitrary precision decimals at a configured precision of 40, and written into the report as text formatted cells rather than numbers, so a spreadsheet cannot silently re-round them on open. The earlier float based implementation remains in the file as a commented block, which shows this was a deliberate replacement rather than an initial choice.
  • The host is configured to process exactly one queue message per instance, with the batch size set to one and the new batch threshold to zero. The pipeline holds the whole file, every partitioned group, and every output buffer in memory at once, so this setting is what prevents several large files from coexisting in one worker.
  • The parser is configured to fail loudly rather than degrade: ragged lines are not truncated, parse errors are not ignored, and an empty file raises. A malformed row aborts the run with a readable message instead of producing a silently short segment. Encoding and decimal separator handling are set explicitly for the source format.

Highlights

  • Domain-aware splitting: records grouped by delivery point and calendar month, whole groups packed into bounded batches; a logical unit is never fragmented.
  • Exact-arithmetic reconciliation using arbitrary-precision decimals written as text, so spreadsheet rounding cannot mask a discrepancy.
  • Self-triggering feedback loop prevented by filtering the function’s own artifacts out of its input queue.
  • Memory instrumented at every stage to stay within serverless plan limits; 100% line coverage.

Technologies

  • Python
  • Polars
  • Azure Functions
  • Azure Storage Queues
  • Microsoft Graph
All projectsView ResumeGet in touch