CSV Conversion Azure Function
An Azure Function that transforms and uploads a CSV file in Blob Storage.
Technologies used:
- Azure Functions 4
- Azure Blob Storage
- C# 6.0
- CSVHelper 27.1.1
- CSVHelper 27.1.1
Year made/current status:
- Created 2022
- Feature complete, in production
What it is:
This Azure Function is piece of a new effort to take in data from many clients in many forms, arrange all of the disparate data, and send the aggregate to another source. Specifically, this app is triggered by an Azure Blob Storage container, is sent large CSV files with a table/header format from Company A, identifies and maps the data to the format of Company B, writes a new pipe-delimited file, and streams the resulting file to a different Blob Storage container. All of the above is asychronously done, and the resulting file is streamed in chunks of a configurable size.
Features of the app include:
- Dynamic object loading from the input file stream
- Robust block streaming into Azure to prevent upload failures
- Modifying and verifying input metadata to prevent files from being processed twice
- Fast runtime (average of XXX ms from when the function is triggered to when the result is uploaded for a 1MB input file)
What I did:
I was given the open ended requirement of "translate a CSV in Blob A and make it appear in Blob B". I was given a sample file from an Advisor with a rough draft of the input/output CSV formats, and was eventually tasked with coordinating a final design of the output based on this. Once I had a final design, I was wrote out the JSON file with the mappings to do the conversion. While facilitating that, I researched C# CSV packages and decided on CSVHelper since it is easy to use, well documented, open-source, and has a good feature set. Once the function was operational in a test environment, I deployed it to its final destination set up by our Cloud Engineer.