Analytics Platform for Microsoft Azure: 3-Month Implementation

Modern systems require a lot of data processing, which usually involves even more pipelines for processing ETL feeds, collecting analytics data, etc. It is usually relatively easy to create new ones, but over time the technical burden related to maintenance may become a major problem.

Grid Dynamics' solution is to apply the same treatment usually reserved for application code in the scope of CI/CD and make use of templating and infrastructure automation frameworks to bring order, proper QA and code review practices, and release management procedures to data pipelines. This helps to greatly reduce maintenance overhead, allowing a relatively small team to support hundreds of pipelines in solutions like Azure Databricks, while avoiding cloning of typical configurations in case new projects should be onboarded within the framework, typically reducing onboarding time from weeks to days or even hours.

Benefits: Our approach allows developers to manage Azure data pipelines using convenient development tools and familiar processes. Using templates eliminates the need to copy-paste the pipeline definitions for similar ETL processes over and over and allows rolling out updates automatically across the organization while enforcing Azure best practices.

Added CI/CD process on top of Azure ETL pipelines allows for running automated tests to catch potential issues earlier and deploy new versions to different environments automatically if needed. Treating Azure pipeline code as an artifact together with implementing service discovery principles ensures its immutability across dev/qa/production infrastructures, reducing potential issues. Leveraging the IaC approach enables more efficient Azure infrastructure management. Onboarding time for a new team or new set of pipelines is typically reduced from days to hours. Standardization and unification for data pipelines across the organization. The framework and approach can be used to both improve existing processes and help get started with best practices in building Azure data pipelines from day one.

Engagement plan: The Grid Dynamics team will conduct a discovery and pre-assessment workshop to give you an overview of our pipeline automation and migration framework. As a part of the workshop we will perform a deep dive into your needs in order to identify the best approach of applying our framework to address them.

As a next step, we will conduct a short-term POC to validate the approach for a limited set of your data pipelines in Azure and fine-tune the framework. After validation of the POC results, Grid Dynamics will implement the same changes for the rest of your Azure data pipelines and perform necessary trainings to the respective engineering teams so you will be able to support and extend the framework on your own.