Data Platform Modernization: 8-Wk Imp

Trace3's approach leverages the Azure native tools to drive "Ingest to Insight" and incorporates tools like ADF, ADLS, Databricks, & Synapse to modernize your platform. Once the platform is modernized, we begin to leverage the other ADS tools to drive insights. Azure DevOps and Azure Data Catalogue are used to govern this end-to-end data program. The team then prepares the data to be consumed by PowerBI so the business user can leverage that data to drive outcomes and begins the next phase of the framework to drive MLOps & AI.

This engagement will focus on an approach that delivers and assesses the following key deliverables that drives more insights through Azure.

Ingest and Transform • Ingest: Ingest data into Azure Synapse Analytics through Azure Data Factory (ADF) and/or CTAS best practices. • Data Transformations using ADF Mapping Data Flows and/or Databricks delta lake bronze, silver and gold transformation patterns before pushing to Azure Synapse Analytics.

Model and Serve • Designing and implementing staging and production data loading strategies. • Designing partitioning and data distribution strategies in Azure Synapse Analytics. • Data Modelling in Azure Synapse Analytics.

Cost Management • Implementing audit logging in Azure Databricks to control workspace budgets. • Implementing Azure Databricks cluster policies to control costs on user defined clusters.

Monitor and Optimize • Workload Isolation and Management • PowerBI dashboards to monitor sessions, query executions and waits from dynamic management views. • Query performance optimization through materialized views, result set caching and clustered column store indices.

Security • Implementing security in Azure using virtual networks, privatelink, Databricks diagnostic logging, customer managed keys and cred pass through.

CI/CD and Automated Infrastructure as Code • CI/CD of Azure Data factory pipelines