Data Engineer

Covestic Inc; Hyderabad; On-site; INR 1,800,000 - 3,800,000; Full time

Job summary

Milestone Technologies is seeking a Data Engineer to support an EDW migration from on-prem SQL Server 2016 to Azure Cloud, focusing on data quality and error handling in new data pipes.

You will design automation for quality measurement, collaborate with report developers, and help evolve CI/CD processes in Azure DevOps to improve reliability. Strong SQL Server and Azure Data services experience required.

Qualifications

• Minimum 5+ years of experience designing, building, and supporting data pipes.
• Minimum 2+ years of experience creating automated solutions for measuring data quality for ETL processes.
• Minimum 1+ year of experience with Azure Data Services (Azure Data Catalog, Blob Storage, Data Lake, Synapse, Analysis Services, Data Factory).
• Strong background with Microsoft SQL Server databases (preferably 2016).
• Understanding of Power BI and DAX functions.
• Able to read and edit PowerShell scripts; experience with CI/CD using Azure DevOps.
• Strong interpersonal and customer relationship skills; able to communicate with non-technical partners.

Responsibilities

• Translate data requirements for new data quality controls and thresholds into technical designs.
• Create automated processes to measure data quality, failures, and errors.
• Analyze existing systems and data pipes to identify improvement opportunities.
• Evolve and improve data quality processes and templates.
• Profile data sources to understand quality issues in ETL processes.
• Collaborate on data modeling using Kimball dimensional modeling practices.
• Aggregate data from multiple sources and explore data quality enhancements.
• Develop tools to measure and report data quality; support remediation processes.
• Assist CI/CD processes in Azure DevOps to improve data quality.
• Support innovative technology POCs with Senior BI and Data Engineers.

Skills

Data pipeline design

ETL & data quality

Azure cloud services

SQL Server

Power BI/DAX

CI/CD with Azure DevOps

Python scripting

Communication

Education

Bachelor's Degree in Computer Science or related field

Tools

Azure Data Factory

Azure Synapse Analytics

Azure Data Lake Storage

Azure Blob Storage

Azure Data Catalog

PowerShell

Python

Job description

Milestone Technologies is looking for a Data Engineer to support a client's EDW migration from on-prem (SQL Server 2016) to the Azure Cloud. This role will focus on the data quality and errors for the new data pipes established from the source to the new target cloud database. You will work closely with the team that is designing and building the data pipes as well as the lead business analyst who is working with internal stakeholders defining the data quality requirements and thresholds. You will be responsible for designing and building the processes for measuring data quality, data transmission failures, and data load errors. In addition to this, you will be expected to work with report developers to create dashboards and reports showcasing the data quality.

• Translate data requirements for new data quality controls and thresholds into technical designs and specifications
• Create automated processes to measure data quality, failures, and errors
• Analyze existing systems and data pipes to identify areas for improvement or opportunities to enhance
• Continue to help evolve and improve the data quality processes and templates
• Profile data sources to understand existing quality issues in the ETL processes
• Collaborate on data modeling tasks including recommending and evaluating designs based on Kimball dimensional data modeling practices
• Aggregate and combine raw information from different data sources
• Explore ways to enhance data quality and reliability including helping with the development of support and remediation processes
• Identify opportunities for data acquisition
• Develop analytical tools and processes to measure and report data quality
• Help adjust and tune existing CI/CD framework and processes (Azure DevOps) as needed to improve data quality
• Assist Senior BI Engineer and Senior Data Engineer on innovative technology POCs

Skills

• Bachelor's Degree or higher in a technical field such as Engineering, Computer Science, etc.
• Minimum 5+ years of experience designing, building, and supporting data pipes
• Minimum 2+ years of experience creating automated solutions for measuring data quality for ETL processes
• Minimum 1+ year of recent experience working with the following Azure Data Services: Azure Data Catalog, Azure Blob Storage, Azure Data Lake Storage, Azure Synapse Analytics, Azure Analysis Services, Azure Data Factory
• Strong background working with Microsoft SQL Server databases (preferably 2016)
• Understanding of Power BI and DAX functions (do not need to be a report developer or data visualization expert)
• Able to read and edit PowerShell scripts
• Prior experience working with CI/CD processes using Azure DevOps
• Strong interpersonal and customer relationship skills
• Ability to communicate to multiple levels of the organization and with non-technical partners
• Experience reading and editing Python scripts