Data Engineer

Milestone Technologies, Inc.; Hyderabad; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Milestone Technologies is seeking a Data Engineer to support an EDW migration from on-prem SQL Server 2016 to Azure Cloud. You will focus on data quality, establish data pipes, and work with analysts to define quality thresholds.

You will design automated processes to measure quality, analyze existing ETL pipelines, and contribute to dashboards with report developers. You will collaborate with data specialists to evolve data quality templates, profile data sources, and support CI/CD for data

Qualifications

• Bachelor's degree in a technical field.
• 5+ years designing, building, and supporting data pipes.
• 2+ years creating automated data quality solutions for ETL.
• 1+ year experience with Azure Data Services (Azure Data Factory, Synapse, Data Lake).
• Strong SQL Server (2016) background and scripting in PowerShell/Python.

Responsibilities

• Translate data requirements for new data quality controls and thresholds into technical designs and specifications.
• Create automated processes to measure data quality, failures, and errors.
• Analyze existing systems and data pipes to identify improvements.
• Evolve data quality processes and templates.
• Profile data sources to understand quality issues in ETL processes.
• Collaborate on data modeling using Kimball dimensional modeling practices.
• Aggregate data from multiple sources and enhance data quality and reliability.
• Develop analytical tools to measure and report data quality.
• Assist CI/CD framework (Azure DevOps) to improve data quality.
• Support Senior BI Engineer and Senior Data Engineer on POCs.

Skills

Data engineering

Data quality measurement

Python scripting

PowerShell scripting

Stakeholder collaboration

CI/CD concepts

Education

Bachelor's degree in Engineering or Computer Science

Tools

Azure Data Factory

Azure Synapse Analytics

Azure Data Lake Storage

Azure Data Catalog

Azure Blob Storage

Power BI

DAX functions

SQL Server 2016

Azure DevOps

Job description

Company Overview

Milestone Technologies is a global IT managed services firm that partners with organizations to scale their technology, infrastructure and services to drive specific business outcomes such as digital transformation, innovation, and operational agility.

Milestone is focused on building an employee-first, performance-based culture and for over 25 years, we have a demonstrated history of supporting category-defining enterprise clients that are growing ahead of the market. The company specializes in providing solutions across Application Services and Consulting, Digital Product Engineering, Digital Workplace Services, Private Cloud Services, AI/Automation, and ServiceNow. Milestone culture is built to provide a collaborative, inclusive environment that supports employees and empowers them to reach their full potential.

Our seasoned professionals deliver services based on Milestone’s best practices and service delivery framework. By leveraging our vast knowledge base to execute initiatives, we deliver both short-term and long-term value to our clients and apply continuous service improvement to deliver transformational benefits to IT. With Intelligent Automation, Milestone helps businesses further accelerate their IT transformation. The result is a sharper focus on business objectives and a dramatic improvement in employee productivity. Through our key technology partnerships and our people-first approach, Milestone continues to deliver industry-leading innovation to our clients. With more than 3,000 employees serving over 200 companies worldwide, we are following our mission of revolutionizing the way IT is deployed.

Job Overview

Milestone Technologies is looking for a Data Engineer to support a client's EDW migration from on-prem (SQL Server 2016) to the Azure Cloud. This role will focus on the data quality and errors for the new data pipes established from the source to the new target cloud database. You will work closely with the team that is designing and building the data pipes as well as the lead business analyst who is working with internal stakeholders defining the data quality requirements and thresholds. You will be responsible for designing and building the processes for measuring data quality, data transmission failures, and data load errors. In addition to this, you will be expected to work with report developers to create dashboards and reports showcasing the data quality.

• Translate data requirements for new data quality controls and thresholds into technical designs and specifications
• Create automated processes to measure data quality, failures, and errors
• Analyze existing systems and data pipes to identify areas for improvement or opportunities to enhance
• Continue to help evolve and improve the data quality processes and templates
• Profile data sources to understand existing quality issues in the ETL processes
• Collaborate on data modeling tasks including recommending and evaluating designs based on Kimball dimensional data modeling practices
• Aggregate and combine raw information from different data sources
• Explore ways to enhance data quality and reliability including helping with the development of support and remediation processes
• Identify opportunities for data acquisition
• Develop analytical tools and processes to measure and report data quality
• Help adjust and tune existing CI/CD framework and processes (Azure DevOps) as needed to improve data quality
• Assist Senior BI Engineer and Senior Data Engineer on innovative technology POCs

Skills

• Bachelor's Degree or higher in a technical field such as Engineering, Computer Science, etc.
• Minimum 5+ years of experience designing, building, and supporting data pipes
• Minimum 2+ years of experience creating automated solutions for measuring data quality for ETL processes
• Minimum 1+ year of recent experience working with the following Azure Data Services: Azure Data Catalog, Azure Blob Storage, Azure Data Lake Storage, Azure Synapse Analytics, Azure Analysis Services, Azure Data Factory
• Strong background working with Microsoft SQL Server databases (preferably 2016)
• Understanding of Power BI and DAX functions (do not need to be a report developer or data visualization expert)
• Able to read and edit PowerShell scripts
• Prior experience working with CI/CD processes using Azure DevOps
• Strong interpersonal and customer relationship skills
• Ability to communicate to multiple levels of the organization and with non-technical partners
• Experience reading and editing Python scripts

Compensation

Estimated Pay Range:

Our Commitment to Diversity & Inclusion

At Milestone we strive to create a workplace that reflects the communities we serve and work with, where we all feel empowered to bring our full, authentic selves to work. We know creating a diverse and inclusive culture that champions equity and belonging is not only the right thing to do for ouremployees butis also critical to our continued success.

Milestone Technologies provides equal employment opportunity for all applicants and employees. All qualified applicants will receive consideration for employment and will not be discriminated against on the basis of race, color, religion, gender, gender identity, marital status, age, disability, veteran status, sexual orientation, national origin, or any other category protected by applicable federal and state law, or local ordinance. Milestone also makes reasonable accommodations for disabled applicants and employees.

We welcome the unique background, culture, experiences, knowledge, innovation, self-expression and perspectives you can bring to our global community. Our recruitment team is looking forward to meeting you.