Azure Lead Data Engineer

Mindware INC; Edison (NJ); Hybrid; USD 90,000 - 150,000; Full time

Job summary

An established industry player is seeking a skilled data engineer to tackle a large-scale project involving extensive ERP data processing. In this role, you will design and manage scalable data architectures using cutting-edge Azure technologies and Databricks. Collaborating with cross-functional teams, you'll ensure data quality and governance while optimizing workflows for performance and security. This is an exciting opportunity to contribute to impactful data solutions in a dynamic environment where your expertise will drive innovation and efficiency.

Qualifications

• 6+ years of experience with Azure Databricks and expertise in PySpark & Python development.
• Proven expertise in designing and managing scalable data architectures.

Responsibilities

• Design, build, and manage scalable data architectures using Databricks and Azure technologies.
• Optimize data workflows for performance, reliability, and security using Azure tools.

Skills

Azure Databricks

PySpark

Python

Data modeling

Data governance

Azure Synapse

Data Factory

Data pipeline design

Tools

Azure

Unity Catalog

Azure Purview

Job description

Location: Edison and Montvale, NJ (1 day in Edison, NJ & 3 days in Montvale, NJ)

Job Type: Full-Time

H1B Transfer is allowed.

VISA Type: All VISAs allowed (H1B, H4 EAD, L2, Green Card, US Citizen).

Job Description:

A large-scale project involving processing hundreds of TB of ERP data and other relevant information. Auditors utilize this processed data to provide insights and value-added services to clients. The current technology stack includes Azure data technologies and Databricks.

About the Job:

Seeking an experienced professional with at least 6+ years of hands-on experience with Databricks technology.

What Candidates Will Do:

1. Design, build, and manage scalable data architectures using Databricks and Azure data technologies.
2. Architect and implement modern data lakehouse solutions, ensuring seamless data integration, storage, and processing.
3. Collaborate with cross-functional teams to gather and translate business requirements into effective technical designs.
4. Ensure data quality and governance practices are implemented throughout the data lifecycle.
5. Optimize data workflows for performance, reliability, and security using Azure Synapse, Data Factory, and Databricks.
6. Develop and enforce best practices in data modeling, pipeline design, and storage architecture.
7. Conduct regular assessments of data systems to identify and address performance bottlenecks or areas for improvement.

Must-Have:

1. 10+ Years of total IT experience in Data engineering and Data warehouse project development.
2. 6+ years of hands-on experience with Azure Databricks and expertise in PySpark & Python development.
3. Proven expertise in designing and managing scalable data architectures.
4. 6+ years’ experience with Azure Synapse, Data Factory, and other Azure data technologies.
5. 8+ years’ experience in data modeling.
6. 6+ years experience with data pipeline design and implementation, and cloud storage architecture.
7. Deep understanding of data quality and governance practices, with hands-on experience using Unity Catalog or Azure Purview.
8. Ability to collaborate with cross-functional teams and translate business requirements into technical designs.

Nice-to-Have:

1. Familiarity with other modern data processing tools and technologies.
2. Previous experience working with large-scale data systems.

Additional Details:

• Seniority level: Mid-Senior level
• Employment type: Full-time
• Job function: Information Technology