Azure Data Engineer

TechWish; New York (NY); On-site; USD 140,000 - 190,000; Full time

Job summary

TechWish in New York is seeking a senior Data Engineer to design and deliver scalable data platforms and pipelines across the SDLC. You will build robust data solutions with a focus on performance, reliability, and governance.

Leveraging Azure Databricks, Delta Lake, and Delta Live Tables, you will implement lakehouse architectures, ensure data quality, and optimize costs for enterprise-scale workloads. The role requires strong SQL, Python, and Spark skills and collaboration across distributed

Qualifications

• 6–10 years of experience in Data Engineering, Application Development, System Integration, or EDW projects.
• Proven expertise building scalable data pipelines on Azure Databricks with Delta Lake and DLT.
• Experience developing batch and streaming pipelines with PySpark, Spark SQL, and Python.
• Strong SQL programming and data modeling for relational, dimensional, and lakehouse architectures.
• Knowledge of Delta Lake features: ACID, schema evolution, time travel, CDC, and data quality.
• Experience with data quality, monitoring, and orchestration using Delta Live Tables, Databricks Workflows, and Azure services.

Skills

Data engineering

Python

PySpark

SQL

Azure Databricks

Delta Lake

Delta Live Tables

Data modeling

SDLC & CI/CD

Cost optimization

Education

Bachelor's degree in CS/IT/Engineering or related

Tools

Azure Databricks

Delta Lake

Delta Live Tables

Medallion Architecture

Spark SQL

Job description

• Bachelor's degree in computer science, Information Technology, Engineering, or a related discipline, with 6-10 years of experience in Data Engineering, Application Development, System Integration, or Enterprise Data Warehouse projects.
• Strong hands-on experience designing and implementing modern data platforms on Azure Databricks, leveraging Delta Lake, Delta Live Tables (DLT), and Medallion Architecture (Bronze, Silver, Gold) to build scalable and reliable data pipelines.
• Proven expertise in developing and optimizing batch and streaming data pipelines using PySpark, Spark SQL, and Python, with a strong focus on performance tuning, scalability, and cost optimization.
• Extensive experience in Databricks job performance optimization, including query tuning, partitioning strategies, data skew mitigation, caching, cluster sizing, workload optimization, and Spark execution plan analysis.
• Strong proficiency in SQL programming and data modeling concepts, including relational, dimensional, and lakehouse data architectures.
• Deep understanding of Delta Lake features such as ACID transactions, schema evolution, time travel, change data capture (CDC), and data quality enforcement.
• Experience implementing data quality, monitoring, and orchestration frameworks using Delta Live Tables expectations, Databricks Workflows, and Azure-native services.
• Strong exposure to the full Software Development Life Cycle (SDLC), including design, development, testing, CI/CD, deployment, and production support.
• Solid understanding of Lakehouse architecture, cloud-based analytics platforms, data governance, and data engineering best practices.
• Knowledge of real-time data processing, data integration patterns, and enterprise-scale data migration initiatives is a strong advantage.
• Demonstrated ability to stay current with evolving cloud and big data technologies and contribute effectively to architecture discussions, technology strategy, and platform modernization initiatives.
• Self-motivated, proactive, and capable of working independently in a fast-paced environment while collaborating with globally distributed teams.
• Strong analytical, problem-solving, communication, and stakeholder management skills.