Senior Azure Data Engineer

Impetus Technologies; United States; On-site; USD 120,000 - 180,000; Full time

Job summary

Impetus Technologies in the United States is seeking a Senior Azure Databricks engineer with 5 to 8 years of experience to design and implement scalable data pipelines using Azure Databricks and Apache Spark. You will collaborate with client and offshore teams and contribute to ETL/ELT workflows.

Responsibilities include configuring clusters for performance and cost, ensuring data quality, and building CI/CD pipelines for notebooks and jobs.

Qualifications

• 5–8 years of experience in data engineering or a related role.
• Strong experience with Azure Databricks and Spark.
• Experience building and deploying data pipelines (ETL/ELT).
• Proficiency in Python, SQL, and/or Scala; DBT is a plus.

Responsibilities

• Design and build data pipelines.
• Develop scalable data ingestion pipelines using Azure Databricks and Spark.
• Create and maintain data transformation scripts/notebooks using Spark with Python/SQL/Scala.
• Configure and optimize Databricks clusters for performance and cost.
• Develop CI/CD pipelines for notebooks, jobs, and configurations.
• Monitor pipelines and resolve performance and reliability issues.

Skills

Azure Databricks

Apache Spark

Python

SQL

Scala

CI/CD

Data Quality

Tools

Azure Databricks

Databricks notebooks

DBT

Job description

Looking for a Senior Azure Databricks engineer with 5 to 8 years of experience.

Roles and Responsibilities

• To Work with client and offshore teams.
• Design and Build Data Pipelines
• Develop scalable and efficient data ingestion pipelines using Azure Databricks and Apache Spark. Working knowledge of DBT a plus
• Data Transformation and Processing
• Create and maintain data transformation scripts and notebooks using Spark with Python, SQL, or Scala. Implement ELT/ETL workflows to prepare data for analytics and reporting. Cluster Configuration and Optimization.
• Configure Azure Databricks clusters for optimal performance, cost-efficiency, and resource utilization. Manage cluster policies, including autoscaling, instance types and runtime versions. Monitor cluster health and tune Spark configurations to improve job execution times.
• Data Quality and Validation. Implement data quality checks and validation processes within Databricks notebooks and workflows.
• Automation and CI/CD. Develop and maintain CI/CD pipelines for automated deployment of Databricks notebooks, jobs, and configurations.
• Monitoring and Troubleshooting. Monitor Databricks jobs, clusters, and pipelines for failures or performance bottlenecks. Platform usage metrics Troubleshoot and resolve issues related to data processing, cluster performance, job execution and platform related issues.
• Cost Management and Optimization. Track and analyse Databricks usage and costs. Optimize cluster configurations and usage patterns to reduce expenses. Implement policies to control resource consumption and prevent cost overruns.
• Collaboration and Support Provide technical support and guidance on Databricks platform usage and best practices.