Senior Azure Databricks Data Engineer

Impetus Technologies; United States; On-site; USD 120,000 - 180,000; Full time

Job summary

Impetus Technologies in the United States is seeking a Senior Azure Databricks engineer with 5 to 8 years of experience to design and implement scalable data pipelines using Azure Databricks and Apache Spark. You will collaborate with client and offshore teams and contribute to ETL/ELT workflows.

Responsibilities include configuring clusters for performance and cost, ensuring data quality, and building CI/CD pipelines for notebooks and jobs.

Qualifications

• 5–8 years of experience in data engineering or a related role.
• Strong experience with Azure Databricks and Spark.
• Experience building and deploying data pipelines (ETL/ELT).
• Proficiency in Python, SQL, and/or Scala; DBT is a plus.

Responsibilities

• Design and build data pipelines.
• Develop scalable data ingestion pipelines using Azure Databricks and Spark.
• Create and maintain data transformation scripts/notebooks using Spark with Python/SQL/Scala.
• Configure and optimize Databricks clusters for performance and cost.
• Develop CI/CD pipelines for notebooks, jobs, and configurations.
• Monitor pipelines and resolve performance and reliability issues.

Skills

Azure Databricks

Apache Spark

Python

SQL

Scala

CI/CD

Data Quality

Tools

Azure Databricks

Databricks notebooks

DBT

Job description

Impetus Technologies in the United States is seeking a Senior Azure Databricks engineer with 5 to 8 years of experience to design and implement scalable data pipelines using Azure Databricks and Apache Spark. You will collaborate with client and offshore teams and contribute to ETL/ELT workflows.

Responsibilities include configuring clusters for performance and cost, ensuring data quality, and building CI/CD pipelines for notebooks and jobs.