Data Engineer

Acme Services Private Limited; Bengaluru; On-site; INR 1,500,000 - 2,600,000; Full time

Job summary

Acme Services Private Limited is seeking a skilled Data Engineer in Bengaluru to design and maintain scalable ETL/ELT pipelines using Databricks, PySpark, and Python. You will build data ingestion frameworks, implement Medallion architecture, and develop Delta Lake solutions emphasizing quality and governance.

The role involves optimizing Spark workloads, orchestrating workflows with ADF, and supporting CI/CD with Git and Azure DevOps, while collaborating with analysts and data scientists to

Qualifications

• Experience building ETL/ELT pipelines with Databricks and PySpark.

Responsibilities

• Develop and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Spark SQL, and Python.
• Build data ingestion frameworks for structured and unstructured data sources.
• Design data transformation processes using Medallion Architecture (Bronze, Silver, Gold).
• Develop Delta Lake solutions with focus on data quality, reliability, and governance.
• Develop and maintain Databricks Notebooks, Jobs, and Workflows.
• Optimize Spark jobs and Databricks workloads for performance and cost efficiency.
• Orchestrate data workflows using Azure Data Factory (ADF) and Databricks Workflows.
• Implement data validation, reconciliation, monitoring, and exception handling.
• Support CI/CD pipelines using Git and Azure DevOps.
• Participate in solution design, code reviews, and production support.
• Collaborate with business analysts, data scientists, and application teams to deliver analytical datasets.

Skills

Databricks

PySpark

Spark SQL

Python

ETL/ELT

Delta Lake

Data governance

Azure Data Factory

Git

Azure DevOps

Tools

Azure Data Factory

Azure DevOps

Git

Job description

Job Description:

• Develop and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Spark SQL, and Python.


• Build data ingestion frameworks for structured and unstructured data sources.


• Design data transformation processes using Medallion Architecture (Bronze, Silver, Gold).


• Develop Delta Lake solutions with focus on data quality, reliability, and governance.


• Develop and maintain Databricks Notebooks, Jobs, and Workflows.


• Optimize Spark jobs and Databricks workloads for performance and cost efficiency.


• Orchestrate data workflows using Azure Data Factory (ADF) and Databricks Workflows.


• Implement data validation, reconciliation, monitoring, and exception handling.


• Support CI/CD pipelines using Git and Azure DevOps.


• Participate in solution design, code reviews, and production support.


• Collaborate with business analysts, data scientists, and application teams to deliver analytical datasets.