Data Bricks ETL AWS Engineer

Hexaware Technologies; Ahmedabad District, Chennai District, Bengaluru; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

Hexaware Technologies is seeking a BIG DATA LEAD to design and implement high-performance ETL/ELT pipelines using Databricks (PySpark, SQL). The role requires 9-12 years of experience and covers batch, incremental, and streaming pipelines with Kafka, and proficiency in AWS services like S3 and Glue.

You will optimize Spark workloads, set up CI/CD, enforce coding standards, and drive data governance and quality reports while acting as the main technical contact for delivery in

Qualifications

• Hands-on in building ETL/ELT pipelines using Databricks.
• Experience with batch, incremental, and streaming pipelines (Kafka).
• Proficiency in AWS services like S3 and Glue.
• Ability to optimize Spark jobs and size clusters efficiently.
• Familiarity with CI/CD, version control, and DevOps practices in Databricks.
• Experience executing data governance, data lineage, and data quality reports.

Responsibilities

• Develop/build high-performance ETL/ELT pipelines on Databricks.
• Lead and deliver sprint work as the primary technical contact.
• Review code, enforce standards, and ensure reusable solutions.
• Execute and maintain data governance, lineage, and quality reports.
• Prepare for UAT with technical tests and sign-off (Bronze layer).

Skills

ETL

Databricks

AWS

Spark optimization

CI/CD DevOps

Data governance

Data quality reports

Delivery lead

Kafka

Tools

Kafka

Job description

Skill : ETL+ DataBricks+ AWS

Exp : 9-12 yrs

Location : Chennai/ Bangalore/ Ahmedabad

BIG DATA LEAD

• Domain Expertise Clinical research Data (Good to have)
• Develop / Build high-performance and maintain ETL/ELT pipelines using Databricks (PySpark, SQL, Notebooks). Handson development of Batch and incremental pipelines, streaming pipelines (Kafka, etc) Ensure high code quality and reusability
• Proficiency in AWS (S3, Glue)
• Perform advanced Spark optimization and cluster sizing
• Implement CI/CD, version control, and DevOps best practices within Databricks.
• Execute and Maintain data governance framework, scripts and Data lineage reports & Data Quality reports
• Perform Technical Tests before UAT and provide sign-off (Bronze layer)
• Plan and execute sprint deliveries
• Review code, enforce standards
• Act as the primary technical point of contact for delivery