Lead AWS Data Engineer -Databricks

Ascendion Engineering; Bengaluru; Hybrid; INR 2,500,000 - 4,200,000; Full time

Job summary

Ascendion Engineering is seeking a Senior Databricks Engineer in Bengaluru to design, build, and optimize data pipelines and platforms for advanced analytics. You will lead a team of data engineers, collaborating with cross-functional stakeholders to deliver scalable, secure, and high-performing data solutions.

The role requires strong Databricks expertise, SQL proficiency, and extensive AWS experience, with responsibilities spanning design, governance, and BI delivery.

Qualifications

• 6-8 years of experience in data engineering, with at least 3 years in Databricks.
• Deep hands-on expertise in: PySpark, Spark SQL, Delta Lake, Unity Catalog.
• Strong proficiency in SQL for data modeling and query optimization.
• Experience with AWS services (S3, Redshift, Glue, Lambda, Kinesis, etc.)

Responsibilities

• Design, build, and optimize data pipelines and platforms for analytics and BI.
• Lead design, development, and maintenance of ETL/ELT using Databricks and Airflow.
• Build reusable frameworks, templates, and standards for pipelines and data quality.
• Ensure cloud-native designs aligned with AWS platform best practices.
• Mentor and guide junior data engineers and collaborate with stakeholders.

Skills

Databricks

PySpark

Spark SQL

Delta Lake

Unity Catalog

Databricks Workflows

Delta Live Tables

SQL Warehouses

Photon

Streaming

Batch processing

Real-time architectures

Elastic Beanstalk

Kubernetes

Java

Scala

Python

Kinesis

Datadog

SQL

AWS services

Tools

Airflow

Databricks

AWS Lambda

Job description

Databricks Senior Engineer Job Description

Role Overview

The Data Senior Engineer will be responsible for designing, building, and optimizing data pipelines and platforms to enable advanced analytics and business intelligence across the organization. This role requires strong expertise in Databricks, SQL, AWS, and Airflow, combined with a solid foundation in data analysis and engineering best practices. The ideal candidate will lead a team of data engineers, collaborate with cross-functional stakeholders, and ensure scalable, secure, and high-performing data solutions.

Location: Bengaluru

Key Responsibilities

Technical Design

• Build design specifications, data models, pipeline patterns, and solution architectures for the workstream.
• Lead the design, development, and maintenance of data pipelines and ETL workflows using Databricks and Airflow.
• Build reusable frameworks, templates, and engineering standards for ETL/ELT, streaming, and batch pipelines.
• Ensure cloud-native designs aligned with AWS platform best practices.

Governance, Security & Optimization

• Ensure data quality, integrity, and governance across data platforms.
• Optimize SQL queries and data models for performance and scalability.
• Build standards for Unity Catalog, lineage, metadata, glossary, and security policies.
• Build data quality frameworks, observability, and operational monitoring.

Cross-Functional Collaboration

• Work closely with stakeholders to understand data requirements and translate them into technical solutions.
• Collaborate with analytics and business teams to deliver actionable insights through BI dashboards and reports.
• Support estimations, and sizing for new programs and migrations
• Experience in engaging and leading technical discussions with business stakeholders

AI/LLM Enablement (Advantage)

• Design frameworks for agentic workflows, GenAI-based automation, code generation, data quality validation, and metadata enrichment.
• Build Databricks Model Serving, Vector Search, and AI/BI integration.

Required Skills & Experience

• 6-8 years of experience in data engineering, with at least 3 years in Databricks.
• Deep hands-on expertise in:
• PySpark, Spark SQL, Delta Lake, Unity Catalog
• Databricks Workflows, Delta Live Tables (DLT), SQL Warehouses, Photon
• Streaming, batch, and real-time architectures
• Elastic bean stack, Kubernetes, Java, Scala pipelines, Python, Kinesis, Datadog
• Expertise in Databricks for big data processing and analytics.
• Strong proficiency in SQL for data modeling and query optimization.
• Solid understanding of AWS services (S3, Redshift, Glue, Lambda, SQS, SNS etc.) for data engineering.
• Strong analytical and problem-solving skills with attention to detail.
• Excellent communication and leadership skills to manage teams and stakeholders.
• Mentor and guide junior data engineers, fostering a culture of best practices and continuous improvement.

Preferred Qualifications

• Databricks Certified Data Engineer Professional or Databricks Architect certification.
• Experience with Databricks AI/BI, Lakehouse Federation, and Airflow.
• Familiarity with Airflow DAGs.
• AWS certification in data analytics or related specialization.
• Knowledge of LLMs, agentic frameworks, or GenAI automation (strong advantage).