Senior Databricks Engineer

Aubrant Digital; Hyderabad; Presencial; INR 3.000.000 - 6.000.000; Jornada completa

Descripción de la vacante

Aubrant Digital LLC is seeking a Senior Databricks Engineer to design, build, and optimize scalable ELT pipelines for enterprise data modernization. The role focuses on Data Lakehouse architectures and high-quality data movement from diverse sources into curated layers.

Collaborate with Data Architects, Cloud Engineers, and Application teams to implement repeatable processing frameworks, enforce data quality, and operationalize Bronze/Silver/Gold datasets. Strong SQL/PySpark skills are required.

Formación

• Bachelor’s/Master’s in Computer Science, Information Systems, Engineering, or related field (or equivalent experience).
• 5–8 years of experience building data ingestion and transformation pipelines.
• Strong proficiency in SQL and PySpark for large‑scale data processing.
• Hands‑on experience with dbt for modular data transformations and documentation.
• Hands‑on experience with Databricks, Spark jobs, and Delta Lake transformations.

Responsabilidades

• Build scalable ELT pipelines using Databricks (PySpark/SQL), dbt, and Delta Lake.
• Develop Bronze, Silver, and Gold transformations aligned with enterprise modeling standards.
• Implement ingestion using tools like Airbyte, ADF, or custom connectors.
• Collaborate with Data Modelers to operationalize business rules, mappings, and canonical definitions.
• Integrate data pipelines with downstream applications (mobile apps, web apps, APIs).
• Implement data quality and validation rules, error handling, and automated remediation steps.
• Manage schema evolution, CDC ingestion, late‑arriving data, and incremental refresh patterns.
• Optimize Spark/Delta workloads through partitioning, indexing, caching, Z‑ordering, and cluster tuning.
• Develop reusable pipeline templates and frameworks for cross‑domain scalability.
• Collaborate with Cloud Engineers to ensure pipelines meet security, monitoring, and DevOps standards.
• Produce technical documentation, lineage, and metadata for governance teams.
• Support ML workloads by preparing feature datasets and inference‑ready tables.
• Participate in cross‑functional design sessions, reviews, and architectural decisions.

Conocimientos

SQL

PySpark

Databricks

Delta Lake

ELT

Airflow/Orchestration

Data quality

DBT

Educación

Bachelor's/Master's in CS/IS/Engineering

Herramientas

Databricks

dbt

Airbyte

ADF

Delta Lake

Spark

PySpark

Azure

ADLS Gen2

Synapse

Descripción del empleo

Aubrant Digital is a leader in multi-shore custom application development. We are passionate about solving our clients’ business problems through consultative teamwork, innovative software, and proven processes. We’ve served more than 50 clients and delivered hundreds of high quality, custom enterprise applications. Our clients value us as integral team members who get the job done on time and on spec, and we are proud of our high client retention rate and under 2% staff turnover. With offices in New Jersey, Boston, Costa Rica, and Eastern Europe, we execute the full software lifecycle, from architecture and design through development, QA and application maintenance & support. Our company culture emphasizes client service, trust-based relationships, and innovation.

About the role

T he Senior Databricks Engineer will design, build, and optimize scalable data ingestion and transformation pipelines to support enterprise‑grade data modernization programs. This role focuses on implementing ELT workflows within modern Data Lakehouse architectures, ensuring high‑quality, reliable, secure, and efficient movement of data from diverse source systems into curated analytical and application‑ready layers.

You will work closely with Data Architects, Data Modelers, Cloud Engineers, and Application teams to develop repeatable processing frameworks, enforce data quality rules, and operationalize Medallion‑layered datasets (Bronze/Silver/Gold or equivalents). This role is domain‑agnostic but requires the ability to translate complex business and technical requirements into performant data pipelines. Experience with high‑volume operational data domains (e.g., logistics, telematics, IoT, ERP, mobility platforms) is a plus.

Key Qualifications:

• Bachelor’s/Master’s in Computer Science, Information Systems, Engineering, or related field (or equivalent experience).
• 5–8 years of experience building data ingestion and transformation pipelines.
• Strong proficiency in SQL and PySpark for large‑scale data processing.
• Hands‑on experience with dbt (data build tool) for building, testing, and documenting modular data transformation pipelines — a core requirement for this role.
• Hands‑on experience with Databricks, Spark jobs, and Delta Lake‑based transformations.
• Experience with ELT patterns and Medallion Architectures (Bronze/Silver/Gold).
• Practical knowledge of Airbyte, ADF, or equivalent orchestration tools.
• Experience with data quality frameworks such as Great Expectations or Databricks expectations.
• Strong understanding of schema evolution, partitioning, optimization, and Delta Lake performance tuning.
• Familiarity with Azure components including ADLS Gen2, Event Hub, Synapse Serverless/Dedicated SQL.
• Experience integrating pipelines with applications and APIs (read/write patterns).
• Strong debugging skills for data integrity, ingestion failures, schema drift, and performance issues.
• Excellent communication, documentation, and cross‑team collaboration skills.

Responsibilities:

• Build scalable ELT pipelines using Databricks (PySpark/SQL), dbt, and Delta Lake.
• Develop Bronze, Silver, and Gold transformations aligned with enterprise modeling standards.
• Implement ingestion using tools like Airbyte, ADF, or custom connectors.
• Collaborate with Data Modelers to operationalize business rules, mappings, and canonical definitions.
• Integrate data pipelines with downstream applications (mobile apps, web apps, APIs).
• Implement data quality and validation rules, error handling, and automated remediation steps.
• Manage schema evolution, CDC ingestion, late‑arriving data, and incremental refresh patterns.
• Optimize Spark/Delta workloads through partitioning, indexing, caching, Z‑ordering, and cluster tuning.
• Develop reusable pipeline templates and frameworks for cross‑domain scalability.
• Collaborate with Cloud Engineers to ensure pipelines meet security, monitoring, and DevOps standards.
• Produce technical documentation, lineage, and metadata for governance teams.
• Support ML workloads by preparing feature datasets and inference‑ready tables.
• Participate in cross‑functional design sessions, reviews, and architectural decisions.