Senior/Lead Data Engineer

Dreampath Services; Hyderabad; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Dreampath Services in Hyderabad, India, seeks a Senior Data Engineer to design, build, and scale data platforms using AWS, DBT, Databricks, PySpark, Airflow, and SQL.

You will create scalable data pipelines and lakehouse architectures, collaborate with data science and analytics teams, implement data quality and testing, and drive engineering best practices.

This role emphasizes ownership, communication, and mentoring, with opportunities to influence data strategy across business units.

Qualifications

• 7+ years of data engineering experience.
• Experience designing and deploying large-scale data pipelines in production environments.
• Experience working within AWS-based data ecosystems.

Responsibilities

• Design, develop, and maintain scalable DBT models.
• Build and manage data pipelines using Databricks and PySpark.
• Develop and optimize data lakes and lakehouse architectures with Delta Lake/Apache Iceberg.
• Orchestrate end-to-end workflows using Airflow and Prefect.
• Collaborate with Data Science teams to productionize feature engineering pipelines.
• Implement data quality frameworks, automated testing, monitoring, and documentation standards.
• Optimize SQL queries, Spark jobs, and DBT transformations for performance and cost.
• Design and manage cloud-native data solutions within the AWS ecosystem.
• Participate in code reviews and promote engineering best practices.
• Build deep business understanding to ensure data products reflect operations.
• Leverage AI-powered tools to improve engineering productivity.

Skills

Communication skills

Ownership mindset

Mentoring junior engineers

Problem-solving

Analytical mindset

Tools

DBT (Data Build Tool)

AWS

Databricks

PySpark

Apache Airflow

Delta Lake

Apache Iceberg

Terraform

CI/CD for Data Engineering

Prefect

Job description

We are seeking an experienced Senior Data Engineer with strong expertise in AWS, DBT, Databricks, PySpark, Airflow, and SQL to build and scale modern data platforms. The ideal candidate will have a proven track record of designing, developing, and optimizing large-scale data pipelines and data models that support analytics, machine learning, experimentation, and business reporting.

This role requires exceptional communication skills, a strong ownership mindset, and the ability to collaborate closely with Data Scientists, Analytics teams, and business stakeholders to deliver reliable, high-quality data solutions.

Key Responsibilities

• Design, develop, and maintain scalable DBT models that deliver trusted datasets, business metrics, and analytical features.
• Build and manage data pipelines using Databricks and PySpark to transform raw operational data into analytics-ready datasets.
• Develop and optimize data lakes and lakehouse architectures using Delta Lake and/or Apache Iceberg.
• Orchestrate end‑to‑end workflows using Apache Airflow and Prefect, ensuring high reliability and SLA adherence.
• Partner with Data Science teams to design, develop, and productionize feature engineering pipelines.
• Implement data quality frameworks, automated testing, monitoring, and documentation standards.
• Optimize SQL queries, Spark jobs, and DBT transformations for performance, scalability, and cost efficiency.
• Design and manage cloud‑native data solutions within the AWS ecosystem.
• Participate in code reviews and contribute to engineering best practices.
• Build deep business understanding to ensure data products accurately represent operational processes.
• Leverage AI‑powered development tools such as Cursor, GitHub Copilot, Claude Code, or similar to improve engineering productivity.

Required Skills

Must‑Have Technologies

• DBT (Data Build Tool)
• AWS (Strong Hands‑on Experience)
• Databricks
• PySpark
• Advanced SQL

Additional Technical Skills

• Python
• Data Modeling & Data Warehousing
• Distributed Data Processing Systems
• Infrastructure as Code (Terraform, AWS CDK, or Pulumi)
• CI/CD for Data Engineering
• Data Quality & Testing Frameworks

Required Experience

• 7+ years of experience in Data Engineering.
• Strong experience designing and deploying large‑scale data pipelines in production environments.
• Hands‑on experience with distributed computing systems and concepts such as consistency, fault tolerance, throughput, and latency.
• Proven expertise in building scalable ELT/ETL frameworks using DBT, Databricks, Spark, and Airflow.
• Experience working within AWS‑based data ecosystems.
• Exposure to modern lakehouse architectures and big data technologies.
• Experience supporting analytics, business intelligence, and machine learning workloads.

Preferred Qualifications

• Experience with Kinesis, EMR, Sigma, or Pulumi.
• Exposure to supply chain, logistics, transportation, or operations‑focused data domains.
• Familiarity with feature stores, MLOps, and production ML pipelines.
• Experience working in fast‑paced, high‑growth environments.

Soft Skills

• Outstanding verbal and written communication skills.
• Strong stakeholder management and collaboration abilities.
• Self‑driven, proactive, and capable of working independently.
• Strong problem‑solving and analytical mindset.
• Ability to quickly learn and adapt to new technologies.
• Comfortable mentoring junior engineers and driving engineering excellence.

Keywords

AWS, DBT, Databricks, PySpark, Airflow, SQL, Delta Lake, Apache Iceberg, Data Engineering, Data Modeling, ETL, ELT, Spark, Terraform, Data Warehouse, Big Data, Python, Analytics Engineering, Lakehouse Architecture