Data Engineer

Jobtailor; Pune District; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

Jobtailor seeks a Senior Data Engineer to design, build, and optimize scalable ETL/ELT pipelines on AWS using Python, PySpark, and Databricks. You will own data platform components, implement governance, and drive cost and performance improvements.

You will collaborate with product managers and data scientists, implement CI/CD for data deployments with Databricks Asset Bundles, and advance state-of-the-art data engineering practices in a startup environment.

Qualifications

• Extensive experience in building data pipelines and data platforms.
• Proficiency with Python, SQL, PySpark for automation and analytics.
• Hands-on with Databricks on AWS and DBT.
• Experience with CI/CD for data deployments.

Responsibilities

• Develop, test, and maintain scalable ETL/ELT pipelines.
• Lead data feature development from design to deployment.
• Implement data governance, quality, and security practices.
• Optimize cost and performance of data platforms.
• Collaborate across product, data science, and engineering teams.
• Monitor workloads and implement dashboards for cost control.

Skills

Data Engineering

Databricks & DBT

SQL & Python

Terraform & Airflow

Databricks Certification

Tools

Databricks Asset Bundles

Spark Structured Streaming

Delta Lake

Unity Catalog

CI/CD

Job description

• Develop, test, and maintain high-quality, scalable ETL/ELT data pipelines using Python, PySpark, and Databricks on AWS
• Lead development of new data features and services across the full product lifecycle, from architectural design to deployment
• Use Terraform, Spark Structured Streaming, and DLT
• Create CI/CD-controlled data engineering deployments using Databricks Asset Bundles
• Leverage Databricks serverless compute and cost optimization techniques to deliver performant data products
• Collaborate with Product Managers, Data Scientists, and fellow engineers to solve complex data challenges
• Ensure data quality, integrity, and security throughout the data lifecycle
• Implement data governance best practices
• Take technical ownership of major data platform architecture components and lead design discussions
• Analyze workloads, queries, clusters/warehouses, jobs, and services to identify inefficiencies and savings opportunities
• Implement continuous monitoring and intelligence dashboards for cost and performance optimization
• Stay current with new Snowflake/Databricks features and translate them into marketable product differentiators
• Spend approximately 70% of time on data engineering projects and 30% on data operations

Requirements

• Extensive experience with data engineering
• DataBricks & DBT (Mandatory)
• Proven expertise in Databricks clusters, jobs, Delta Lake, Spark optimization, autoscaling, and Unity Catalog
• Strong SQL + Python skills for automation, monitoring, and recommendation logic
• Experience with frameworks like Terraform & Airflow
• Track record of delivering performance gains and cost savings through platform-level optimizations
• Hands-on, automation-driven, customer-value-obsessed startup mindset
• Databricks Certification (preferred but not mandatory)
• Familiarity with compliance frameworks like SOC2, GDPR (preferred but not mandatory)
• Knowledge of LLMs and AI frameworks (preferred but not mandatory)

Core Competencies

Demonstrates expertise in developing and maintaining scalable ETL/ELT data pipelines using Python, PySpark, and Databricks on AWS, while ensuring data quality and implementing governance best practices. Proven ability to lead architectural design and deployment of data features, optimizing performance and cost through advanced data engineering techniques.

Highest-signal resume keywords

• Data Engineering
• Databricks & DBT
• SQL & Python
• Terraform & Airflow
• Databricks Certification

ATS Optimization Keywords

Hard Skills

• ETL/ELT Development
• Python Programming
• PySpark
• SQL
• Databricks
• Spark Optimization
• Terraform
• Airflow
• Data Governance
• Cost Optimization

Soft Skills

• Collaboration
• Problem-Solving
• Customer-Value Focused
• Leadership

Certifications & Qualifications

• Databricks Certification

Industry Keywords

• SOC2
• GDPR
• Data Quality
• Data Integrity
• Data Security

Tools & Technologies

• Databricks Asset Bundles
• Spark Structured Streaming
• Delta Lake
• Unity Catalog
• CI/CD