Senior Databricks Data Engineer - PySpark - Bangalore / Pune

Datavruti; Bengaluru; Hybrid; INR 2,250,000 - 2,750,000; Full time

Job summary

Datavruti is seeking an experienced PySpark ETL Developer with 4+ years of experience to design, develop, and optimize enterprise data pipelines. The role focuses on PySpark, Spark SQL, Databricks, and Snowflake, delivering scalable data engineering solutions for BI and analytics projects.

You will work closely with business stakeholders and cross-functional teams to build high-performance pipelines, handle data from multiple sources, and ensure data quality across lakes and warehouses.

Qualifications

• 4+ years of experience as a Data Engineer or ETL Developer.
• Strong hands-on experience with PySpark, Apache Spark, Spark SQL, and Python.
• Expertise in Databricks.
• Strong understanding of ETL design, data transformation, and data integration.
• Experience with Data Lakes, Data Warehouses, and Big Data technologies.
• Knowledge of distributed computing, parallel processing, and data partitioning concepts.
• Strong SQL skills with experience in performance tuning and query optimization.
• Experience working with batch and streaming data processing.
• Excellent analytical, debugging, and problem-solving skills.
• Strong verbal and written communication skills.

Responsibilities

• Design, develop, and maintain scalable ETL pipelines using PySpark and Spark SQL.
• Collaborate with stakeholders to gather business requirements and translate them into efficient data engineering solutions.
• Extract data from multiple sources, including databases, APIs, data lakes, files, and streaming platforms.
• Transform, cleanse, and validate data using PySpark to ensure high data quality and consistency.
• Develop and optimize Spark jobs for performance, scalability, and efficient resource utilization.
• Build and maintain batch and streaming data pipelines to support real-time and near real-time processing.
• Load transformed data into data lakes, data warehouses, and analytical platforms.
• Implement robust error handling, logging, monitoring, and troubleshooting mechanisms for ETL workflows.
• Document ETL processes, data lineage, transformation logic, and technical specifications.
• Develop unit, integration, and performance tests to ensure reliable and accurate data processing.
• Collaborate with cross-functional teams to deliver scalable, secure, and high-quality data solutions.

Skills

PySpark

Apache Spark

Python

SQL

Databricks

ETL design

Data pipelines

Data lakes/warehouses

Data quality

Debugging

Tools

Snowflake

Job description

Hiring for: A technology company delivering enterprise data engineering and analytics solutions, with a strong focus on modern cloud data platforms and Databricks.

Positions: 10

Experience: 4 to 7 years

Type: On-site, Remote / Permanent

Salary: Up to INR 25 LPA

Notice Period: Immediate to 30 days

Must Have: Databricks Development, Pyspark, Python, SQL, Unity Catalogs,

Job Summary

We are seeking an experienced PySpark ETL Developer with 4+ years of experience in designing, developing, and optimizing enterprise ETL pipelines using PySpark. The ideal candidate should have strong expertise in Python, Apache Spark, Databricks, and Snowflake, along with hands-on experience in processing large-scale data and building scalable data engineering solutions. You will work closely with business stakeholders and cross-functional teams to develop high-performance data pipelines that support business intelligence and analytics initiatives.

Key Responsibilities

• Design, develop, and maintain scalable ETL pipelines using PySpark and Spark SQL.
• Collaborate with stakeholders to gather business requirements and translate them into efficient data engineering solutions.
• Extract data from multiple sources, including databases, APIs, data lakes, files, and streaming platforms.
• Transform, cleanse, and validate data using PySpark to ensure high data quality and consistency.
• Develop and optimize Spark jobs for performance, scalability, and efficient resource utilization.
• Build and maintain batch and streaming data pipelines to support real-time and near real-time processing.
• Load transformed data into data lakes, data warehouses, and analytical platforms.
• Implement robust error handling, logging, monitoring, and troubleshooting mechanisms for ETL workflows.
• Document ETL processes, data lineage, transformation logic, and technical specifications.
• Develop unit, integration, and performance tests to ensure reliable and accurate data processing.
• Collaborate with cross-functional teams to deliver scalable, secure, and high-quality data solutions.

Required Skills

• 4+ years of experience as a Data Engineer or ETL Developer.
• Strong hands-on experience with PySpark, Apache Spark, Spark SQL, and Python.
• Expertise in Databricks.
• Strong understanding of ETL design, data transformation, and data integration.
• Experience with Data Lakes, Data Warehouses, and Big Data technologies.
• Knowledge of distributed computing, parallel processing, and data partitioning concepts.
• Strong SQL skills with experience in performance tuning and query optimization.
• Experience working with batch and streaming data processing.
• Excellent analytical, debugging, and problem-solving skills.
• Strong verbal and written communication skills.

Preferred Skills

• Experience with cloud platforms such as AWS or Azure.
• Knowledge of Hadoop ecosystem and modern data engineering technologies.
• Experience with API integration and data ingestion from multiple data sources.
• Familiarity with CI/CD pipelines and Agile development methodologies.