Python Data Engineer

Wissen Technology; Pune District, Mumbai, Bengaluru; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Wissen Technology in Pune, India seeks a seasoned Data Engineer to design and operate scalable data pipelines. You will work with Python, PySpark, Databricks, and Delta Lake to build robust ETL/ELT processes, ensure data quality, and optimize performance.

The role involves Airflow orchestration, Kubernetes deployments, and close collaboration with cross-functional teams. Ideal candidates bring strong SQL, Git, and Spark expertise, with hands-on experience in data lakehouse environments and CI/CD

Qualifications

• Experience building scalable ETL/ELT data pipelines.
• Proficient in Python data manipulation, exception handling, and debugging.
• Strong knowledge of Spark, Delta Lake, and data governance.
• Experience with Airflow DAGs, scheduling, retries, and troubleshooting.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python and PySpark.
• Work with Databricks and Delta Lake including Delta tables, MERGE, schema evolution, Time Travel, and data optimization.
• Develop, schedule, and monitor Airflow DAGs for workflow orchestration and dependency management.
• Collaborate with Kubernetes for containerized deployment, orchestration, and troubleshooting.
• Write reusable, optimized Python code following software engineering best practices.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement data quality checks, error handling, logging, and monitoring mechanisms.
• Use Git for version control, branching, PRs, code reviews, and collaboration.
• Troubleshoot pipeline failures, investigate data issues, and implement reliable solutions.
• Collaborate with cross-functional teams to understand business requirements and deliver data solutions.

Skills

Python

PySpark

Databricks

Delta Lake

Apache Airflow

Kubernetes

Git

SQL

Tools

Docker

CI/CD pipelines

Unity Catalog

Job description

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python and PySpark.
• Work extensively with Databricks and Delta Lake, including Delta tables, MERGE operations, schema evolution, Time Travel, and data optimization.
• Develop, schedule, and monitor Apache Airflow DAGsfor workflow orchestration and dependency management.
• Work with Kubernetesfor containerized application deployment, orchestration, monitoring, and troubleshooting.
• Develop reusable and optimized Python code following software engineering best practices.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement data quality checks, error handling, logging, and monitoring mechanisms.
• Use Gitfor version control, branching, pull requests, code reviews, and collaborative development.
• Troubleshoot pipeline failures, investigate data issues, and implement reliable solutions.
• Collaborate with cross-functional teams to understand business requirements and deliver efficient data solutions.

Required Technical Skills

• Python:Strong programming skills, including data manipulation, exception handling, modular programming, and debugging.
• PySpark:Hands-on experience with Spark DataFrames, Spark SQL, joins, transformations, partitioning, and performance tuning.
• Databricks:Strong understanding of Databricks architecture, notebooks, workflows, compute management, and job execution.
• Delta Lake:Experience with Delta tables, ACID transactions, MERGE/UPSERT, schema enforcement, schema evolution, VACUUM, and Time Travel.
• Apache Airflow:Experience developing DAGs, managing task dependencies, configuring retries, scheduling workflows, and troubleshooting failures.
• Kubernetes:Practical knowledge of pods, deployments, services, ConfigMaps, Secrets, resource management, and troubleshooting containerized workloads.
• Git:Proficiency in branching strategies, pull requests, merging, and resolving conflicts.
• SQL:Strong SQL skills, including complex joins, CTEs, window functions, and query optimization.

Good to Have

• Experience with Dockerand containerization best practices.
• Experience with Databricks Unity Catalog and data governance.
• Knowledge of CI/CD pipelines and automated deployments.
• Experience deploying and managing Airflow workloads on Kubernetes.
• Familiarity with Kubernetes monitoring, autoscaling, and resource optimization.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Understanding of data lakehouse architecture and data modeling.
• Experience with batch and incremental data processing.
• Familiarity with data quality frameworks and automated testing.

Preferred Candidate Profile

• Strong analytical and problem-solving skills.
• Ability to independently design, develop, deploy, and troubleshoot data pipelines.
• Good understanding of distributed data processing and container orchestration.
• Experience working in an Agile development environment.
• Strong communication and collaboration skills.