Python DE

Wissen Technology; Pune District, Mumbai, Bengaluru; Hybrid; INR 1,200,000 - 2,100,000; Full time

Job summary

Wissen Technology in Pune is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines using Python, PySpark, and Databricks. You will work with Delta Lake, develop Airflow DAGs, and manage Kubernetes deployments.

The role focuses on performance tuning, data quality, and collaboration with cross-functional teams to deliver robust data solutions in a fast-paced environment.

Qualifications

• Strong Python programming for data manipulation, exception handling, modular design and debugging.
• Hands-on PySpark with DataFrames, Spark SQL, joins, transformations, partitioning, and tuning.
• Deep knowledge of Databricks architecture, notebooks, workflows, compute management and job execution.
• Experience with Delta Lake tables, ACID, MERGE/UPSERT, schema evolution, VACUUM, and Time Travel.
• Airflow DAG development for orchestrating workflows, retries, scheduling and failure handling.
• Kubernetes knowledge for containerized workloads, deployments, services, and resource management.
• Proficiency in Git for version control, review processes, and collaboration.
• Strong SQL skills including complex joins, CTEs, window functions and optimization.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python and PySpark.
• Work with Databricks and Delta Lake, including MERGE, schema evolution and Time Travel.
• Develop, schedule, and monitor Airflow DAGs for workflow orchestration.
• Use Kubernetes for containerized deployment, orchestration, monitoring, and troubleshooting.
• Write reusable Python code following software engineering best practices.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement data quality checks, error handling, logging, and monitoring mechanisms.
• Collaborate with cross-functional teams to understand business requirements and deliver data solutions.

Skills

Python

PySpark

Databricks

Delta Lake

Apache Airflow

Kubernetes

Git

SQL

Tools

Docker

Databricks Unity Catalog

CI/CD

Airflow on Kubernetes

Kubernetes Monitoring

Job description

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python and PySpark.
• Work extensively with Databricks and Delta Lake, including Delta tables, MERGE operations, schema evolution, Time Travel, and data optimization.
• Develop, schedule, and monitor Apache Airflow DAGsfor workflow orchestration and dependency management.
• Work with Kubernetesfor containerized application deployment, orchestration, monitoring, and troubleshooting.
• Develop reusable and optimized Python code following software engineering best practices.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement data quality checks, error handling, logging, and monitoring mechanisms.
• Use Gitfor version control, branching, pull requests, code reviews, and collaborative development.
• Troubleshoot pipeline failures, investigate data issues, and implement reliable solutions.
• Collaborate with cross-functional teams to understand business requirements and deliver efficient data solutions.

Required Technical Skills

• Python:Strong programming skills, including data manipulation, exception handling, modular programming, and debugging.
• PySpark:Hands-on experience with Spark DataFrames, Spark SQL, joins, transformations, partitioning, and performance tuning.
• Databricks:Strong understanding of Databricks architecture, notebooks, workflows, compute management, and job execution.
• Delta Lake:Experience with Delta tables, ACID transactions, MERGE/UPSERT, schema enforcement, schema evolution, VACUUM, and Time Travel.
• Apache Airflow:Experience developing DAGs, managing task dependencies, configuring retries, scheduling workflows, and troubleshooting failures.
• Kubernetes:Practical knowledge of pods, deployments, services, ConfigMaps, Secrets, resource management, and troubleshooting containerized workloads.
• Git:Proficiency in branching strategies, pull requests, merging, and resolving conflicts.
• SQL:Strong SQL skills, including complex joins, CTEs, window functions, and query optimization.

Good to Have

• Experience with Dockerand containerization best practices.
• Experience with Databricks Unity Catalog and data governance.
• Knowledge of CI/CD pipelines and automated deployments.
• Experience deploying and managing Airflow workloads on Kubernetes.
• Familiarity with Kubernetes monitoring, autoscaling, and resource optimization.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Understanding of data lakehouse architecture and data modeling.
• Experience with batch and incremental data processing.
• Familiarity with data quality frameworks and automated testing.

Preferred Candidate Profile

• Strong analytical and problem-solving skills.
• Ability to independently design, develop, deploy, and troubleshoot data pipelines.
• Good understanding of distributed data processing and container orchestration.
• Experience working in an Agile development environment.
• Strong communication and collaboration skills.