Python, SQL & Spark Engineer (Solutions Team)

Datagaps; Hyderabad; Hybrid; INR 1,500,000 - 3,000,000; Full time

Job summary

Datagaps is seeking a Senior Data Engineer / Solutions Engineer in Hyderabad (Hybrid) to design, develop, and implement scalable data solutions using Python, SQL and Apache Spark. You will work with Databricks and Snowflake and collaborate with business users to translate requirements into technical outcomes.

You will optimize Spark jobs and SQL for performance, support data migration, quality, and analytics initiatives, and ensure governance and security standards are followed in a fast-paced,

Qualifications

• : Strong hands-on experience in Python
• : Advanced proficiency in SQL
• : Extensive experience with Apache Spark including Spark DataFrames, Spark SQL, performance tuning, and distributed data processing
• : Strong understanding of ETL/ELT development and data integration
• : Experience with large-scale structured and semi-structured datasets
• : Expertise in data analysis, troubleshooting, and performance optimization
• : Strong problem-solving and stakeholder communication skills

Responsibilities

• Develop and maintain scalable data pipelines using Python, SQL, and Spark
• Work closely with business users, architects, and client stakeholders to understand requirements and translate them into technical solutions
• Design and implement data ingestion, transformation, and integration processes
• Optimize Spark jobs and SQL queries for performance and scalability
• Support data migration, data quality, and reporting initiatives
• Troubleshoot production issues and ensure timely resolution
• Participate in solution design discussions and contribute to best practices
• Ensure data governance, security, and quality standards are followed

Skills

Python

SQL

Apache Spark

ETL/ELT

Data analysis

Stakeholder communication

Tools

Databricks

Snowflake

Azure

AWS

GCP

Data Lake concepts

Data Warehouse concepts

Job description

Python, SQL & Spark Engineer (Solutions Team)

Job Title: Senior Data Engineer / Solutions Engineer
Experience: 4-10 Years
Location: Hyderabad (Hybrid)

Role Overview

We are seeking a highly skilled Python, SQL, and Apache Spark Engineer to join our Solutions Team. The ideal candidate will be responsible for designing, developing, and implementing scalable data solutions that address business and client requirements. This role requires strong expertise in data processing, ETL development, data transformation, and analytics support. Experience with Databricks and Snowflake is preferred but not mandatory.

Mandatory Skills

• Strong hands-on experience in Python
• Advanced proficiency in SQL
• Extensive experience with Apache Spark
  • Spark DataFrames
  • Spark SQL
  • Performance Tuning
  • Distributed Data Processing
• Strong understanding of ETL/ELT development and data integration
• Experience working with large-scale structured and semi-structured datasets
• Expertise in data analysis, troubleshooting, and performance optimization
• Strong problem-solving and stakeholder communication skills

Preferred Skills

• Experience with Databricks
• Experience with Snowflake
• Exposure to cloud platforms such as Azure, AWS, or GCP
• Understanding of Data Lake and Data Warehouse concepts
• Experience in data migration, modernization, and transformation initiatives

Key Responsibilities

• Develop and maintain scalable data pipelines using Python, SQL, and Spark.
• Work closely with business users, architects, and client stakeholders to understand requirements and translate them into technical solutions.
• Design and implement data ingestion, transformation, and integration processes.
• Optimize Spark jobs and SQL queries for performance and scalability.
• Support data migration, data quality, and reporting initiatives.
• Troubleshoot production issues and ensure timely resolution.
• Participate in solution design discussions and contribute to best practices.
• Ensure data governance, security, and quality standards are followed.

Required Experience

• 4+ years of experience in Python development.
• Strong expertise in SQL and query optimization.
• Hands-on experience with Apache Spark in enterprise environments.
• Experience delivering data solutions for business or client-facing projects.
• Ability to work independently in a fast-paced solutions environment.

Nice to Have

• Databricks certification or practical implementation experience.
• Snowflake development experience.
• Experience in analytics, reporting, or data platform modernization projects.