Senior Data Engineer

Pratham Software (PSI); Jaipur; On-site; INR 1,400,000 - 2,200,000; Full time

Job summary

Pratham Software (PSI) is seeking a hands-on Data Engineer with expertise in SQL, PySpark, Spark, and Databricks to build scalable data pipelines and data products. The role emphasizes performance optimization, Delta Lake integration, and GenAI-assisted coding tools to accelerate development while ensuring data quality and cost efficiency.

You will collaborate with cross-functional teams, troubleshoot complex data workloads, and contribute to a modern data-platform architecture.

Qualifications

• Strong problem-solving and analytical skills.
• Hands-on, engineering-oriented mindset.
• Ability to understand why a technical approach is selected, not just implement it.
• Clear and structured communication.
• Comfortable working independently and learning new technologies.
• Expert SQL and PySpark development.
• Strong understanding of Spark internals: distributed processing, partitioning, shuffle, data skew, execution model.
• Strong Databricks experience with Delta Lake, optimization, compute management, and cost optimization.
• Strong data-engineering fundamentals: ETL/ELT, data modelling, incremental processing, and data quality.
• Experience troubleshooting and optimizing large-scale data workloads.

Responsibilities

• Design, develop, and maintain scalable data pipelines using SQL, PySpark, Spark, and Databricks.
• Build reliable batch/incremental data-processing solutions and data products.
• Troubleshoot and optimize Spark workloads for performance, scalability, and cost efficiency.
• Work with Delta Lake and modern data-platform architecture.
• Use AI coding agents to accelerate development, debugging, testing, and documentation.
• Collaborate with application teams, consultants, and managers.

Skills

SQL

PySpark

Apache Spark

Databricks

GenAI awareness

AI-assisted coding tools

ETL/ELT

Data modeling

Data quality

Troubleshooting

Tools

Delta Lake

NoSQL databases

Job description

We are looking for a hands-on Data Engineer with strong expertise in SQL, PySpark, Apache Spark, and Databricks, along with awareness of GenAI and AI-assisted coding tools. The candidate should be capable of building scalable data pipelines, troubleshooting performance issues, and optimizing data processing for both performance and cost.

Responsibilities:

• Design, develop, and maintain scalable data pipelines using SQL, PySpark, Spark, and Databricks.
• Build reliable batch/incremental data-processing solutions and data products.
• Troubleshoot and optimize Spark workloads for performance, scalability, and cost efficiency.
• Work with Delta Lake and modern data-platform architecture.
• Use AI coding agents to accelerate development, debugging, testing, and documentation.
• Collaborate with application teams, consultants, and managers.

Requirements:

• Strong problem-solving and analytical skills.
• Hands-on, engineering-oriented mindset.
• Ability to understand why a technical approach is selected, not just implement it.
• Clear and structured communication.
• Comfortable working independently and learning new technologies.
• Expert SQL and PySpark development.
• Strong understanding of Apache Spark internals: distributed processing, partitioning, shuffle, data skew, and execution model.
• Strong Databricks experience with Delta Lake, optimization, compute management, and cost optimization.
• Strong data-engineering fundamentals: ETL/ELT, data modelling, incremental processing, and data quality.
• Experience troubleshooting and optimizing large-scale data workloads.

GenAI and AI-Assisted Development:

• LLMs, RAG, embeddings, agents, and agentic AI.
• AI-assisted coding tools such as Claude Code, Codex, Cursor, Antigravity, or similar.
• Ability to review, validate, and improve AI-generated code.

Good to Have:

• DevOps knowledge: Git, CI/CD, Docker, and deployment practices.
• Exposure to NoSQL databases and their use cases.
• Data Engineer/Analytics Certificate in any of the clouds (AWS, Azure, or GCP).