Senior Data Engineer

Jobgether; India; Remote; INR 2,000,000 - 2,800,000; Full time

Benefits offered by this job

Remote-friendly

Full-time employment

Opportunities to work with Databricks,

Professional learning and mentorship

Job summary

Partner Company, listed via Jobgether, is seeking a Senior Data Engineer in India to design and optimize scalable data platforms supporting analytics and ML workloads. The role emphasizes batch and real-time pipelines, lakehouse architectures, and cloud migration using Databricks, Python, SQL, and Spark.

You will automate CI/CD for data workflows with Terraform and GitHub Actions, containerize workloads with Docker, and mentor teammates in a fast-paced remote environment across Indian time zones.

Qualifications

• 4–7 years of professional experience in data engineering or related field.
• Advanced Python, SQL, and PySpark skills with large-scale datasets.

Responsibilities

• Design, build, and optimize scalable data pipelines (ETL/ELT) for batch and real-time processing.
• Implement Medallion Architecture (Bronze/Silver/Gold) for data quality and analytics.
• Migrate and manage cloud data platforms and lakehouse components across GCP/AWS/Azure.

Skills

Python

SQL

PySpark

Databricks

Spark Structured Streaming

Delta Live Tables

Terraform

GitHub Actions

Docker

Flask

Tools

Databricks

Terraform

GitHub Actions

Docker

Flask

Spark

Job description

This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Senior Data Engineer based in India.We are seeking an experienced Senior Data Engineer to design, build, and optimize scalable data platforms that power advanced analytics and data-driven business decisions. In this role, you will develop high-performance batch and real-time data pipelines using Python, SQL, PySpark, and Databricks, working with terabyte-scale datasets and distributed processing architectures. You will contribute to modern lakehouse solutions, cloud migration initiatives, and secure data management practices across major cloud platforms. You will also automate deployment workflows and infrastructure provisioning using modern DevOps tools and CI/CD practices. Collaborating with business stakeholders and cross-functional engineering teams, you will translate complex requirements into reliable, efficient technical solutions. This remote opportunity is suited to a technically strong professional who enjoys solving complex data challenges, improving platform performance, and mentoring colleagues in a fast-paced environment.

Accountabilities

• Design, develop, and optimize scalable ETL/ELT data pipelines for batch and real-time processing using Python, SQL, PySpark, Spark Structured Streaming, and Delta Live Tables.
• Implement and maintain Medallion Architecture, organizing data into Bronze, Silver, and Gold layers to support data quality, performance, and analytical requirements.
• Process terabyte-scale datasets using distributed computing techniques, ensuring pipeline reliability, scalability, and efficient resource utilization.
• Manage Databricks environments, including workflows, clusters, Delta Lake, and related lakehouse components.
• Implement Unity Catalog for metadata management, granular access controls, OAuth-based authentication, and enterprise-level data security and governance.
• Develop and execute cloud migration strategies for legacy databases, including Oracle, Teradata, and SQL Server, to cloud-based data platforms and storage solutions such as Google Cloud Storage, Amazon S3, Azure Data Lake Storage, Snowflake, and Amazon Redshift.
• Build and maintain CI/CD pipelines using Terraform and GitHub Actions to automate infrastructure provisioning, data pipeline deployments, testing, and monitoring.
• Develop custom data processing scripts and data-driven APIs using Python and web frameworks such as Flask.
• Use Docker to containerize data processing workloads and support portable, consistent deployment environments.
• Evaluate emerging technologies and approaches for data-intensive machine learning workloads and other advanced engineering use cases.
• Collaborate with business stakeholders, data teams, and cross-functional engineering partners to translate business needs into technical architectures and actionable data solutions.
• Apply best practices for SQL query optimization, data processing efficiency, security, monitoring, and operational reliability.
• Contribute to technical problem-solving, project ownership, knowledge sharing, and mentoring to strengthen team capabilities and delivery quality.

Requirements

• Four to seven years of professional experience in data engineering, big data development, or a closely related technical field.
• Advanced hands‑on proficiency in Python, SQL, and PySpark, including expert‑level SQL query optimization for large‑scale datasets.
• Strong practical experience with Apache Spark, Spark Structured Streaming, Delta Live Tables, and the design of scalable batch and real‑time data pipelines.
• Demonstrated experience working with terabyte‑scale datasets and distributed data processing architectures.
• Extensive working knowledge of Databricks, including Workflows, Unity Catalog, Delta Lake, and lakehouse architecture.
• Experience with at least one major cloud platform: Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure.
• Familiarity with cloud data storage and analytics services, including Google Cloud Storage, Amazon S3, Amazon Redshift, Azure Data Lake Storage Gen2, and Snowflake.
• Experience migrating data from legacy database environments, such as Oracle, Teradata, or SQL Server, to modern cloud data platforms.
• Strong understanding of infrastructure automation, Terraform, GitHub Actions, Docker, and CI/CD best practices for data engineering environments.
• Practical experience developing APIs or web‑based data services using Python frameworks such as Flask.
• Strong problem‑solving and analytical skills, with the ability to troubleshoot complex data processing challenges and optimize system performance.
• Excellent communication and collaboration skills, with the ability to work effectively with business stakeholders and cross‑functional technical teams.
• A proactive, ownership‑driven mindset, with the ability to manage priorities, work in an agile environment, and mentor other team members.
• Availability to work the US business‑hours shift, from 6:00 PM to 3:00 AM IST.
• Candidates who can join immediately or have a notice period of 0–15 days are preferred.

Benefits

• Remote working arrangement within India.
• Full‑time employment opportunity.
• Opportunities to work with modern data engineering technologies, including Python, Spark, Databricks, cloud platforms, and advanced data architectures.
• Exposure to large‑scale data processing, cloud migration, automation, and machine learning‑related engineering initiatives.
• Opportunities for professional learning, technical development, and collaboration with colleagues from diverse backgrounds.
• A workplace culture emphasizing transparency, diversity, integrity, continuous learning, and career growth.
• Specific salary details, healthcare coverage, paid leave, bonuses, and other benefits are not specified in the available job description.

Data Privacy Notice

By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre‑contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time. We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses and identifying potential inconsistencies or verification signals in application materials based on available information. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.