Senior Data Engineer

SOTI Inc; Ernakulam; On-site; INR 1,500,000 - 2,000,000; Full time

Job summary

SOTI Inc in Ernakulam, Kerala is seeking a Senior Data Engineer to design, build, and maintain ETL/ELT pipelines. The ideal candidate should have over 7 years of experience in Data Engineering and be proficient in SQL, Python, and cloud platforms such as Azure Synapse Analytics.

You will collaborate with cross-functional teams and oversee the transition of data platforms while ensuring the stability of critical data flows. Strong problem-solving skills and mentorship experience are essential.

This role demands hands-on experience with distributed data environments.

Qualifications

• 7+ years of experience in Data Engineering or related fields.
• Strong hands-on experience with SQL, Python, PySpark, and Spark SQL.
• Proven experience building ETL/ELT pipelines in the cloud.

Responsibilities

• Design and maintain ETL/ELT pipelines using SQL and Python.
• Collaborate on cloud migration initiatives.
• Troubleshoot complex pipeline failures.

Skills

SQL

Python

PySpark

Spark SQL

Data Engineering

Azure Synapse Analytics

Tools

Azure Data Factory

Microsoft Fabric

Job description

Key Responsibilities

• Design, build, and maintain production grade ETL/ELT pipelines using SQL, Python, PySpark, and Spark SQL.
• Own end‑to‑end data pipelines from ingestion through curated, analytics‑ready datasets.
• Maintain and enhance existing integrations to ensure stability, performance, and scalability.
• Troubleshoot complex pipeline failures and act as a key escalation point for production support.
• Collaborate with Lead Engineers and Data Architects on on‑prem to cloud migration initiatives.
• Assess legacy pipelines and integrations, contributing to migration sequencing, refactoring, and risk mitigation.
• Ensure continuity of business‑critical data flows during phased migrations and platform transitions.
• Build and operate solutions across hybrid data platforms, including SQL Server, Azure Synapse, and Microsoft Fabric.
• Develop and maintain Fabric Lakehouses, Warehouses, Pipelines, and Notebooks aligned with architectural standards.
• Optimize workloads for performance, reliability, and cost efficiency.
• Implement data validation, reconciliation, monitoring, logging, and alerting to ensure trusted data products.
• Apply disciplined engineering practices: version control, CI/CD, testing, and documentation.
• Improve the operational maturity and production readiness of data solutions.
• Mentor junior data engineers through code reviews, guidance, and best practice sharing.
• Promote consistent engineering standards and support team onboarding.
• Collaborate closely with BI teams, data owners, and governance stakeholders to align with analytics needs.
• Identify and drive improvements in pipeline efficiency, migration effectiveness, and developer productivity.

Required Qualifications

• 7+ years of experience in Data Engineering, Analytics Engineering, or a related technical role.
• Strong hands‑on experience with SQL, Python, PySpark, and Spark SQL in distributed data environments.
• Proven experience building and supporting enterprise‑scale ETL/ELT pipelines in the cloud.
• Strong experience with Azure‑based data platforms, including Azure Synapse Analytics and/or Microsoft Fabric.
• Solid understanding of cloud‑native data architectures, analytics engineering patterns, and medallion architectures.
• Experience with Git‑based source control, CI/CD pipelines, and automated testing for data platforms.Strong problem‑solving skills and experience supporting mission‑critical production data systems.
• Proven ability to collaborate across engineering and business teams and mentor junior engineers.

Preferred / Nice to Have

• Hands‑on experience with Microsoft Fabric Lakehouse and Spark workloads.
• Experience with Azure Data Factory, Fabric Pipelines, or orchestration tools.
• Familiarity with data governance, lineage, and certification in cloud platforms.
• Relevant certifications such as Microsoft Certified: Azure Data Engineer Associate, Microsoft Fabric Analytics Engineer.