Lead Data Engineer

Synergy Marine Group; Chennai; In loco; INR 4.000.000 - 6.000.000; Tempo pieno

Descrizione del lavoro

Synergy Marine Group seeks an experienced Lead Data Engineer to design, build, and govern scalable data infrastructure while mentoring a team of four engineers. The role blends hands-on data engineering with delivery ownership and requires comfort with AI-assisted vibe coding tools to rapidly ship data products.

You will own Databricks pipelines, SQL performance tuning, ML data workflows, CI/CD, and cloud integration, coordinating with analytics, product, and business teams to align priorities

Competenze

• 8+ years in data engineering with at least 2+ years in a leadership/lead role.
• Hands-on Databricks, PySpark, and Python expertise.
• Advanced SQL skills and performance tuning (including T-SQL).
• Experience with ML data workflows and lifecycle.
• Experience using AI-assisted coding tools (Claude Code, Cursor, Copilot).
• Strong Git workflows, CI/CD, and code review practices.
• Ability to lead Agile/Scrum ceremonies and mentor 3–5 engineers.

Mansioni

• Architect, build, and optimize scalable data pipelines using Databricks, PySpark, and Python.
• Lead a team of 4 engineers; provide technical guidance and career development.
• Own data infrastructure, including compute, orchestration, deployment, and monitoring.
• Lead ML data product development, feature engineering, training, deployment, monitoring.
• Promote AI-assisted development tools to improve velocity and quality.
• Coordinate with analytics, product, and business teams to align priorities.

Conoscenze

Databricks

PySpark

Python

SQL

T-SQL

Git workflows

AI coding tools

Agile/Scrum

Team leadership

Cloud platforms

Machine Learning concepts

Formazione

Bachelor's degree in Computer Science

Master's degree in CS

Strumenti

Azure DevOps

GitHub Actions

Claude Code

Cursor

GitHub Copilot

Descrizione del lavoro

We are looking for an experienced Lead Data Engineer to drive the design, development, and management of our data infrastructure while leading a team of 4 engineers. This role combines strong hands-on technical expertise with people management and delivery ownership and requires someone comfortable using modern AI-assisted ("vibe coding") tools to rapidly build and ship data products, while maintaining the infrastructure that powers them.

Key Responsibilities

Technical Leadership & Development

• Architect, build, and optimize scalable data pipelines using Databricks, PySpark, and Python
• Write and review complex SQL queries for data transformation, validation, and performance tuning
• Design and maintain data models and ETL/ELT workflows across Databricks environments
• Support or lead development of ML models/pipelines for data products (feature engineering, model training, deployment, and monitoring)
• Build data product applications using AI-assisted "vibe coding" tools (Claude Code, Cursor, or similar) to rapidly prototype, develop, and ship internal tools, dashboards, and data-driven applications
• Establish and enforce best practices for code quality, version control, and CI/CD using Git
• Champion the adoption of AI-assisted development tools across the team to improve developer velocity, code review efficiency, and documentation
• Conduct code reviews, architecture reviews, and technical design sessions

Infrastructure & Platform Ownership

• Own and maintain the infrastructure underlying data products — compute clusters, transactional databases, job orchestration, environments, deployment pipelines, and monitoring
• Ensure data pipeline reliability, observability, and cost optimization across Databricks
• Manage access controls, environment configuration, and platform scaling for data product apps
• Troubleshoot infrastructure issues end-to-end (compute, storage, networking, connectivity)

Team & People Management

• Lead, mentor, and grow a team of 4 engineers, providing technical guidance and career development support
• Conduct 1:1s, performance reviews, and skill-development planning for team members
• Foster a culture of continuous learning and rapid upskilling in response to evolving tech/business needs
• Encourage and enable the team to adopt new tools and frameworks quickly

Delivery & Process Management

• Own sprint planning, backlog grooming, and sprint retrospectives for the team (Agile/Scrum)
• Break down business requirements into well-scoped technical tasks and user stories
• Track sprint velocity, manage blockers, and ensure timely delivery of data engineering initiatives
• Coordinate with cross-functional stakeholders (analytics, product, business teams) to align priorities

Required Skills & Experience

• 8+ years of experience in data engineering, with at least 2+ years in a technical leadership or team lead role
• Strong hands-on expertise in Databricks (notebooks, clusters, job orchestration, Delta Lake)
• Proficiency in PySpark and Python for large-scale data processing
• Advanced SQL skills (query optimization, stored procedures, performance tuning)
• Experience with Microsoft SQL Server (T-SQL, database design, troubleshooting)
• Experience with OCR data extraction (Azure Document Intelligence, AWS Textract)
• Solid understanding of Git workflows (branching strategies, PR reviews, merge conflict resolution)
• Hands-on experience building applications using AI coding tools / vibe coding (Claude Code, Cursor, GitHub Copilot, or similar)
• Experience maintaining and troubleshooting infrastructure supporting data products (compute, orchestration, deployment)
• Working knowledge of Machine Learning concepts and lifecycle (model training, deployment, monitoring)
• Proven experience running Agile sprint planning, backlog management, and team ceremonies
• Experience managing and mentoring a small engineering team (3-5 members)
• Demonstrated ability to quickly learn and adopt new technologies in response to business needs

Preferred Qualifications

• Experience with cloud platforms (Azure preferred, Databricks stack)
• Familiarity with CI/CD pipelines (Azure DevOps, GitHub Actions, etc.)
• Experience with data governance frameworks and data quality tools
• Bachelor's/Master's degree in Computer Science, Engineering, or related field

Soft Skills

• Strong communication skills to bridge technical and non-technical stakeholders
• Ability to balance hands-on coding with people management responsibilities
• High adaptability — comfortable navigating ambiguity and fast-changing tech/business priorities
• Problem-solving mindset with a focus on scalable, maintainable solutions