Lead Data Engineer (Lead Software Engineer)

DigitalXNode; Ahmedabad District; On-site; INR 1,800,000 - 2,600,000; Full time

Job summary

DigitalXNode is seeking a Lead Data Engineer to drive design, development, and optimization of large-scale data platforms in an enterprise setting. You will lead complex data initiatives, build modern pipelines, and shape engineering best practices across the organization.

Key responsibilities include leading architecture, building scalable ETL/ELT data pipelines, and mentoring teams to deliver robust, compliant data solutions on AWS/Azure/GCP. Strong Python, Spark, and CI/CD experience required.

Qualifications

• Bachelors in Engineering, Computer Science or related field required.
• 5+ years of software/data engineering or equivalent practical experience.
• Hands-on with data technologies and modern pipelines.

Responsibilities

• Lead design and optimization of scalable data platforms.
• Build metadata-driven ingestion frameworks and reusable components.
• Develop and maintain Apache Spark pipelines for batch/streaming data.
• Ensure data quality, observability, and governance across pipelines.
• Mentor team members and drive best practices in data engineering.

Skills

Python

SQL

Apache Spark

Hadoop

Airflow

AWS

Azure

GCP

REST APIs

CI/CD

Docker

Kubernetes

Data Lakehouse

ETL/ELT

Agile

Education

B.Tech / BE or equivalent

Postgraduate degree (preferred)

Tools

Terraform

GitHub Actions

Jenkins

Job description

We are seeking a highly skilled Lead Data Engineer to drive the design, development, and optimization of large-scale data platforms. This role involves leading complex, enterprise-wide initiatives, building modern data pipelines, and shaping best practices for data engineering across the organization.

Key Skills

Data Engineering, Python, SQL, Apache Spark, Hadoop, Airflow, AWS, Azure, GCP, REST APIs, CI/CD, Docker, Kubernetes, Data Lakehouse, ETL/ELT, Agile Key Responsibilities

Technical Leadership & Architecture

• Lead large-scale, high-impact technology initiatives across teams
• Define and implement best practices for data engineering and platform architecture
• Review and evaluate complex system designs aligned with business and enterprise goals
• Mentor team members and provide technical leadership

Data Engineering & Pipeline Development

• Design, build, and maintain scalable data pipelines (ETL/ELT) for structured and unstructured data
• Develop metadata-driven ingestion frameworks, validation layers, and reusable components
• Ensure high performance, reliability, and scalability of data systems

Distributed Computing & Lakehouse Engineering

• Build and optimize Apache Spark pipelines for batch and streaming workloads
• Work with modern data lakehouse technologies (Iceberg, Delta Lake, Hudi)
• Implement Medallion architecture (Bronze/Silver/Gold layers)

Data Quality & Observability

• Implement data quality frameworks (e.g., Great Expectations, Deequ)
• Build monitoring systems with SLAs/SLOs, anomaly detection, and lineage tracking
• Ensure robust validation during migrations and onboarding processes

API & Microservices Development

• Develop RESTful APIs using Python frameworks (FastAPI, Flask)
• Enable secure and governed data access across platforms

Cloud & DevOps

• Design and deploy pipelines on cloud platforms (AWS, Azure, GCP)
• Build CI/CD pipelines using tools like Jenkins, GitHub Actions, or Azure DevOps
• Implement infrastructure as code (Terraform, Helm) and secure engineering practices

Orchestration & Workflow Management

• Build and manage workflows using tools like Airflow or Autosys
• Design resilient pipelines with retries, alerts, and dependency handling

Collaboration & Delivery

• Work with cross-functional Agile teams including Product, Architecture, and Business stakeholders
• Analyze requirements, propose solutions, and contribute to technical roadmaps
• Independently deliver complex engineering solutions

Required Qualifications

• Bachelor's degree in Engineering, Computer Science, or related field
• 5+ years of experience in software/data engineering or equivalent practical experience

Technical Skills & Experience

Core Skills

• Strong hands-on experience with Python, SQL, and Bash scripting
• Experience with big data technologies: Apache Spark, Hadoop, Hive
• Expertise in building scalable data pipelines and distributed systems

Data Platforms & Storage

• Experience with data lakehouse architectures and storage formats (Parquet, ORC)
• Knowledge of optimization techniques (partitioning, clustering, compaction, Z-ordering)

Streaming & Advanced Processing

• Experience with streaming frameworks such as Spark Structured Streaming or Apache Flink

APIs & Integration

• Working knowledge of REST APIs, object storage, and data access layers

Cloud & DevOps

• Hands-on experience with AWS, Azure, or GCP
• Familiarity with CI/CD tools, containerization (Docker, Kubernetes), and automation

Data Governance & Quality

• Experience with governance tools (Collibra, Alation, Purview)
• Understanding of compliance standards (SOX, PCI) and data validation practices

Additional (Good to Have)

• Experience with GenAI applications in data engineering (metadata extraction, anomaly detection, automation)
• Domain exposure to financial services, treasury, or risk management

Key Competencies

• Strong problem-solving and analytical thinking
• Leadership and mentoring capabilities
• Excellent communication and stakeholder management skills
• Ability to work in fast-paced Agile environments

Education

• UG: B.Tech / B.E. or equivalent in any specialization
• PG: Any Postgraduate (preferred)