Senior Fullstack Developer

agilisium; Chennai District; On-site; INR 800,000 - 1,200,000; Full time

Job summary

A leading data solutions firm located in Tamil Nadu is seeking a Data Engineer to join their innovative team. The successful candidate will design and optimize scalable data pipelines utilizing advanced technologies such as PySpark and Databricks. A focus on collaboration with cross-functional teams is essential for understanding data requirements. Candidates should possess strong programming skills in Python and SQL, along with experience in AWS services. This is an exciting opportunity to contribute to data-driven decision-making in a cloud-first environment.

Qualifications

• 4 to 6 years of experience in Data Engineering or related field.
• Strong programming experience with Python for data wrangling.
• Solid hands-on experience with PySpark.

Responsibilities

• Design and optimize scalable data pipelines using PySpark and SQL.
• Collaborate with teams to understand data requirements.
• Implement data ingestion frameworks from various sources.

Skills

Python

SQL

AWS

PySpark

Databricks

Data Modeling

Data Governance

Tools

AWS Glue

Airflow

Git

CloudWatch

Job description

Introduction to the Role

Are you passionate about unlocking the power of data to drive innovation and transform business outcomes? Join our cutting-edge Data Engineering team and be a key player in delivering scalable, secure, and high-performing data solutions across the enterprise. As a Data Engineer, you will play a central role in designing and developing modern data pipelines and platforms that support data-driven decision-making and AI-powered products. With a focus on Python, SQL, AWS, PySpark, and Databricks, you'll enable the transformation of raw data into valuable insights by applying engineering best practices in a cloud-first environment.

We are looking for a highly motivated professional who can work across teams to build and manage robust, efficient, and secure data ecosystems that support both analytical and operational workloads.

Accountabilities

• Design, build, and optimize scalable data pipelines using PySpark, Databricks, and SQL on AWS cloud platforms.
• Collaborate with data analysts, data scientists, and business users to understand data requirements and ensure reliable, high-quality data delivery.
• Implement batch and streaming data ingestion frameworks from a variety of sources (structured, semi-structured, and unstructured data).
• Develop reusable, parameterized ETL/ELT components and data ingestion frameworks.
• Perform data transformation, cleansing, validation, and enrichment using Python and PySpark.
• Build and maintain data models, data marts, and logical/physical data structures that support BI, analytics, and AI initiatives.
• Apply best practices in software engineering, version control (Git), code reviews, and agile development processes.
• Ensure data pipelines are well-tested, monitored, and robust with proper logging and alerting mechanisms.
• Optimize performance of distributed data processing workflows and large datasets.
• Leverage AWS services (such as S3, Glue, Lambda, EMR, Redshift, Athena) for data orchestration and lakehouse architecture design.
• Participate in data governance practices and ensure compliance with data privacy, security, and quality standards.
• Contribute to documentation of processes, workflows, metadata, and lineage using tools such as Data Catalogs or Collibra (if applicable).
• Drive continuous improvement in engineering practices, tools, and automation to increase productivity and delivery quality.

Essential Skills Experience

• 4 to 6 years of professional experience in Data Engineering or a related field.
• Strong programming experience with Python and experience using Python for data wrangling, pipeline automation, and scripting.
• Deep expertise in writing complex and optimized SQL queries on large-scale datasets.
• Solid hands‑on experience with PySpark and distributed data processing frameworks.
• Expertise working with Databricks for developing and orchestrating data pipelines.
• Experience with AWS cloud services such as S3, Glue, EMR, Athena, Redshift, and Lambda.
• Practical understanding of ETL/ELT development patterns and data modeling principles (Star/Snowflake schemas).
• Experience with job orchestration tools like Airflow, Databricks Jobs, or AWS Step Functions.
• Understanding of data lake, lakehouse, and data warehouse architectures.
• Familiarity with DevOps and CI/CD tools for code deployment (e.g., Git, Jenkins, GitHub Actions).
• Strong troubleshooting and performance optimization skills in large-scale data processing environments.
• Excellent communication and collaboration skills, with the ability to work in cross‑functional agile teams.

Desirable Skills Experience

• AWS or Databricks certifications (e.g., AWS Certified Data Analytics, Databricks Data Engineer Associate/Professional).
• Exposure to data observability, monitoring, and alerting frameworks (e.g., Monte Carlo, Datadog, CloudWatch).
• Experience working in healthcare, life sciences, finance, or another regulated industry.
• Familiarity with data governance and compliance standards (GDPR, HIPAA, etc.).
• Knowledge of modern data architectures (Data Mesh, Data Fabric).
• Exposure to streaming data tools like Kafka, Kinesis, or Spark Structured Streaming.
• Experience with data visualization tools such as Power BI, Tableau, or QuickSight.