Data Engineer

Scaletrix.AI; Gurugram District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Scaletrix.AI is seeking a skilled Data Engineer with 4–6 years of experience to design, build, and maintain scalable data platforms and pipelines. The role focuses on cloud-based data engineering, ETL/ELT development, data warehousing, and big data technologies, collaborating with data scientists, analysts, BI teams, and business stakeholders.

You will implement and maintain data ingestion, event-driven architectures, and CI/CD pipelines while ensuring data quality, governance, and compliance

Qualifications

• Proficient in Python and SQL for data engineering tasks.
• Experience with PySpark, Pandas, and large-scale data processing.
• Expertise with Databricks, DBT, and Airflow for pipeline orchestration.
• Knowledge of Snowflake Data Warehouse and Snowpipe.
• Strong AWS experience (S3, EMR, Redshift, Lambda, RDS).
• Experience with REST APIs and data integration frameworks.
• Understanding of Data Lake and Data Warehouse architectures.

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines.
• Develop batch and real-time data processing workflows.
• Manage cloud-based data lakes and data warehouses (AWS, Snowflake).
• Implement event-driven architectures using AWS Lambda and Kafka.
• Collaborate with Data Science, Analytics, BI, and Product teams.
• Expose APIs for data integration and data-sharing use cases.
• Ensure data quality, governance, privacy, and compliance.
• Implement CI/CD pipelines for data projects using GitHub/GitLab.
• Optimize pipelines for performance, scalability, and cost.
• Create operational dashboards and reporting with Tableau or BI tools.

Skills

Python

SQL

PySpark

Airflow

Databricks

DBT

Snowflake

AWS

REST APIs

Data governance

Education

Bachelor's Degree

Tools

Snowflake

GitHub

GitLab

CI/CD

Docker

Job description

We are looking for a highly skilled Data Engineer with 4–6 years of experience in designing, building, and maintaining scalable data platforms and data pipelines. The ideal candidate should possess strong expertise in cloud-based data engineering, ETL/ELT development, data warehousing, and big data technologies. The candidate will work closely with Data Scientists, Analysts, BI teams, and business stakeholders to enable reliable and efficient data solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Python, PySpark, SQL, Pandas, and DBT.
• Build and optimize batch and real-time data processing workflows using Apache Airflow and Databricks.
• Develop and manage cloud-based data lakes and data warehouses using AWS services such as S3, EMR, Redshift, Lambda, and RDS.
• Implement and maintain Snowflake data warehouse solutions, including Snowpipe, external stages, and data-sharing capabilities.
• Develop robust data ingestion frameworks for structured and semi-structured data from multiple sources.
• Design and implement event-driven architectures using AWS Lambda, Kafka, S3 Event Notifications, and other cloud-native services.
• Collaborate with cross-functional teams including Data Science, Analytics, BI, and Product teams to support data requirements.
• Build and expose APIs for data integration and data-sharing use cases.
• Ensure data quality, governance, privacy, and compliance through validation, masking, and monitoring frameworks.
• Implement CI/CD pipelines for data engineering projects using GitHub Actions, GitLab, JFrog, and automated testing frameworks.
• Optimize data pipelines for performance, scalability, and cost efficiency.
• Create operational dashboards and reporting solutions using Tableau or similar BI tools.

Required Skills & Qualifications

Technical Skills

• Strong programming skills in Python and SQL.
• Hands-on experience with PySpark, Pandas, and large-scale data processing.
• Expertise in Databricks, DBT, and Apache Airflow.
• Experience with Snowflake Data Warehouse and Snowpipe.
• Strong understanding of AWS ecosystem including:
  • S3
  • EMR
  • Redshift
  • Lambda
  • EC2
  • RDS
• Experience working with Apache Kafka and event-driven data architectures.
• Hands-on experience with PostgreSQL, MongoDB, and relational databases.
• Knowledge of Docker and containerized deployments.
• Experience with REST APIs and data integration frameworks.
• Familiarity with GitHub, GitLab, CI/CD pipelines, and automated testing practices.
• Understanding of Data Lake and Data Warehouse architectures.

Preferred Qualifications

• Experience in Healthcare, Life Sciences, Retail, or eCommerce domains.
• Knowledge of data governance, security, and compliance best practices.
• Exposure to cloud certifications (AWS, OCI, Azure, or GCP).
• Experience working in Agile/Scrum environments.

Education

• Bachelor's Degree in Computer Science, Information Technology, Engineering, or a related field.

Nice to Have

• Experience with DuckDB.
• Exposure to Generative AI, Machine Learning, or MLOps platforms.
• Experience designing enterprise-scale data platforms and modern lakehouse architectures.

What Success Looks Like

• Deliver highly reliable and scalable data pipelines.
• Ensure high-quality and trusted data for analytics and business decision-making.
• Improve data processing efficiency and reduce operational overhead.
• Contribute to the evolution of the organization's modern data platform strategy.