Data Engineer

Taleo; Hyderabad; On-site; INR 1,200,000 - 2,600,000; Full time

Job summary

Taleo in Hyderabad, India is seeking a Data Engineer with 4-7 years of experience to design and maintain scalable data pipelines. You will leverage SQL, Spark, and Python/Scala to deliver robust ingestion and processing frames, while implementing SCDs and data models such as fact/dimension tables.

You will also work with AWS Glue/S3, Airflow and DBT to ensure reliable, cost-efficient operations, with a focus on data quality, partitioning, and scalable architectures.

Qualifications

• 4-7 years of data engineering experience.
• Strong SQL skills are required.
• Hands-on Spark experience with Python or Scala.
• Experience with AWS data services (Glue, S3) and serverless tools.
• Solid data modelling knowledge with fact/dimension design and SCD patterns.

Responsibilities

• Develop and maintain scalable data ingestion and processing pipelines using SQL, Spark, and Python/Scala.
• Develop and optimize Apache Spark batch and streaming jobs for large-scale data processing.
• Design and implement data models including Dimension and Fact tables with clear grain and relationships.
• Implement Slowly Changing Dimensions (SCD) and incremental processing strategies.
• Work with AWS data engineering services such as Glue and S3, plus EMR and Step Functions.
• Develop and maintain reliable, scalable data pipelines with error handling, logging, monitoring and data quality checks.
• Apply data engineering fundamentals including partitioning, incremental processing, schema evolution and pipeline reliability.
• Work with Apache Iceberg for table design, partitioning, schema evolution and incremental processing.
• Develop data transformation workflows using DBT and orchestrate pipelines with Airflow.
• Troubleshoot and optimize pipelines for performance, scalability and cost efficiency.

Skills

Strong SQL

Apache Spark

Python/Scala

AWS data services

Data modeling

SCD Type 1/Type 2

Data quality & monitoring

Tools

DBT

Airflow

Apache Iceberg

Kafka streaming

AWS Glue

S3

EMR

Step Functions

Job description

Data Engineer - Responsibilities (4-7 Years Exp)

• Develop and maintain scalable data ingestion and processing pipelines using SQL, Spark, and Python/Scala.
• Develop and optimize Apache Spark batch and streaming jobs for large-scale data processing.
• Design and implement data models including Dimension and Fact tables, with clearly defined grain, relationships, and measures.
• Implement Slowly Changing Dimensions (SCD) and appropriate incremental processing strategies.
• Work with AWS data engineering and serverless services such as Glue, S3, Lambda, EMR, and Step Functions.
• Develop and maintain reliable, scalable data pipelines with appropriate error handling, logging, monitoring, and data quality checks.
• Apply data engineering fundamentals including partitioning, incremental processing, schema evolution, data quality, and pipeline reliability.
• Work with Apache Iceberg, including table design, partitioning, schema evolution, and incremental processing.
• Develop data transformation workflows using DBT and orchestrate pipelines using Airflow.
• Troubleshoot and optimize data pipelines for performance, scalability, and cost efficiency.
• Participate in code reviews, testing, debugging, and continuous improvement of data engineering solutions.

Required Skills

• 4-7 years of relevant Data Engineering experience.
• Strong SQL skills - SQL is a core requirement.
• Strong hands-on experience with Apache Spark and Python or Scala.
• Hands-on experience with AWS data engineering and serverless services, particularly AWS Glue and S3.
• Good understanding of Data Modelling, including:
  • Dimension and Fact table design
  • Defining appropriate Fact grain
  • Dimension-to-Fact relationships
  • Implementing SCD Type 1 / Type 2 and other appropriate SCD patterns
• Strong understanding of data engineering fundamentals including partitioning, incremental processing, schema evolution, data quality, and pipeline reliability.
• Good understanding of distributed data processing and Spark performance optimization.

Additional Responsibilities / Good to Have

• Experience with Apache Iceberg and lakehouse architectures.
• Experience with DBT for data transformation and modelling.
• Experience with Airflow for workflow orchestration.
• Kafka / Kafka Streaming experience is a good to have.
• Exposure to AI/LLM tools for development, debugging, testing, and documentation.