Lead Data Engineer

Pyramid It Consulting; Chennai District, Coimbatore District, Bengaluru; Hybrid; INR 900,000 - 1,300,000; Full time

Job summary

Pyramid It Consulting is seeking a data engineer to design and implement scalable data pipelines and warehousing solutions in a fast-paced environment. You will work with AWS data services, Spark, and SQL to deliver reliable data models andBI-ready insights.

The role involves building streaming and batch ETL processes, collaborating with analytics teams, and operating within Agile sprints to meet evolving business requirements. Chennai-based opportunities may apply.

Qualifications

• Experience with AWS data services and data warehousing concepts.
• Strong PySpark/Scala/Java programming and SQL querying.
• Experience building ETL/ELT pipelines and data warehousing solutions.
• Hands-on with Kafka or Kinesis streams and real-time data processing.
• Knowledge of data modeling and data lake architectures; Hadoop ecosystem familiarity preferred.
• Experience with Tableau or BI reporting tools.
• Awareness of Agile environments.

Skills

ETL/ELT pipelines

Data Warehousing

Data Modeling

Data Lake architectures

Agile

Tools

AWS

PySpark

Apache Spark

SQL

Kafka/Kinesis

Hadoop ecosystem

Tableau/BI Reporting

Python/Scala/Java

Job description

Required Skills

• Strong experience in AWS (Redshift, S3, EMR, Glue, Lambda, EC2)
• Expertise in PySpark, Apache Spark, SQL
• Experience building ETL/ELT pipelines and Data Warehousing solutions
• Hands-on experience with Kafka or Kinesis Streams
• Strong knowledge of Data Modeling and Data Lake architectures, Data Warehousing.
• Experience with Hadoop ecosystem (Hive, HDFS, MapReduce) preferred
• Experience with Tableau/BI Reporting and Analytics platforms
• Proficiency in Python/Scala/Java
• Experience working in Agile environments