Senior Data Engineer

Agilisium; Hyderabad, Pune District, Chennai District; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

Agilisium is seeking an experienced Data Engineer in Hyderabad to design and implement scalable data pipelines. You will work with Python, SQL, PySpark, and Databricks, leveraging AWS services to deliver robust data solutions.

The role requires strong SQL skills, knowledge of data modelling and ETL/ELT concepts, and experience with Airflow or Step Functions. Collaboration and clear communication are essential for success in a fast-paced environment.

Qualifications

• Proven experience as a Data Engineer (6-12 years, depending on level).
• Strong hands-on experience with Python for data engineering tasks.
• Proficiency in SQL (writing complex queries, performance tuning).
• Strong expertise in PySpark and Databricks.
• Hands-on experience with AWS cloud services (S3, Glue, EMR, Redshift, Lambda).
• Experience with data pipeline orchestration tools (Airflow, AWS Step Functions).
• Solid understanding of data modelling, warehousing, and ETL/ELT concepts.
• Knowledge of Git and CI/CD practices.
• Excellent problem-solving and communication skills.

Responsibilities

• Design and implement end-to-end data pipelines.
• Collaborate with stakeholders to gather requirements and deliver insights.
• Optimize SQL queries and data models for performance and scalability.
• Develop PySpark jobs in Databricks and manage 데이터 workloads.
• Work with AWS data services and orchestration tools to ensure reliability.
• Implement CI/CD practices for data pipelines and version control.

Skills

Python

SQL

PySpark

Databricks

AWS (S3, Glue, EMR, Redshift, Lambda)

Airflow

Git

CI/CD

Data modelling

ETL/ELT

Tools

AWS Step Functions

Job description

Role and Responsibility :

Proven experience as a Data Engineer (typically 6-12 years, depending on level).

• Strong hands-on experience with Python for data engineering tasks.
• Proficiency in SQL (writing complex queries, performance tuning).
• Strong expertise in PySpark and Databricks.
• Hands-on experience working in AWS cloud environment (S3, Glue, EMR, Redshift, Lambda, etc).
• Experience with data pipeline orchestration tools (e.g., Airflow, AWS Step Functions).
• Solid understanding of data modelling, warehousing, and ETL/ELT concepts.
• Knowledge of version control systems (Git) and CI/CD practices.
• Excellent problem-solving and communication skills