AWS Data Engineer

Lumiq; Pune District; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

LUMIQ is seeking a data engineering professional to design and implement scalable cloud-based data architectures in Pune. The role involves building automated ETL/ELT pipelines with Python, PySpark, SQL, Airflow, and AWS Glue, and collaborating with BI/analytics teams to create visualizations.

You'll work on data modeling for Redshift and related warehouses, ensure data integrity and governance, and perform RCA on data issues, with opportunities to optimize performance across the data stack.

Qualifications

• Proficiency in SQL, Python and PySpark for data pipelines and analytics.
• Hands-on experience with AWS data services (e.g., Glue) and orchestration (Airflow).
• Strong understanding of data modeling, data warehousing, and data quality practices.
• Experience with DBT for transformation and modeling; familiarity with BI tools is a plus.

Responsibilities

• Solution Design & Implementation.
• Translate complex business requirements into scalable cloud data architectures.
• Build and maintain automated ETL/ELT pipelines using Python, PySpark, SQL, Airflow and AWS Glue.
• Integrate with DBT for data transformation and modeling.
• Design and implement data models for cloud warehouses like Redshift.
• Optimize performance and ensure data integrity and governance.
• Monitor pipeline health and resolve data anomalies or failures.
• Collaborate with BI/analytics teams to support dashboards and visualizations.
• Maintain documentation for solutions and workflows.

Skills

SQL

Python

PySpark

Data modeling

ETL/ELT design

CI/CD basics

Tools

AWS Glue

Airflow

DBT

Redshift

Power BI

Tableau

Looker

Job description

• At LUMIQ, we strive to create a community of passionate data professionals who aim to transcend the usual corporate dynamics. We offer you the freedom to ideate, commit, and navigate your career trajectory at your own pace.
• Culture of ownership empowerment to drive outcomes.
• Our culture encourages 'Tech Poetry' combining creativity and technology to create solutions that revolutionize the industry. We trust our people to manage their responsibilities with minimal policy constraints. Our team is composed of the industry's brightest minds, from PhDs and engineers to industry specialists from Banking, Insurance, NBFCs, AMCs, who will challenge and inspire you to reach new heights.

Key Responsibilities:

• Solution Design & Implementation
• Understand complex business requirements and translate them into scalable and maintainable cloud-based data architectures.
• Build and maintain automated ETL/ELT pipelines using Python, PySpark, SQL, Airflow, and AWS Glue.
• Integrate with DBT for data transformation and modeling.
• Data Modeling & Warehousing
• Design and implement efficient data models for Redshift and other cloud-based warehouses.
• Optimize performance and ensure data integrity and governance.
• Perform Root Cause Analysis (RCA) for issues in the existing data warehouse, automations, and reporting systems.
• Monitor pipeline health and proactively identify and resolve data anomalies or system failures.
• Visualization & Dashboards
• Collaborate with BI and analytics teams to support the creation of dashboards and visualizations.
• Design data models specifically tailored for efficient visual analytics.
• Work cross-functionally with business analysts, product owners, and other engineering teams.
• Maintain technical documentation for solutions and workflows.

Technical Skills Requirements:

• Strong proficiency in SQL, Python, and PySpark
• Hands-on experience with AWS services:
• Experience with Airflow, DBT, and data orchestration frameworks
• Solid understanding of data modeling, warehousing, and data quality practices
• Experience in dashboarding and data visualization support (Power BI/Tableau/Looker optional)
• Familiarity with CI/CD and version control systems like Git.