Lead Data Engineer

Tredence Inc.; Pune District, Chennai District, Bengaluru; On-site; INR 1,400,000 - 2,100,000; Full time

Job summary

Tredence Inc. in Pune is seeking a Data Engineer to develop modern data warehouse solutions with Databricks and Azure Stack. You will collaborate with DW/BI leads to define ETL requirements, triage issues, and build data models that meet reporting needs.

The role demands hands-on SQL, Python and Spark (PySpark), plus experience in ETL/ DW transformations, Unix shell scripting, and Databricks architecture. Agile experience and strong communication are essential to thrive in a dynamic,

Qualifications

• Bachelor's or master's degree in computer science or equivalent.
• Hands-on SQL, Python and Spark (PySpark).
• Experience building ETL / data warehouse transformations.
• Databricks Data & AI platform knowledge and Delta Lake architecture.

Responsibilities

• Develop modern Data Warehouse solutions using Databricks and Azure Stack.
• Collaborate with DW/BI leads to define ETL requirements.
• Triage issues to fix gaps in pipelines.
• Model data to fulfill reporting needs for stakeholders.
• Drive technical discussions with client architects and team members.
• Orchestrate data pipelines using Airflow.

Skills

Databricks

PySpark

SQL

Python

Spark

Airflow

Unix Shell Scripting

ETL

Education

Bachelor's or Master's degree in Computer Science or equivalent

Tools

Databricks

Azure Data Factory

Azure Data Lake

Airflow

Job description

Role & responsibilities

• Developing Modern Data Warehouse solutions using Databricks and Azure Stack
• Ability to provide solutions that are forward-thinking in data engineering and analytics space
• Collaborate with DW/BI leads to understand new ETL pipeline development requirements.
• Triage issues to find gaps in existing pipelines and fix the issues
• Work with business to understand the need in reporting layer and develop data model to fulfill reporting needs
• Drive technical discussion with client architect and team members
• Orchestrate the data pipelines in scheduler via Airflow Skills and Qualifications:
• Bachelor's and/or masters degree in computer science or equivalent experience.
• Good understanding of Databricks Data & AI platform and Databricks Delta Lake Architecture
• Should have hands-on experience in SQL, Python and Spark (PySpark)
• Experience in building ETL / data warehouse transformation processes
• Proficiency in RDBMS, complex SQL, PL/SQL, Unix Shell Scripting, performance tuning and troubleshoot
• Databricks Certified Data Engineer Associate/Professional Certification (Desirable).
• Comfortable working in a dynamic, fast-paced, innovative environment with several ongoing concurrent projects
• Should have experience working in Agile methodology
• Strong verbal and written communication skills.
• Strong analytical and problem-solving skills with a high attention to detail.

Mandatory Skills

Azure Databricks, Pyspark, Azure Data Factory, Azure Data Lake.

Notice Period: Immediate joiners, 1 Month, 2 Months notice.