Databricks (Remote)

PradeepIT Consulting Services Pvt Ltd; Bengaluru; Remote; INR 1,200,000 - 1,800,000; Full time

Job summary

PradeepIT Consulting Services Pvt Ltd in Bengaluru is seeking a Data Warehouse/ETL specialist with 6+ years of IT experience and 3+ years in DW/ETL projects. You will design and implement modern data warehouse solutions using Databricks on AWS/Azure, build scalable ETL pipelines, and collaborate with DW/BI leads to fulfill reporting needs.

You will triage pipeline issues, optimize data models, and drive discussions with client architects while leveraging Spark/PySpark, SQL, and Python.

Qualifications

• Bachelor's or Master's degree in computer science or equivalent experience.
• 6+ years IT experience and 3+ years in Data warehouse/ETL projects.
• Strong knowledge of star and snowflake dimensional modeling.
• Hands-on with SQL, Python, and Spark (PySpark).
• Experience with AWS/Azure stack; Databricks experience beneficial.

Responsibilities

• Develop Modern Data Warehouse solutions using Databricks and AWS/Azure Stack.
• Collaborate with DW/BI leads to design ETL pipelines.
• Triage issues to identify gaps in pipelines and fix them.
• Model data to fulfill reporting needs with business users.
• Drive technical discussions with client architects and team members.
• Orchestrate data pipelines in Airflow.

Skills

Databricks

AWS/Azure

PySpark

Python

SQL

Spark

Kafka

Hadoop

NoSQL

MongoDB

Unix Shell

Terraform

CircleCI

Git

Education

Bachelor's or Master's in CS

Tools

Airflow

Databricks Delta Lake

Job description

• Developing Modern Data Warehouse solutions using Databricks and AWS/ Azure Stack
• Ability to provide solutions that are forward-thinking in data engineering and analytics space
• Collaborating with DW/BI leads to understanding new ETL pipeline development requirements.
• Triage issues to find gaps in existing pipelines and fix the issues
• Work with businesses to understand the need in the reporting layer and develop a data model to fulfill
• reporting needs
• Help joiner team members to resolve issues and technical challenges.
• Drive technical discussion with client architects and team members
• Orchestrate the data pipelines in the scheduler via Airflow
• Qualification & experience
• Bachelor's and/or master's degree in computer science or equivalent experience.
• Must have a total of 6+ yrs. of IT experience and 3+ years' experience in Data warehouse/ETL projects.
• Deep understanding of Star and Snowflake dimensional modeling.
• Strong knowledge of Data Management principles
• Good understanding of Databricks Data & AI platform and Databricks Delta Lake Architecture
• Should have hands-on experience in SQL, Python, and Spark (PySpark)
• Candidate must have experience in AWS/ Azure stack
• Desirable to have ETL with batch and streaming (Kinesis).
• Experience in building ETL / data warehouse transformation processes
• Experience with Apache Kafka for use with streaming data / event-based data
• Experience with other Open-Source big data products Hadoop (incl. Hive, Pig, Impala)
• Experience with Open Source non-relational / NoSQL data repositories (incl. MongoDB,
• Experience working with structured and unstructured data including imaging & geospatial data.
• Experience working in a Dev/Ops environment with tools such as Terraform, CircleCI, and GIT.
• Proficiency in RDBMS, complex SQL, PL/SQL, Unix Shell Scripting, performance tuning, an troubleshooting.
• Databricks Certified Data Engineer Associate/Professional Certification (Desirable).
• Comfortable working in a dynamic, fast-paced, innovative environment with several ongoing
• Should have experience working in Agile methodology.
• Strong verbal and written communication skills.
• Strong analytical and problem-solving skills with a high attention to detail. Mandatory Skills:
• Python/ PySpark / Spark with Azure/ AWS Databricks