Senior Databricks Developer

TechDigital Group; Princeton (NJ); On-site; USD 90,000 - 150,000; Full time

Job summary

An established industry player is seeking a skilled Data Engineer with extensive experience in Apache Spark and Databricks. This exciting role involves developing and maintaining ETL pipelines, optimizing data processing workflows, and collaborating with cross-functional teams to drive insights from complex datasets. You'll play a crucial part in implementing machine learning workflows and ensuring data quality, all while staying updated with the latest advancements in big data technologies. If you're passionate about data and eager to contribute to innovative projects, this opportunity is perfect for you.

Qualifications

• 7-10 years of experience in data processing workflows using Apache Spark and Databricks.
• Proficiency in Python, Scala, or SQL for data manipulation and analytics.

Responsibilities

• Develop and maintain ETL pipelines using Databricks for large datasets.
• Collaborate with teams to design scalable data processing workflows.

Skills

Azure Databricks

Apache Spark

Data Modelling

Azure Data Lake Creation

Python Programming

Predictive Analytics

Client Libraries

Job description

Experience Level Required: 7-10 years

Mandatory Required Skills:

Azure Databricks

Apache Spark

Data Modelling

Azure Data Lake Creation

Python Programming

Preferred /Desired Skills:

predictive analytics

Experience in Client Libraries

Responsibilities:

• Develop and maintain ETL (Extract, Transform, Load) pipelines using Databricks to process and transform large datasets.
• Collaborate with data engineers and data scientists to design and implement scalable and efficient data processing workflows.
• Build and optimize Apache Spark jobs and clusters on the Databricks platform.
• Develop and maintain data ingestion processes to acquire data from various sources and systems.
• Implement data quality checks and validation procedures to ensure accuracy and integrity of data.
• Perform data analysis and exploratory data mining to derive insights from complex datasets.
• Design and implement machine learning workflows using Databricks for predictive analytics and model training.
• Troubleshoot and debug issues related to data processing, performance, and job failures.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Stay updated with the latest advancements in big data technologies and contribute to the improvement of existing systems and processes.

Requirements:

• Solid experience in developing data processing workflows using Apache Spark and Databricks.
• Proficiency in programming languages like Python, Scala, or SQL for data manipulation and analytics.
• Strong understanding of distributed computing principles and experience with large-scale data processing frameworks.
• Familiarity with cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP).
• Experience with data modeling, database systems, and SQL.
• Knowledge of machine learning concepts and experience with Client libraries and frameworks.
• Excellent problem-solving skills and ability to work independently and in a team.
• Strong communication skills to collaborate with stakeholders from different technical backgrounds.