Senior Data Engineer, Databricks

Jobtailor; Massachusetts; On-site; USD 140,000 - 190,000; Full time

Job summary

Jobtailor is seeking a Senior Big Data Engineer in Massachusetts to design and implement end-to-end data pipelines on Azure and Databricks. You will own ETL processes, data ingestion, and warehousing concepts while collaborating in an agile team and following best practices.

The role requires 10+ years in big data, 5+ years in Databricks, and strong cloud experience including Azure, Kubernetes, and Docker. Snowflake exposure is a plus.

Qualifications

• 10+ years of overall Big Data data pipeline experience.
• 5+ years of hands-on Databricks experience.
• 5+ years of cloud-based development experience, including Azure Services, Azure DevOps, Kubernetes, and Docker.
• Bachelor’s Degree in computer or IT-related subject.
• Experience with Snowflake is a plus.

Responsibilities

• Design, build, and test end-to-end data pipelines using Azure, Databricks, and cloud infrastructure.
• Develop ETL processes for data ingestion, transformation, and loading into data lakes.
• Work on Databricks and data warehousing concepts.
• Follow best practices and coding standards.
• Collaborate in an Agile project team.
• Manage full software development lifecycle.

Skills

Databricks

Apache Spark

Scala

SQL

Data Pipelines

Data Quality

Git

Shell Scripting

Unix

Azure

Kubernetes

Docker

Cosmos DB

Snowflake

Delta Lake

Azure DevOps

Harness

TOAD

Education

Bachelor’s Degree in Computer or IT-related Subject

Tools

Databricks

Cosmos DB

Delta Lake

Azure DevOps

Kubernetes

Docker

Snowflake

TOAD

Git

Harness

ADO

Job description

• Design, build, and test end-to-end data pipelines, including streaming, events, and batch data ingestion, on cloud-based infrastructure using Azure and Cosmos DB
• Develop and implement ETL processes for data ingestion, transformation, and loading into data lakes
• Work extensively on Databricks and data warehousing concepts
• Design and develop custom high-throughput and configurable frameworks and libraries
• Drive change through collaboration, influence, and demonstration of POCs
• Manage all aspects of the software development lifecycle, including design, coding, integration testing, deployment, and documentation
• Work collaboratively within an agile project team
• Follow best practices and coding standards
• Grow personal skillset

Requirements

• Extensive technical, people and process skillsets
• Strong background in application architecture and development
• Very good understanding of leading IT technologies
• Experience in micro-services architecture and understanding of Cloud Computing; Azure preferred
• Proficient working knowledge of Databricks, Delta Lake, and Apache Spark
• Hands-on development experience in Databricks SQL and Scala
• Strong understanding of the Databricks platform, including clusters, jobs, and other resources
• Ability to monitor and troubleshoot data pipelines
• Experience implementing data quality checks and validations
• Experience performing data analysis and data exploration
• Strong hands-on experience troubleshooting DevOps pipelines, including ADO and Harness
• Experience in a multi-developer environment using version control such as Git
• Experience handling semi-structured data including Avro, JSON, and XML
• Basic knowledge of shell scripting, UNIX, TOAD, and SQL Developer
• Bachelor’s Degree level qualification in a computer or IT-related subject
• 10+ years of overall Big Data data pipeline experience
• 5+ years of hands-on Databricks experience
• 5+ years of cloud-based development experience, including Azure Services, Azure DevOps, Kubernetes, and Docker
• Experience with Snowflake is a plus
• Effective professional written and oral communication
• Team player with a positive attitude, enthusiasm, initiative, and self-motivation
• Ability to multitask; energetic, fast learner, and problem solver
• Experience working in the financial industry
• Experience with agile development methodology

Core Competencies

Demonstrates expertise in designing and implementing end-to-end data pipelines using Azure, Databricks, and cloud-based infrastructure, with a strong focus on ETL processes and data quality. Proficient in application architecture, micro-services, and agile methodologies, with a proven ability to collaborate effectively within teams.

Highest-signal resume keywords

• Azure Cloud Computing
• Databricks SQL
• ETL Processes
• Data Pipeline Monitoring
• Agile Development Methodology

ATS Optimization Keywords

Hard Skills

• Data Pipeline Development
• Databricks
• Apache Spark
• Scala
• SQL
• Data Quality Checks
• Shell Scripting
• UNIX
• Version Control (Git)
• Data Analysis

Soft Skills

• Effective Communication
• Team Collaboration
• Problem Solving
• Self-Motivation
• Positive Attitude

Certifications & Qualifications

• Bachelor’s Degree in Computer or IT-related Subject

Industry Keywords

• Big Data
• Financial Industry
• Agile Methodology
• Micro-Services Architecture

Tools & Technologies

• Azure DevOps
• Kubernetes
• Docker
• Cosmos DB
• Snowflake
• TOAD
• Delta Lake
• ADO
• Harness