Senior Data Engineer

Compunnel, Inc.; Waltham (MA); On-site; USD 110,000 - 140,000; Full time

Job summary

A tech company is seeking a Senior Data Engineer to join their cross-functional team. The role involves designing scalable data pipelines and modernizing architecture. Candidates need a Bachelor's degree and a minimum of 5 years of experience in data engineering, with strong skills in Python, PySpark, SQL, and cloud technologies like Azure. This position offers the opportunity to work with innovative data solutions and the chance to mentor junior engineers.

Qualifications

• Minimum 5 years of experience in analytics, data engineering, or business intelligence.
• Hands-on experience with Databricks.
• Experience developing integrations across multiple systems and APIs.

Responsibilities

• Develop, test, document, and support scalable data pipelines.
• Build and evolve data integrations, including APIs.
• Establish and follow data governance processes to ensure data availability, consistency, integrity, and security.

Skills

Data engineering

API integration

Cloud technologies

Python

PySpark

SQL

Education

Bachelor’s degree in Computer Science, Mathematics, Statistics, or related field

Tools

Databricks

Azure Data Lake

ADF

Azure DevOps

Azure Functions

Snowflake

Job description

We are seeking a Senior Data Engineer to join a cross-functional team focused on building a forecasting platform.

This role involves designing and developing highly scalable data pipelines, modernizing architecture, and ensuring data quality and accessibility.

The ideal candidate will have strong experience in data engineering, API integration, and cloud technologies, with a passion for delivering efficient and reliable data solutions.

Key Responsibilities

• Develop, test, document, and support scalable data pipelines.
• Build and evolve data integrations, including APIs, to handle increasing data volume and complexity.
• Establish and follow data governance processes to ensure data availability, consistency, integrity, and security.
• Implement scalable solutions aligned with architectural roadmaps for data storage, reporting, and analytics.
• Collaborate with analytics and business teams to improve data models for BI tools and foster data-driven decision-making.
• Design and develop data integrations and a data quality framework; write unit/integration tests and document work.
• Automate deployment of distributed systems for collecting and processing streaming events.
• Perform data analysis to troubleshoot and resolve data-related issues.
• Mentor junior engineers on coding best practices and optimization.

Required Qualifications

• Bachelor’s degree in Computer Science, Mathematics, Statistics, or related field (or equivalent experience).
• Minimum 5 years of experience in analytics, data engineering, or business intelligence.
• Strong proficiency in Python, PySpark, and SQL.
• Hands-on experience with Databricks.
• Skilled in object-oriented programming (Python preferred).
• Experience developing integrations across multiple systems and APIs.
• Familiarity with Agile software development methodologies.
• Experience with cloud-based databases and Azure technologies (Azure Data Lake, ADF, Azure DevOps, Azure Functions).
• Expertise in SQL queries and working with large-scale, complex datasets.
• Experience with data warehouse technologies and creating ETL/ELT jobs.
• Excellent problem-solving and troubleshooting skills.
• Strong documentation and process-oriented approach.

Preferred Qualifications

• Experience with Kafka, Flink, Fivetran, and Matillion.
• Knowledge of Data Science and Machine Learning.
• Experience with Snowflake.
• Background in software engineering.
• Familiarity with designing data schemas and operating SQL/NoSQL databases.