Senior Data Engineer: Scalable Pipelines & Data Quality

TechDigital Group; Burbank (CA); Sur place; USD 120 000 - 150 000; Plein temps

Résumé du poste

A technology solutions provider is seeking a Senior Data Engineer to design and develop scalable data pipelines for high-quality data management. The successful candidate will have a robust background in Python, SQL, and data engineering, along with experience in Azure technologies. Responsibilities include collaborating with teams to improve data models and mentoring junior engineers. This position promises opportunities for professional growth in a dynamic environment.

Qualifications

• 5 years of relevant work experience in analytics, data engineering, business intelligence or related field.
• Strong experience in Python and SQL.
• Experience developing integrations across multiple systems and APIs.

Responsabilités

• Develop, test, document and support scalable data pipelines.
• Establish and follow data governance processes to ensure data quality.
• Mentor junior engineers on coding best practices.

Connaissances

Object-oriented programming (Python)

Data engineering

API integration

SQL proficiency

Azure technologies

PySpark

Data warehousing

Unit testing

Kafka

Formation

4-year college degree in Computer Science, Mathematics, Statistics or related field

Outils

Databricks

SQL/NoSQL database systems

Flink

Fivetran

Matillion

Snowflake

Description du poste

Job Description

A Senior Data Engineer will design and develop highly scalable and extensible data pipelines which enable collection, storage, distribution, modeling, and analysis of large data sets from many channels. This position requires an innovative software engineer who is passionate about data & data quality. The ideal candidate will have strong data warehousing and API integration experience and the ability to develop scalable data pipelines that make data management and analytics/reporting faster, more insightful, and more efficient.
· Develop, test, document and support scalable data pipelines
· Build out and evolve data integrations including APIs to support continuing increases in data volume and complexity
· Establish and follow data governance processes and guidelines to ensure data availability, usability, consistency, integrity, and security
· Build, implement, and maintain scalable solutions that align to our data governance standards and architectural road maps for data integrations, data storage, reporting, and analytic solutions
· Collaborate with analytics and business teams to improve data models that feed business intelligence tools, increasing data accessibility, and fostering data-driven decision making across the organization
· Design and develop data integrations and a data quality framework. Write unit/integration/functional tests and document work
· Design, implement, and automate deployment of our distributed system for collecting and processing streaming events from multiple sources
· Perform data analysis needed to troubleshoot data-related issues and aid in the resolution of data issues
· Guide and mentor junior engineers on coding best practices and optimization
Qualifications:
· Education: 4-year college degree or equivalent combination of education and experience. Prefer an academic background in Computer Science, Mathematics, Statistics, or related technical field
· 5 years of relevant work experience in analytics, data engineering, business intelligence or related field
· Skilled in object-oriented programming (Python in particular)
· Strong Experience in Python, PySpark and SQL
· Strong Experience in Databricks
· Experience developing integrations across multiple systems and APIs
· Experience with or knowledge of Agile software development methodologies
· Experience with cloud-based databases, specifically Azure technologies (e.g., Azure data lake, ADF, Azure DevOps and Azure Functions)
· Experience using SQL queries as well as writing and perfecting SQL queries in a business environment with large-scale, complex datasets
· Experience with data warehouse technologies. Experience creating ETL and/or ELT jobs
· Excellent problem solving and troubleshooting skills
· Process oriented with great documentation skills
· Experience designing data schemas and operating SQL/NoSQL database systems is a plus
· Experience in Kafka, Flink ,Fivetran and Matillion is nice to have
· Experience in Data Science and Machine Learning is nice to have
Nice to have:
· Experience in software engineering
· Experience in Snowflake