Data Engineer

TechDigital Group; United States; On-site; USD 140,000 - 190,000; Full time

Job summary

TechDigital Group is seeking an experienced Senior Data Engineer to design, build, and maintain a robust data infrastructure with a focus on Snowflake data warehousing and Azure cloud services.

You will develop scalable ETL/ELT pipelines, optimize Snowflake performance, and ensure data quality and governance across diverse sources. Collaboration with data scientists and analysts is essential to delivering actionable insights.

Qualifications

• Bachelor's or Master's degree in a related field.
• 7+ years of professional data engineering experience.
• Deep Snowflake warehousing expertise and best practices.
• Strong SQL and Snowflake performance optimization.
• Experience with Snowflake utilities (SnowSQL, Snowpipe, Tasks).
• Hands-on Azure data services experience across ADF, Synapse, Databricks, ADLS, SQL DB.

Responsibilities

• Data Warehouse Design and Development (Snowflake): design and maintain scalable data warehouses.
• ETL/ELT Pipeline Development: ingest, transform, load data from diverse sources into Snowflake.
• Azure Data Services: implement Azure data services integration for pipelines and data storage.
• Data Integration: connect internal and external data sources.
• Performance Optimization: resolve bottlenecks in pipelines and queries.
• Data Quality and Governance: enforce data quality checks and governance policies.
• Monitoring and Alerting: build monitoring for pipelines and infrastructure.

Skills

Snowflake data warehousing

Azure data services

SQL

ETL/ELT pipelines

Python/Scala

Data modeling

Data governance

Data quality

Snowflake utilities (SnowSQL, Snowpipe

Education

Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related field

Tools

SnowSQL

Snowpipe

Tasks

Azure Data Factory

Azure Synapse Analytics

Azure Databricks

Azure Data Lake Storage

Azure SQL Database

Job description

Experienced Senior Data Engineer to join our growing data and analytics team. You will play a crucial role in designing, building, and maintaining our data infrastructure, with a strong focus on leveraging Snowflake for data warehousing and Microsoft Azure for cloud-based data services. Your deep expertise in ETL/ELT processes will be essential for transforming raw data into insightful and actionable information that drives business decisions. You will collaborate closely with data scientists, analysts, and other stakeholders to understand their data needs and deliver robust and scalable data solutions.

Responsibilities:

• Data Warehouse Design and Development (Snowflake): Design, develop, and maintain scalable and efficient data models and data warehouses within the Snowflake environment.
• ETL/ELT Pipeline Development: Design, build, and optimize robust ETL/ELT pipelines using various tools and technologies to ingest, transform, and load data from diverse sources into Snowflake and other data stores.
• Azure Data Services: Utilize and integrate various Azure data services, including but not limited to Azure Data Factory, Azure Synapse Analytics (SQL Data Warehouse, Data Lake), Azure Databricks, Azure Blob Storage, Azure SQL Database, and Azure Data Lake Storage (ADLS).
• Data Integration: Develop and implement data integration solutions to connect various internal and external data sources.
• Performance Optimization: Identify and resolve performance bottlenecks in data pipelines and Snowflake queries.
• Data Quality and Governance: Implement and enforce data quality checks, data governance policies, and data security measures.
• Monitoring and Alerting: Design and implement monitoring and alerting systems for data pipelines and data infrastructure to ensure reliability and timely issue resolution.

Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related field.
• Minimum of 7 years of professional experience in data engineering.
• Deep expertise in Snowflake data warehousing:
• Strong understanding of Snowflake architecture, features, and best practices.
• Proven experience in designing and implementing data models and schemas in Snowflake.
• Proficient in writing complex SQL queries and optimizing Snowflake performance.
• Experience with Snowflake utilities and tools (e.g., SnowSQL, Snowpipe, Tasks).
• Strong experience with Microsoft Azure data services:
• Hands-on experience with Azure Data Factory for building ETL/ELT pipelines.
• Familiarity with other Azure data services such as Azure Synapse Analytics, Azure Databricks, Azure Data Lake Storage, and Azure SQL Database.
• Extensive experience with ETL/ELT tools and techniques:
• Proven ability to design, develop, and maintain complex ETL/ELT workflows.
• Experience with various data integration patterns and technologies.
• Strong proficiency in SQL and experience working with different database systems.
• Experience with at least one programming language relevant to data engineering (e.g., Python, Scala).
• Understanding of data warehousing concepts, dimensional modeling, and data lake architectures.