Senior Data Engineer for ML & Analytics Pipelines

TechDigital Group

Minneapolis (MN)

On-site

USD 120,000 - 165,000

Full time

6 days ago
Be an early applicant

Get a reply from this recruiter

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

TechDigital Group is seeking a Senior Data Engineer to design and implement ETL/ELT pipelines on Azure Databricks and Spark, delivering ML-enabled data products and dashboards. You will ingest data from diverse sources, write optimized SQL, and raise data reliability and quality across platforms.

You will mentor other data engineers, influence architectural patterns, and collaborate with cross-functional teams to advance data capabilities and governance in a fast-paced environment.

Qualifications

  • 7+ years of experience in data engineering, data integration, data modeling, data architecture, and ETL/ELT processes to provide quality data and analytics solutions.
  • 5+ years of experience in Python.
  • 2+ years of experience in Apache Spark (PySpark/Spark SQL).
  • 2+ years building and deploying Cloud based solutions using Azure Databricks with UC, Snowflake, Functions, Service Bus.
  • 2+ years of experience in SQL with designing complex data schemas and query performance optimization.

Responsibilities

  • Design and develop ETL/ELT solutions on Azure Databricks, LakeBase and Spark.
  • Develop, implement, and deploy large scale data pipelines empowering machine learning algorithms, insights generation, business intelligence dashboards, reporting and new data products.
  • Design, build, optimize, and manage modern large-scale data pipelines ETL/ELT processing to support data integration for analytics, machine learning features and predictive modelling.
  • Consume data from a variety of sources (RDBMS, APIs, FTPs and other cloud storage) & formats (Excel, CSV, XML, JSON, Parquet, Unstructured).
  • Write advanced / complex SQL with performance tuning and optimization.
  • Identify ways to improve data reliability, data integrity, system efficiency and quality.
  • Participate in architectural evolution of data engineering patterns, frameworks, systems, and platforms including defining best practices and standards for managing data collections and integration.
  • Mentor other data engineers and provide technical direction by teaching other data engineers how to leverage cloud data platforms.

Skills

Python
SQL
Apache Spark
Azure Databricks
ETL/ELT
Terraform
CI/CD
GitHub Actions
Data Modeling
Data Warehousing
ML Pipelines

Education

Bachelor's degree in Computer Science, Engineering, Mathematics or related field

Tools

Snowflake
Git
Terraform
Artifactory
Sonar

Job description

TechDigital Group is seeking a Senior Data Engineer to design and implement ETL/ELT pipelines on Azure Databricks and Spark, delivering ML-enabled data products and dashboards. You will ingest data from diverse sources, write optimized SQL, and raise data reliability and quality across platforms.

You will mentor other data engineers, influence architectural patterns, and collaborate with cross-functional teams to advance data capabilities and governance in a fast-paced environment.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer — Azure & Databricks Pipelines
Senior Data Engineer — Azure & Databricks Pipelines

Sansar Tec, LLC • Haddon Heights (NJ)

On-site
USD 110,000 - 140,000
Senior Data Engineer: Azure Databricks & ML Platform
Senior Data Engineer: Azure Databricks & ML Platform

EPAM Systems • Town of Poland (NY)

On-site
USD 120,000 - 150,000
Senior Data Engineer
Senior Data Engineer

TechDigital Group • Chicago (IL)

On-site
USD 120,000 - 150,000
Senior Databricks Engineer: Azure Data & AI Pipelines
Senior Databricks Engineer: Azure Data & AI Pipelines

Quantiphi • United States

On-site
USD 100,000 - 130,000
Exposure to Fortune 500 clients
Latest technologies in AI and cloud
Leading a talented team
Remote Senior Data Engineer - Azure Databricks & Pipelines
Remote Senior Data Engineer - Azure Databricks & Pipelines

Hylant • Toledo (OH)

Remote
USD 120,000 - 160,000
Senior Data Engineer
Senior Data Engineer

Veriipro • Atlanta (GA)

On-site
USD 140,000 - 180,000
Senior Data Engineer - Azure Databricks & DataOps Lead
Senior Data Engineer - Azure Databricks & DataOps Lead

PlanIT Group • United States

On-site
USD 110,000 - 160,000
Senior Azure Data Engineer: Pipelines, Databricks & SQL
Senior Azure Data Engineer: Pipelines, Databricks & SQL

Ethereum Technologies LLC • Houston (TX)

On-site
USD 120,000 - 180,000
Senior Data Pipeline Engineer - Databricks & Azure
Senior Data Pipeline Engineer - Databricks & Azure

Centraprise • United States

On-site
USD 120,000 - 180,000
Senior Data Engineer: Scalable Azure Databricks Pipelines
Senior Data Engineer: Scalable Azure Databricks Pipelines

Care Access • United States

Remote
USD 120,000 - 180,000
PTO
Medical insurance
Dental insurance
+7