Senior Data Engineer - Databricks & Delta Lake (Remote)

LorganTech; United States; Remote; USD 83.000 - 90.000; Vollzeit

Zusammenfassung

Lorgan Tech Solutions, LLC is seeking a Senior Data Engineer to build and optimize enterprise-grade data platforms using Databricks and Delta Lake. You will design scalable ETL/ELT pipelines and curate datasets for analytics across the organization.

Lead batch and streaming processes with robust data quality, governance, and monitoring. This remote role offers collaboration with stakeholders and mentoring of engineering teams.

Qualifikationen

• 10+ years of professional Data Engineering experience.
• Strong hands-on experience with Databricks, Apache Spark, PySpark, and SQL.
• Proficiency in Python and distributed data processing.
• Experience with Delta Lake, medallion architecture, and data modeling.
• Experience with at least one cloud platform: Azure, AWS, or GCP.
• Knowledge of orchestration using Databricks Workflows, Airflow, or Azure Data Factory.
• Experience implementing incremental loads, CDC, and data validation.
• Familiarity with Unity Catalog, security, and data governance.
• Experience with Git, CI/CD, and production troubleshooting.
• Strong communication, problem-solving, and collaboration skills.

Aufgaben

• Develop and maintain scalable ETL/ELT pipelines using Databricks, PySpark, and SQL.
• Implement Bronze, Silver, and Gold layers using Delta Lake and medallion architecture.
• Build reliable batch and streaming pipelines with incremental processing, deduplication, and recovery capabilities.
• Design data models and curated datasets for reporting, analytics, and downstream applications.
• Optimize Spark workloads, including cluster configuration and query performance.
• Implement data quality checks, monitoring, logging, and error handling.
• Manage data access, governance, and lineage using Unity Catalog.
• Automate deployments using Git and CI/CD practices.
• Collaborate with architects, analysts, and business stakeholders.
• Mentor engineers and contribute to technical design and code reviews.

Kenntnisse

10+ years data engineering experience

Databricks

Apache Spark

PySpark

SQL

Python

data processing

Git

CI/CD

communication

Tools

Databricks Workflows

Delta Lake

Unity Catalog

Airflow

Azure Data Factory

Azure

AWS

GCP

dbt

Terraform

Jobbeschreibung

Lorgan Tech Solutions, LLC is seeking a Senior Data Engineer to build and optimize enterprise-grade data platforms using Databricks and Delta Lake. You will design scalable ETL/ELT pipelines and curate datasets for analytics across the organization.

Lead batch and streaming processes with robust data quality, governance, and monitoring. This remote role offers collaboration with stakeholders and mentoring of engineering teams.