Senior Databricks Data Engineer

Alephys; Houston (TX); On-site; USD 120,000 - 180,000; Full time

Job summary

Alephys seeks a Databricks Data Engineer to design, build, and optimize a scalable data architecture. You will modernize ETL workloads, migrate from legacy systems, and ensure data assets are secure, discoverable, and performant.

You will leverage PySpark, Spark SQL, and Delta Live Tables, implement Unity Catalog governance, and build reliable CI/CD pipelines with DABs and Git-based tooling to deliver high-quality data products.

Qualifications

• 5+ years of Data Engineering experience with Databricks.
• Expert Python (PySpark) and Spark SQL.
• Deep knowledge of Databricks features: Delta Live Tables and Unity Catalog.
• CI/CD pipelines for data assets using Git and automation tools.
• Experience migrating legacy ETL (e.g., DataStage, Informatica) to cloud-native Spark.
• Strong analytical and problem-solving skills.

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines using PySpark.
• Implement declarative pipelines (Delta Live Tables) and data quality checks.
• Migrate legacy workloads to modern Databricks architectures.
• Enforce centralized data governance and security using Unity Catalog.
• Automate deployment of pipelines with DABs and CI/CD tools.
• Profile and optimize PySpark jobs and Spark SQL queries.

Skills

PySpark

Spark SQL

Databricks

CI/CD pipelines

Data governance

ETL modernization

Problem solving

Distributed computing

Tools

Git

GitHub Actions

Jenkins

Terraform

Databricks Asset Bundles

Job description

About the Role:

We are seeking a highly skilled Databricks Data Engineer to design, build, and optimize our next-generation data architecture. In this role, you will be instrumental in modernizing our data infrastructure, transitioning legacy ETL workloads to highly scalable, cloud-native solutions, and ensuring our data assets are secure, discoverable, and performant. You will leverage the latest capabilities of the Databricks platform—from declarative pipelines to advanced governance—to deliver high-quality data products to the business.

Key Responsibilities

• Pipeline Engineering: Design, build, and maintain robust, scalable ETL/ELT pipelines using PySpark and Spark SQL to process large volumes of structured and semi-structured data.
• Declarative Frameworks: Implement and manage Spark Declarative Pipelines (Delta Live Tables) to simplify pipeline development, automate data quality checks, and streamline operations.
• Legacy Modernization: Lead the migration of complex workloads from legacy enterprise ETL systems into modernized, scalable PySpark architectures.
• Data Governance & Security: Architect and enforce centralized data governance, access controls, auditing, and data lineage tracking across the organization utilizing Unity Catalog.
• CI/CD & Automation: Automate the deployment lifecycle of data pipelines, notebooks, and infrastructure using Databricks Asset Bundles (DABs) integrated with enterprise CI/CD workflows.
• Performance Optimization: Profile, tune, and optimize complex PySpark jobs and Spark SQL queries for maximum performance and cost-efficiency.

Required Skills & Qualifications

• Experience: 5+ years of experience in Data Engineering, with a heavy focus on the Databricks Data Intelligence Platform.
• Core Languages: Expert-level proficiency in Python (PySpark) and advanced Spark SQL.
• Databricks Ecosystem: Deep hands-on experience with modern Databricks features, specifically Spark Declarative Pipelines and Unity Catalog for centralized governance.
• DevOps / DataOps: Proven ability to implement CI/CD pipelines for data assets using Declarative Asset Bundles (DABs), Git, and automation tools (e.g., GitHub Actions, Jenkins, or GitLab CI).
• Architecture & Migration: Strong understanding of distributed computing principles and experience migrating legacy on-premise ETL logic (e.g., DataStage, Informatica) to cloud-native Spark environments.
• Problem Solving: Strong analytical skills with the ability to troubleshoot complex data processing issues and optimize massive data transformations.

Nice-to-Haves

• Experience with event-driven data ingestion architectures and storage-layer triggers (e.g., S3).
• Familiarity with cloud infrastructure (AWS, Azure, or GCP) and infrastructure-as-code (Terraform).
• Experience building and designing internal frameworks to accelerate data pipeline development.