Lead Databricks Data Engineer for Financial Analytics

Compunnel, Inc.

Northern (KY)

Hybrid

USD 120,000 - 170,000

Full time

5 days ago

Be an early applicant

Application generator

Turn this role into an interview — a resume and cover letter built around what this employer wants.

Get past ATS filters

Job summary

Compunnel, Inc. is seeking a Lead Data Engineer to drive scalable, cloud-native data platforms for finance analytics.

You will collaborate with data scientists, economists, and business stakeholders to translate complex analytical requirements into production ETL pipelines and robust data models. Responsibilities include building Databricks-based platforms, implementing PySpark/Delta Lake frameworks, ensuring data quality and governance, and shaping lakehouse architecture.

Qualifications

Bachelor's or Master's degree in Computer Science, Data Engineering, Economics, Finance or related field.
5+ years of experience in Data Engineering or Data Platform development.
Finance or payroll domain experience including payroll data structures and pay-period logic.
Experience handling large-scale datasets including billions of records and multi-terabyte environments.
Strong proficiency with Databricks including Unity Catalog, Delta Lake, and Workflows.
Experience with Python, SQL, PySpark, data modeling, and ETL/ELT development.
Analytical fluency with EDA, distributions, time-series patterns, and feature engineering.
Experience implementing CI/CD using Bitbucket Pipelines, Jenkins, and automated deployment frameworks.
Experience with AI-assisted development tools such as GitHub Copilot, Amazon Q, Kiro, or equivalents.
Experience with data governance, data quality, and metadata management.

Responsibilities

Design, develop, and maintain scalable data pipelines for ingestion, transformation, and distribution of payroll, macroeconomic, and financial datasets.
Build and support Databricks-based platforms enabling financial research and analytical workloads.
Implement ETL/ELT using PySpark and Delta Lake for structured and unstructured data.
Develop data models and data marts for analytical, reporting, and ML use cases.
Ensure data quality, lineage, governance, and observability across data assets.
Optimize performance for large-scale datasets
Translate business requirements into scalable data solutions.
Perform exploratory data analysis and dataset profiling.
Translate data scientist logic into efficient PySpark implementations.
Build validation dashboards and notebooks to verify outputs and quality.
Support feature engineering by implementing complex aggregation and transformation logic at scale.
Independently validate analytical outputs and conduct sanity checks.
Conduct ad hoc analytics using pandas and NumPy alongside PySpark.
Contribute to lakehouse architecture discussions and data mesh concepts.
Implement CI/CD pipelines using Databricks Asset Bundles, Bitbucket Pipelines, Jenkins.