Databrick Engineer

Akaasa Technologies; Malvern (Chester County); On-site; USD 120,000 - 160,000; Full time

Job summary

Akaasa Technologies is seeking a senior Data Engineer to lead design and optimization of data pipelines using Databricks. You will collaborate with business analysts to ensure data availability and accuracy powering financial reporting.

The role emphasizes migrating workloads from EMR to Databricks, building scalable pipelines, and mentoring analysts. Strong Python, SQL, Spark, and AWS experience are essential.

Qualifications

• Bachelor's degree in CS, Engineering, Information Systems, or related field.
• 5–7+ years of enterprise Data Engineering experience.
• Strong Python development experience.
• Advanced SQL programming skills.
• Extensive experience with Apache Spark and PySpark.
• Strong experience with Databricks.
• Hands-on AWS data services experience (S3, Glue, Athena, Lambda, Step Functions, EventBridge).
• Experience building enterprise ETL/ELT pipelines.
• Understanding of distributed data processing.
• Experience with large-scale cloud data platforms.
• Knowledge of dimensional data modeling.
• CI/CD pipelines with Git and GitHub Actions; CloudFormation/Terraform.
• Agile/Scrum methodologies.
• Excellent communication and stakeholder management.

Responsibilities

• Orchestrate data extraction from diverse sources, engineer high-performance pipelines and provide production support to ensure reliability.
• Migrate jobs from EMR to Databricks.
• Create job aids and train analysts.
• Ensure data availability and integrity of core financial engines and reporting ecosystems.

Skills

Python development

SQL programming

Apache Spark / PySpark

Databricks

AWS data services (S3, Glue, Athena, L

ETL / ELT pipelines

Distributed data processing

Cloud data platforms

Dimensional data modeling

CI/CD pipelines (Git & GitHub Actions)

Agile / Scrum

Communication / stakeholder management

Education

Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline

Tools

Databricks

Apache Spark

Airflow

Kafka / Kinesis

Terraform / CloudFormation

Git / GitHub Actions

Job description

Role Overview

* Candidates must have real hands-on experience building ETL pipeline using Databricks *

We are seeking high-caliber Data Engineers to drive data analysis projects, Migrate jobs from EMR to Databricks, create strong job aids and train analysts. In this pivotal role, you will implement and optimize sophisticated data pipelines, partnering with business analysts to guarantee the seamless availability and absolute integrity of data powering our core financial engines and reporting ecosystems.

Key Responsibilities & Outcomes

• Orchestrate data extraction from diverse legacy and modern sources, engineer high-performance scalable pipelines, and provide expert-level production support to ensure system reliability.
• Migrate jobs from EMR to Databricks.
• Create strong job aids and train analysts.
• Implement and optimize sophisticated data pipelines, partnering with business analysts to guarantee the seamless availability and absolute integrity of data powering core financial engines and reporting ecosystems.

Qualifications & Requirements

Required:

• Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline.
• 5-7+ years of enterprise Data Engineering experience.
• Strong Python development experience.
• Advanced SQL programming skills.
• Extensive experience with Apache Spark and PySpark.
• Strong experience with Databricks application.
• Hands-on experience with AWS data services including: S3, AWS Glue, Athena, Lambda, Step Functions, EventBridge.
• Experience building enterprise ETL and ELT pipelines.
• Strong understanding of distributed data processing.
• Experience working with large-scale cloud data platforms.
• Strong knowledge of dimensional data modeling.
• Experience implementing CI/CD pipelines (Git and GitHub Actions is mandatory, along with CloudFormation/Terraform).
• Familiarity with Agile/Scrum methodologies.
• Excellent communication and stakeholder management skills.

Preferred

• Financial Services or Asset Management industry experience.
• Knowledge of Apache Iceberg, Delta Lake, or Hudi.
• Experience with Airflow.
• Knowledge of Kafka or Kinesis streaming.
• Experience supporting AI/ML and advanced analytics platforms.
• AWS Professional or Specialty Certifications.