Senior Databricks Data Engineer — AWS & Spark

Anagh Technologies Inc; Malvern (Chester County); On-site; USD 120,000 - 160,000; Full time

Job summary

Anagh Technologies Inc. is seeking an experienced Data Engineer in Malvern, PA to design, develop, and optimize scalable data pipelines using Python, PySpark, Apache Spark, and Databricks.

You will migrate ETL workloads from Amazon EMR to Databricks, and build AWS-based data solutions with Glue, S3, Athena, Lambda, and Step Functions. The role requires strong SQL and Python skills, expertise in data lakes and Medallion Architecture, and hands-on experience with CI/CD and IaC tools.

Qualifications

• 5–7+ years of enterprise Data Engineering experience.
• Strong hands-on experience with Databricks, Apache Spark, and PySpark.
• Advanced Python development and SQL programming skills.
• Extensive experience building and optimizing enterprise ETL/ELT data pipelines.
• Strong experience with AWS data services, including S3, Glue, Athena, Lambda, Step Functions, and EventBridge.
• Experience migrating data processing jobs from Amazon EMR to Databricks.
• Strong understanding of data lakes, Medallion Architecture, distributed processing, and dimensional data modeling.
• Experience implementing CI/CD pipelines using Git, GitHub, and GitHub Actions.
• Hands-on experience with Infrastructure as Code tools such as Terraform or AWS CloudFormation.
• Experience with data extraction from legacy and modern sources, pipeline optimization, and production support.
• Strong communication, documentation, stakeholder management, and collaboration skills.
• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field.

Responsibilities

• Design, develop, and optimize scalable data pipelines using Python, PySpark, Apache Spark, and Databricks.
• Migrate existing ETL jobs and data processing workloads from Amazon EMR to Databricks.
• Build and maintain AWS-based data solutions using Glue, S3, Athena, Lambda, Step Functions, and EventBridge.
• Implement data lake solutions following Medallion Architecture principles.
• Ensure data quality, integrity, reliability, and availability across enterprise data platforms.
• Develop and maintain CI/CD workflows using GitHub Actions and Infrastructure as Code tools.
• Troubleshoot production issues and optimize pipeline performance.
• Create technical documentation, job aids, and training materials for analysts.
• Partner with business analysts and stakeholders to support data analysis, financial reporting, and business-critical applications.
• Follow Agile/Scrum practices and contribute to continuous improvement of data engineering processes.

Skills

Databricks

Apache Spark

PySpark

Python

SQL

ETL/ELT pipelines

AWS data services

Migrate EMR to Databricks

Data lakes/Medallion Architecture

CI/CD (GitHub Actions)

Infrastructure as Code (Terraform/CFN)

Education

Bachelor's degree in Computer Science or related field

Tools

Terraform

AWS CloudFormation

GitHub Actions

Git

Job description

Anagh Technologies Inc. is seeking an experienced Data Engineer in Malvern, PA to design, develop, and optimize scalable data pipelines using Python, PySpark, Apache Spark, and Databricks.

You will migrate ETL workloads from Amazon EMR to Databricks, and build AWS-based data solutions with Glue, S3, Athena, Lambda, and Step Functions. The role requires strong SQL and Python skills, expertise in data lakes and Medallion Architecture, and hands-on experience with CI/CD and IaC tools.