Data Engineer – Databricks / AWS

Anagh Technologies Inc; Malvern (Chester County); On-site; USD 120,000 - 160,000; Full time

Job summary

Anagh Technologies Inc. is seeking an experienced Data Engineer in Malvern, PA to design, develop, and optimize scalable data pipelines using Python, PySpark, Apache Spark, and Databricks.

You will migrate ETL workloads from Amazon EMR to Databricks, and build AWS-based data solutions with Glue, S3, Athena, Lambda, and Step Functions. The role requires strong SQL and Python skills, expertise in data lakes and Medallion Architecture, and hands-on experience with CI/CD and IaC tools.

Qualifications

• 5–7+ years of enterprise Data Engineering experience.
• Strong hands-on experience with Databricks, Apache Spark, and PySpark.
• Advanced Python development and SQL programming skills.
• Extensive experience building and optimizing enterprise ETL/ELT data pipelines.
• Strong experience with AWS data services, including S3, Glue, Athena, Lambda, Step Functions, and EventBridge.
• Experience migrating data processing jobs from Amazon EMR to Databricks.
• Strong understanding of data lakes, Medallion Architecture, distributed processing, and dimensional data modeling.
• Experience implementing CI/CD pipelines using Git, GitHub, and GitHub Actions.
• Hands-on experience with Infrastructure as Code tools such as Terraform or AWS CloudFormation.
• Experience with data extraction from legacy and modern sources, pipeline optimization, and production support.
• Strong communication, documentation, stakeholder management, and collaboration skills.
• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field.

Responsibilities

• Design, develop, and optimize scalable data pipelines using Python, PySpark, Apache Spark, and Databricks.
• Migrate existing ETL jobs and data processing workloads from Amazon EMR to Databricks.
• Build and maintain AWS-based data solutions using Glue, S3, Athena, Lambda, Step Functions, and EventBridge.
• Implement data lake solutions following Medallion Architecture principles.
• Ensure data quality, integrity, reliability, and availability across enterprise data platforms.
• Develop and maintain CI/CD workflows using GitHub Actions and Infrastructure as Code tools.
• Troubleshoot production issues and optimize pipeline performance.
• Create technical documentation, job aids, and training materials for analysts.
• Partner with business analysts and stakeholders to support data analysis, financial reporting, and business-critical applications.
• Follow Agile/Scrum practices and contribute to continuous improvement of data engineering processes.

Skills

Databricks

Apache Spark

PySpark

Python

SQL

ETL/ELT pipelines

AWS data services

Migrate EMR to Databricks

Data lakes/Medallion Architecture

CI/CD (GitHub Actions)

Infrastructure as Code (Terraform/CFN)

Education

Bachelor's degree in Computer Science or related field

Tools

Terraform

AWS CloudFormation

GitHub Actions

Git

Job description

Interview: Must interview onsite in Malvern, PA

Job Description

We are seeking an experienced Data Engineer with strong hands-on Databricks expertise to design, develop, optimize, and support scalable data engineering solutions. The ideal candidate will have extensive experience with Python, PySpark, Apache Spark, AWS data services, and enterprise ETL/ELT pipelines.

The candidate will play a key role in migrating existing data processing jobs from Amazon EMR to Databricks, building high-performance data pipelines, and ensuring the accuracy, integrity, and availability of data supporting financial services and reporting platforms. This role also involves collaborating with business analysts, providing production support, developing technical documentation, and training analysts.

Required Skills and Qualifications

• 5–7+ years of enterprise Data Engineering experience.
• Strong hands-on experience with Databricks, Apache Spark, and PySpark.
• Advanced Python development and SQL programming skills.
• Extensive experience building and optimizing enterprise ETL/ELT data pipelines.
• Strong experience with AWS data services, including S3, Glue, Athena, Lambda, Step Functions, and EventBridge.
• Experience migrating data processing jobs from Amazon EMR to Databricks.
• Strong understanding of data lakes, Medallion Architecture, distributed processing, and dimensional data modeling.
• Experience implementing CI/CD pipelines using Git, GitHub, and GitHub Actions.
• Hands-on experience with Infrastructure as Code tools such as Terraform or AWS CloudFormation.
• Experience with data extraction from legacy and modern sources, pipeline optimization, and production support.
• Strong communication, documentation, stakeholder management, and collaboration skills.
• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field.

Preferred Qualifications

• Experience in financial services or asset management.
• Knowledge of Delta Lake, Apache Iceberg, or Apache Hudi.
• Experience with Apache Airflow or other workflow orchestration tools.
• Knowledge of Kafka or AWS Kinesis for streaming data.
• Experience supporting AI/ML and advanced analytics platforms.
• AWS Professional or Specialty certifications.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using Python, PySpark, Apache Spark, and Databricks.
• Migrate existing ETL jobs and data processing workloads from Amazon EMR to Databricks.
• Build and maintain AWS-based data solutions using Glue, S3, Athena, Lambda, Step Functions, and EventBridge.
• Implement data lake solutions following Medallion Architecture principles.
• Ensure data quality, integrity, reliability, and availability across enterprise data platforms.
• Develop and maintain CI/CD workflows using GitHub Actions and Infrastructure as Code tools.
• Troubleshoot production issues and optimize pipeline performance.
• Create technical documentation, job aids, and training materials for analysts.
• Partner with business analysts and stakeholders to support data analysis, financial reporting, and business-critical applications.
• Follow Agile/Scrum practices and contribute to continuous improvement of data engineering processes.