Databricks Engineer — Senior ETL & Spark Expert (Onsite PA)

Interon IT Solutions; Malvern (Chester County); Hybrid; USD 140.000 - 200.000; Vollzeit

Zusammenfassung

Interon IT Solutions is seeking an experienced Databricks Engineer in Malvern, PA. This role focuses on building scalable data pipelines, migrating jobs from EMR to Databricks, and supporting data analysis initiatives for financial apps and reporting platforms.

You will collaborate with analysts in an Agile/Scrum setting, improve data availability, and create job aids for training analysts. 7+ years of enterprise data engineering experience is required, with strong Python/SQL and Spark skills.

Qualifikationen

• 7+ years of enterprise Data Engineering experience.
• Strong Python development and advanced SQL skills.
• Hands-on Spark, PySpark, Databricks, and AWS Glue experience.
• Experience with AWS S3, Athena, Lambda, Step Functions, and EventBridge.
• Proven experience building enterprise ETL/ELT pipelines.
• Knowledge of data lakes, Medallion Architecture, and dimensional modeling.
• CI/CD with Git and GitHub Actions.
• Familiar with Agile/Scrum methodologies.
• Excellent communication and stakeholder management.

Aufgaben

• Build and optimize enterprise ETL/ELT pipelines using Databricks, Python, PySpark, and AWS Glue.
• Migrate existing jobs from Amazon EMR to Databricks.
• Extract and integrate data from legacy and modern sources.
• Support large-scale, distributed data processing and cloud data platforms.
• Ensure data quality, availability, and reliability across financial and reporting systems.
• Provide production support and troubleshoot pipeline issues.
• Implement CI/CD pipelines using Git and GitHub Actions.
• Create technical documentation and job aids, and train analysts.
• Collaborate with business analysts and stakeholders in an Agile/Scrum environment.

Kenntnisse

Python development

SQL programming

Apache Spark

Databricks

AWS Glue

Git & CI/CD

Agile/Scrum

Data pipelines

Ausbildung

Bachelor’s degree in Computer Science/Engineering/IS

Tools

GitHub Actions

Terraform

CloudFormation

AWS

Jobbeschreibung

Interon IT Solutions is seeking an experienced Databricks Engineer in Malvern, PA. This role focuses on building scalable data pipelines, migrating jobs from EMR to Databricks, and supporting data analysis initiatives for financial apps and reporting platforms.

You will collaborate with analysts in an Agile/Scrum setting, improve data availability, and create job aids for training analysts. 7+ years of enterprise data engineering experience is required, with strong Python/SQL and Spark skills.