Databricks Engineer W2 Position

Interon IT Solutions; Malvern (Chester County); Hybrid; USD 140,000 - 200,000; Full time

Job summary

Interon IT Solutions is seeking an experienced Databricks Engineer in Malvern, PA. This role focuses on building scalable data pipelines, migrating jobs from EMR to Databricks, and supporting data analysis initiatives for financial apps and reporting platforms.

You will collaborate with analysts in an Agile/Scrum setting, improve data availability, and create job aids for training analysts. 7+ years of enterprise data engineering experience is required, with strong Python/SQL and Spark skills.

Qualifications

• 7+ years of enterprise Data Engineering experience.
• Strong Python development and advanced SQL skills.
• Hands-on Spark, PySpark, Databricks, and AWS Glue experience.
• Experience with AWS S3, Athena, Lambda, Step Functions, and EventBridge.
• Proven experience building enterprise ETL/ELT pipelines.
• Knowledge of data lakes, Medallion Architecture, and dimensional modeling.
• CI/CD with Git and GitHub Actions.
• Familiar with Agile/Scrum methodologies.
• Excellent communication and stakeholder management.

Responsibilities

• Build and optimize enterprise ETL/ELT pipelines using Databricks, Python, PySpark, and AWS Glue.
• Migrate existing jobs from Amazon EMR to Databricks.
• Extract and integrate data from legacy and modern sources.
• Support large-scale, distributed data processing and cloud data platforms.
• Ensure data quality, availability, and reliability across financial and reporting systems.
• Provide production support and troubleshoot pipeline issues.
• Implement CI/CD pipelines using Git and GitHub Actions.
• Create technical documentation and job aids, and train analysts.
• Collaborate with business analysts and stakeholders in an Agile/Scrum environment.

Skills

Python development

SQL programming

Apache Spark

Databricks

AWS Glue

Git & CI/CD

Agile/Scrum

Data pipelines

Education

Bachelor’s degree in Computer Science/Engineering/IS

Tools

GitHub Actions

Terraform

CloudFormation

AWS

Job description

Databricks Engineer

W2 position ,No C2C

Any Visa is okay except H1b

Location: Malvern, PA — onsite 3 days per week

Experience: 7+ years in enterprise Data Engineering

Interview: First-round interview must be in person in Malvern, PA

Assessment: Ropes test required

Must-Have: Candidates must have strong, hands-on experience building ETL pipelines using Databricks.

Role Overview

Seeking an experienced Data Engineer to build and optimize scalable data pipelines, migrate jobs from Amazon EMR to Databricks, and support data analysis initiatives. This role partners with business analysts to ensure data availability and integrity for financial applications and reporting platforms. Responsibilities also include creating job aids and training analysts.

Key Responsibilities

• Build, implement, and optimize enterprise ETL/ELT pipelines using Databricks, Python, PySpark, and AWS Glue.
• Migrate existing jobs from Amazon EMR to Databricks.
• Extract and integrate data from legacy and modern sources.
• Support large-scale, distributed data processing and cloud data platforms.
• Ensure data quality, availability, and reliability across financial and reporting systems.
• Provide production support and troubleshoot pipeline issues.
• Implement CI/CD pipelines using Git and GitHub Actions.
• Create technical documentation and job aids, and train analysts.
• Collaborate with business analysts and stakeholders in an Agile/Scrum environment.

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, Information Systems, or a related field.
• 7+ years of enterprise Data Engineering experience.
• Strong Python development and advanced SQL programming skills.
• Extensive hands-on experience with Apache Spark, PySpark, Databricks, and AWS Glue.
• Experience with AWS S3, Athena, Lambda, Step Functions, and EventBridge.
• Proven experience building enterprise ETL/ELT pipelines.
• Strong understanding of distributed data processing, data lakes, and Medallion Architecture.
• Strong knowledge of dimensional data modeling.
• Experience with Git, GitHub Actions, and CI/CD implementation.
• Familiarity with Agile/Scrum methodologies.
• Excellent communication and stakeholder management skills.

Additional Technical Stack

• Infrastructure as Code: CloudFormation / Terraform
• AI/ML Platform: Amazon SageMaker

Preferred Qualifications

• Financial Services or Asset Management experience.
• Knowledge of Apache Iceberg, Delta Lake, or Hudi.
• Experience with Apache Airflow.
• Knowledge of Kafka or Amazon Kinesis streaming.
• Experience supporting AI/ML and advanced analytics platforms.
• AWS Professional or Specialty certifications.