Databricks Data Engineer

Intone Inc; Pennsylvania; Vor Ort; USD 120.000 - 160.000; Teilzeit

Zusammenfassung

Intone Inc in Malvern, PA seeks a Data Engineer for a 6+ month contract to design and optimize enterprise-scale data pipelines using Databricks and related cloud technologies. You will migrate workloads to modern platforms and provide production support to ensure data integrity across financial data ecosystems.

The role requires strong Python, SQL, Spark/PySpark expertise and hands-on Databricks experience, plus AWS data services like S3, Glue, Athena and Lambda.

Qualifikationen

• A Bachelor's degree in Computer Science, Engineering, Information Systems or related field.
• 8+ years of enterprise Data Engineering experience.
• Strong hands-on Python development experience.
• Advanced SQL programming skills.
• Extensive experience with Apache Spark and PySpark.
• Strong hands-on experience with Databricks.
• Hands-on experience with AWS data services: S3, AWS Glue, Athena, Lambda, Step Functions, and EventBridge.
• Experience building enterprise ETL and ELT pipelines.
• Strong understanding of distributed data processing.
• Experience with large-scale cloud data platforms.
• Strong knowledge of dimensional data modeling.
• Experience implementing CI/CD pipelines.
• Experience with Git and GitHub Actions.
• Familiarity with Agile/Scrum methodologies.
• Excellent communication and stakeholder management skills.

Aufgaben

• Design and implement high-performance, scalable ETL/ELT pipelines using Databricks, Spark, and AWS services.
• Orchestrate data extraction and transformation from diverse sources.
• Migrate data processing jobs from EMR to Databricks.
• Develop and maintain CI/CD pipelines for data infrastructure.
• Collaborate with business analysts and stakeholders to ensure data availability and quality.
• Provide production support and troubleshooting for data platforms.
• Create documentation and job aids to support operations and training.
• Implement dimensional data modeling and data lake architectures (Medallion Architecture).

Kenntnisse

Python development

SQL

Apache Spark

Databricks

CI/CD pipelines

Ausbildung

Bachelor's degree in Computer Science/Engineering/IS

Tools

Git

GitHub Actions

Apache Airflow

Kafka

Kinesis

Jobbeschreibung

We are seeking a Data Engineer for a contract position (6+ months) based in Malvern, Pennsylvania. In this role, you will design, build, and optimize enterprise-scale data pipelines using Databricks and related cloud technologies. You will orchestrate data extraction from diverse sources, migrate workloads to modern platforms, and provide production support to ensure system reliability and data integrity across financial data ecosystems.

Responsibilities

• Design and implement high-performance, scalable ETL and ELT pipelines using Databricks, Apache Spark, and AWS data services
• Orchestrate data extraction and transformation from legacy and modern data sources
• Migrate data processing jobs from EMR to Databricks
• Develop and maintain CI/CD pipelines for data infrastructure
• Collaborate with business analysts and stakeholders to ensure data availability, quality, and integrity
• Provide expert-level production support and troubleshooting for data platforms
• Create documentation and job aids to support team operations and training
• Implement dimensional data modeling and data lake architectures (Medallion Architecture)

Qualifications

• Required
  • Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline
  • 8+ years of enterprise Data Engineering experience
  • Strong hands-on Python development experience
  • Advanced SQL programming skills
  • Extensive experience with Apache Spark and PySpark
  • Strong hands-on experience with Databricks
  • Hands-on experience with AWS data services: S3, AWS Glue, Athena, Lambda, Step Functions, and EventBridge
  • Experience building enterprise ETL and ELT pipelines
  • Strong understanding of distributed data processing
  • Experience working with large-scale cloud data platforms
  • Strong knowledge of dimensional data modeling
  • Experience implementing CI/CD pipelines
  • Experience with Git and GitHub Actions
  • Familiarity with Agile/Scrum methodologies
  • Excellent communication and stakeholder management skills
• Preferred
  • Financial Services or Asset Management industry experience
  • Knowledge of Apache Iceberg, Delta Lake, or Hudi
  • Experience with Apache Airflow
  • Knowledge of Kafka or Kinesis streaming
  • Experience supporting AI/ML and advanced analytics platforms
  • AWS Professional or Specialty Certifications