We are looking for a Data Engineer with strong Databricks experience to design, build, and maintain scalable data pipelines for healthcare data platforms. You will work with large, complex datasets and partner with analytics, product, and engineering teams to deliver reliable, high-quality data for reporting and advanced analytics.
Key Responsibilities
- Design, build, and optimize ETL/ELT pipelines on Databricks (Delta Lake, Unity Catalog, Workflows/Jobs).
- Develop data transformations using PySpark/Spark and SQL; build and maintain Delta Lake tables following medallion architecture (bronze/silver/gold).
- Ingest data from multiple sources (databases, APIs, files, streaming) into cloud data platforms.
- Implement data quality checks, validation, and monitoring across pipelines.
- Model data for analytics and reporting use cases; optimize performance and cost.
- Work with cloud services (Azure and/or AWS) for storage, compute, and orchestration.
- Collaborate with data architects, analysts, and QA teams in an Agile delivery model.
- Document pipelines, data models, and technical designs.
Required Skills and Experience
- 5+ years of experience in data engineering.
- 3+ years of hands-on experience with Databricks, Apache Spark, and PySpark.
- Strong SQL skills and experience with Delta Lake and data warehousing concepts.
- Experience with cloud platforms (Azure Data Services and/or AWS).
- Experience with ETL/ELT design, data modeling, and pipeline orchestration.
- Familiarity with CI/CD, Git, and DevOps practices for data platforms.
- Advanced English (written and spoken).
Nice to Have
- Healthcare domain experience (payer, provider, or life sciences data).
- Experience with streaming (Kafka, Spark Structured Streaming).
- Databricks certification.
- Experience with dbt, Airflow, or similar orchestration tools.