Senior Data Engineer — Databricks & Scientific Workflows

Tech Observer; Cambridge (MA); On-site; USD 150,000 - 190,000; Full time

Job summary

Tech Observer is seeking a highly skilled Data Engineer in Cambridge, MA to design, develop, and maintain scalable data platforms and automated bioinformatics/data science pipelines for Discovery R&D. You will work across Databricks, Nextflow Tower, and cloud-native architectures with cross-functional teams.

The role emphasizes data lakehouse modernization, medallion architecture, data quality, security, and reproducible workflows in AWS.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Data Engineering, Bioinformatics, or a related technical field.
• Master's degree preferred.
• 5+ years of experience in data engineering, cloud engineering, or analytics platform development.
• 3+ years of hands-on experience with Databricks and Apache Spark.
• 2+ years of experience with Seqera Platform (Nextflow Tower) and Nextflow workflows.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and cloud-native technologies.
• Build and optimize ETL/ELT processes for structured, semi-structured, and unstructured data.
• Develop data ingestion frameworks for research, laboratory, clinical, and external scientific datasets.
• Implement data quality, validation, monitoring, and governance processes.
• Support enterprise data lakehouse architecture and data platform modernization initiatives.
• Databricks Administration & Development: notebooks, workflows, Delta Live Tables, and Jobs.
• Create optimized Spark-based transformations and data processing solutions.
• Implement Medallion Architecture (Bronze, Silver, Gold) for data lifecycle management.
• Manage Delta Lake environments and optimize performance, scalability, and cost.
• Integrate Databricks with cloud-native services and enterprise applications.
• Seqera Platform & Scientific Workflow Management: deploy, configure, and support Nextflow Tower.
• Develop and maintain Nextflow pipelines for bioinformatics, genomics, imaging, AI/ML, and scientific computing workloads.
• Integrate Seqera workflows with AWS compute environments; containerized workflows with Docker/Kubernetes.
• Enable reproducible, scalable scientific data processing workflows; design cloud-based data solutions in AWS.
• Manage cloud storage solutions including S3 and data lifecycle policies.
• Develop Infrastructure-as-Code using Terraform or CloudFormation.
• Implement security controls and access management per enterprise IT standards.
• Partner with data scientists, researchers, bioinformaticians, and business stakeholders to understand data requirements.
• Provide technical guidance on data engineering best practices and workflow automation.
• Troubleshoot pipeline failures, performance issues, and workflow bottlenecks.
• Contribute to platform roadmaps and continuous improvement initiatives.
• Maintain technical documentation, SOPs, and knowledge articles.

Job description

Tech Observer is seeking a highly skilled Data Engineer in Cambridge, MA to design, develop, and maintain scalable data platforms and automated bioinformatics/data science pipelines for Discovery R&D. You will work across Databricks, Nextflow Tower, and cloud-native architectures with cross-functional teams.

The role emphasizes data lakehouse modernization, medallion architecture, data quality, security, and reproducible workflows in AWS.