Senior Data Engineer

Upteam; United States; Remote; USD 100,000 - 150,000; Full time

Job summary

An innovative company is seeking a Senior Data Engineer/Architect to lead the development of a cutting-edge Data Platform for the life sciences sector. This role involves designing and optimizing data pipelines within the Databricks ecosystem, ensuring data integrity and quality while collaborating with data scientists and analysts. With a focus on advanced analytics and cloud technologies, this position offers a unique opportunity to influence data-driven solutions for industry leaders. Join a dynamic team where your expertise will drive impactful results in AI-driven projects, making a significant contribution to the future of healthcare.

Qualifications

• 10+ years in software development, focusing on data engineering and analytics.
• 5+ years experience with Databricks ecosystem and Lake House architecture.

Responsibilities

• Design and optimize data pipelines within the Databricks platform.
• Lead discussions on advanced analytics solutions with cross-functional teams.

Skills

Python

Typescript

Databricks

Data Pipeline Design

Cloud Technologies (AWS)

Data Quality Assurance

Agile Methodologies

Communication Skills

Education

Bachelor's degree in Computer Science

Master's degree in a related field

Tools

Databricks

AWS (S3, Step Functions, Lambda, Airflow)

Git

Job description

We are looking for a Senior Data Engineer/Architect with extensive hands-on experience in the Databricks ecosystem and exceptional communication skills to join our flagship project: a cutting-edge Data Platform for the life sciences industry.

This platform supports industry leaders such as Pfizer, Moderna, and Novartis in developing innovative RNA-based solutions, leveraging data-driven research, cloud computing, and advanced AI capabilities. This role offers a unique opportunity for an experienced data engineer to take the next step in their career, playing a key technical leadership role in the rapidly evolving world of AI-driven solutions.

Responsibilities

1. Design, develop, and optimize data pipelines and workflows within the Databricks platform.
2. Lead architecture discussions with engineering, product managers, and data scientists to implement advanced analytics solutions that drive business insights.
3. Build, optimize, and fine-tune Databricks workflows to improve performance and reliability.
4. Work closely with data scientists and analysts to ensure data quality.
5. Ensure the integrity, accuracy, and security of data across all processing stages.
6. Implement data ingestion from various sources into Databricks, ensuring data quality and reliability.
7. Participate in daily Scrum ceremonies and collaborate with team members in the US and Europe, with required online presence from 9 AM to 5 PM EST.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related field; Master's degree preferred.
• 10+ years of experience in the software development industry, preferably in data engineering, data warehousing, or data analytics teams.
• 5+ years of experience with the Databricks ecosystem.
• Expertise in Python and Typescript.
• Deep understanding and hands-on experience with Lake House architecture.
• Experience with Spark/Glue and Delta tables/Iceberg.
• Proven ability to design and implement complex, scalable data pipelines/ETL processes using Databricks.
• Skilled in cloud-based data storage and processing technologies, especially AWS services such as S3, Step Functions, Lambda, and Airflow.
• Familiarity with CI/CD practices, version control (Git), automated testing, and Agile methodologies.
• Experience working in US-led high-tech companies and startups.
• Strong communication skills to articulate ideas clearly, present findings persuasively, and build rapport with clients and team members.

Nice to Have

• DataBricks certifications.
• AWS or Azure DevOps or SA certifications.
• Knowledge of basic DevOps and MLOps principles.
• Experience collaborating with Data Scientists and ML Developers.
• Previous management and lead developer roles in technology services companies.