Senior Data Engineer / Architect, Databricks, Latin America, Mexic, Canada

CloudGeometry; Porto Alegre; Presencial; BRL 100.000 - 130.000; Tempo integral

Resumo da oferta

A leading company in the life sciences industry is seeking a Senior Data Engineer/Architect to join their flagship project. This role involves designing and optimizing data pipelines within the Databricks ecosystem, ensuring data quality and security, and collaborating with teams across the US and Europe. The ideal candidate will have extensive experience in data engineering and a passion for AI-driven solutions.

Qualificações

• 8+ years of experience in software development, preferably in data engineering.
• 5+ years of experience with the DataBricks ecosystem.

Responsabilidades

• Design, develop, and optimize data pipelines and workflows within Databricks.
• Lead architecture discussions to implement advanced analytics solutions.

Conhecimentos

Python

TypeScript

Communication

Data Quality

Formação académica

Bachelor’s degree in Computer Science

Master’s degree

Ferramentas

Databricks

Spark

Glue

Delta tables

Iceberg

AWS

Git

Descrição da oferta de emprego

CloudGeometry

14/05/2025

We are looking for a Senior Data Engineer/Architect with extensive hands-on experience in the Databricks ecosystem and exceptional communication skills to join our flagship project: a cutting-edge Data Platform for the life sciences industry. This platform supports industry leaders such as Pfizer, Moderna, and Novartis in developing innovative RNA-based solutions, leveraging data-driven research, cloud computing, and advanced AI capabilities. This role offers a unique opportunity for an experienced data engineer to take the next step in their career, playing a key technical leadership role in the rapidly evolving world of AI-driven solutions.

Responsibilities

1. Design, develop, and optimize data pipelines and workflows within the Databricks platform.
2. Lead architecture discussions with engineering, product managers, and data scientists to implement advanced analytics solutions that drive business insights.
3. Build, optimize, and fine-tune Databricks workflows to improve performance and reliability.
4. Work closely with data scientists and analysts to ensure data quality.
5. Ensure the integrity, accuracy, and security of data across all processing stages.
6. Implement data ingestion from various sources into Databricks, ensuring data quality and reliability.
7. Participate in daily Scrum ceremonies and collaborate with team members in the US and Europe, with required online presence from 9 AM to 5 PM EST (for candidates in Europe until 1 PM EST).

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field; Master’s degree preferred.
• 8+ years of experience in software development, preferably in data engineering, data warehousing, or data analytics teams.
• 5+ years of experience with the DataBricks ecosystem.
• Expertise in Python and TypeScript.
• Hands-on experience with Lake House architecture.
• Experience with Spark/Glue, Delta tables, and Iceberg.
• Proven ability to design and implement scalable data pipelines/ETL processes using Databricks.
• Knowledge of cloud-based data storage and processing technologies, especially AWS services like S3, Step Functions, Lambda, and Airflow.
• Familiarity with CI/CD practices, version control (Git), automated testing, and Agile methodologies.
• Experience working in US-led high-tech companies and startups.

Nice to Have

• DataBricks certifications.
• AWS or Azure DevOps or SA certifications.
• Knowledge of DevOps and MLOps principles.
• Experience working with Data Scientists and ML Developers.
• Leadership experience in technology services companies.