Data Developer (AI / Databricks) Sênior

Jobgether; United States; Remote; USD 140,000 - 190,000; Full time

Benefits offered by this job

Health insurance

Dental insurance

Food and meal allowances

Childcare assistance

Extended parental leave

Fitness and wellbeing programs

Profit-sharing (PLR)

Life insurance

Continuous learning platform

Online education partnerships

Language learning platform

Discount program

Wellbeing platform access

Job summary

Jobgether is seeking a senior data engineer to design and optimize cloud-based data pipelines using Azure, Databricks, PySpark and SQL. You will build scalable ETL/ELT workflows, implement CI/CD deployments, and collaborate with data scientists and engineers to deliver reliable data products.

You will mentor juniors, enforce data governance, monitor performance, and contribute to data architecture and migration projects as part of a global team.

Qualifications

• Senior-level data engineering with production-grade pipelines.
• Strong experience with Azure data services and Databricks.
• Proficiency in PySpark, Python, and SQL for transformation and automation.
• Experience with CI/CD practices and automated deployment workflows.
• Git/GitHub for source control and Azure DevOps for deployment automation.

Responsibilities

• Design, develop, maintain, and optimize ETL/ELT data pipelines using Azure, Databricks, and PySpark.
• Implement automated workflows for data ingestion, transformation, integration, processing, and delivery to downstream systems.
• Monitor pipeline performance and implement improvements to data processing workflows.
• Establish and maintain data quality controls and governance.
• Collaborate with business stakeholders and data scientists to deploy AI models in cloud environments.

Skills

Azure Data Services

Databricks

PySpark

Python

SQL

CI/CD

Git & GitHub

Azure DevOps

Data Modeling

Data Pipelines

Education

Bachelor's degree in Computer Science or related

Tools

Azure Data Factory

Azure Synapse Analytics

Azure Data Lake Storage

Apache Spark

Databricks

Job description

Accountabilities

• Design, develop, maintain, and optimize ETL/ELT data pipelines using Microsoft Azure, Databricks, and PySpark.
• Implement automated workflows for data ingestion, transformation, integration, processing, and delivery to downstream systems and applications.
• Process and manage large volumes of customer registration and master data, ensuring data accuracy, consistency, integrity, and availability.
• Develop and maintain CI/CD pipelines to automate deployments, streamline development workflows, and support reliable releases.
• Manage source code, version control, and collaborative development workflows using GitHub and Azure DevOps.
• Design, implement, and optimize conceptual, logical, and physical data models to support scalability, performance, and effective data governance.
• Work with Azure Data Factory (ADF), Azure Synapse Analytics, Azure Data Lake Storage, and Spark to build and evolve cloud-based data solutions.
• Contribute to cloud migration initiatives, ensuring secure data transfer, integrity, quality, and continuity throughout the migration process.
• Collaborate with business stakeholders, data scientists, and engineering teams to integrate data solutions and deploy AI models in cloud environments.
• Establish and maintain data quality controls, including data profiling, validation, consistency checks, and ongoing monitoring.
• Monitor pipeline performance and delivery indicators, identify anomalies and bottlenecks, and implement improvements to data processing workflows.
• Automate recurring processes and queries to improve operational efficiency, reliability, and maintainability.
• Investigate data-related issues, analyze root causes, and recommend effective technical solutions.
• Document data architectures, pipeline designs, technical processes, and engineering best practices to support knowledge sharing and maintainability.
• Contribute to the evolution of data architecture, governance practices, and AI-enabled workflows, applying sound engineering principles across the data lifecycle.

Requirements

• Substantial professional experience in senior-level data engineering or data development, including the design and maintenance of production-grade data pipelines.
• Strong experience with Microsoft Azure data services and Databricks, including the development and optimization of distributed data processing solutions.
• Proficiency in PySpark, Python, and SQL for data transformation, analysis, integration, and automation.
• Strong knowledge of ETL/ELT processes and data pipeline architecture, including ingestion, transformation, orchestration, and delivery.
• Experience with Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, and Apache Spark.
• Practical knowledge of CI/CD practices and automated deployment workflows.
• Experience with Git and GitHub for source control, as well as familiarity with Azure DevOps for versioning and deployment automation.
• Demonstrated experience in conceptual, logical, and physical data modeling, with an understanding of performance optimization, scalability, and data governance.
• Experience integrating and processing large datasets, maintaining data consistency, and ensuring data quality across complex workflows.
• Familiarity with cloud migration projects and the technical considerations involved in secure and reliable data transfers.
• Ability to collaborate with business and data science teams to support AI model deployment and cloud-based data solutions.
• Strong analytical and problem-solving skills, with the ability to investigate data issues, identify root causes, and implement effective improvements.
• Excellent technical documentation and communication skills, with the ability to work collaboratively across multidisciplinary teams.
• Preferred qualifications: Experience with AI projects and automated data pipelines; knowledge of data-oriented architectures and governance best practices; involvement in Master Data Management (MDM) initiatives; familiarity with Databricks MLflow for model lifecycle management; and experience with Data Lake and Data Warehouse architectures.
• Experience with Azure Machine Learning or similar tools for deploying and managing AI models is an advantage.

Benefits

• Remote work: Home-office arrangement for professionals based in Brazil, subject to the specific office attendance policy applicable to residents of the Campinas metropolitan region.
• Health insurance: Medical coverage.
• Dental insurance: Dental coverage.
• Food and meal allowances: Benefits to support food and meal expenses.
• Childcare assistance: Financial support for eligible childcare expenses.
• Extended parental leave: Enhanced parental leave provisions to support employees and their families.
• Fitness and wellbeing: Partnerships with fitness, health, and wellness platforms, including Wellhub (Gympass) and TotalPass.
• Profit-sharing: Participation in a profit-sharing program (PLR), subject to applicable program terms.
• Life insurance: Life insurance coverage.
• Continuous learning: Access to a dedicated learning platform, including internal professional development resources.
• Online education: Partnerships with online course providers to support technical and professional development.
• Language learning: Access to a language-learning platform.
• Discount program: Access to a club offering discounts and special offers.
• Physical and mental wellbeing: Access to a free online platform dedicated to health, mental wellbeing, and overall wellness.
• Parenting support: Courses focused on pregnancy and responsible parenting.
• Inclusive workplace support: Dedicated health and wellbeing professionals, inclusion specialists, and employee affinity groups. Candidates with disabilities can request appropriate support and accommodations during the recruitment process.
• Compensation details: The original job description does not specify a salary range, bonus amount, or equity compensation. Confirm compensation, benefit eligibility, and applicable terms with the hiring team.