Principal Data Engineering - Sumare, Brasil

3M; Sumaré; Presencial; BRL 240 000 - 420 000; Tempo integral

Resumo da oferta

3M is seeking a Principal Data Engineer to join the Corporate Research Digital Platforms (CRDP) in shaping scalable Data Mesh and enterprise data solutions. You will lead data pipelines, governance, and architecture across global teams, enabling insights for multiple industries.

This role requires deep expertise in Python, Spark, Databricks, and cloud data platforms. You will mentor teams, define standards, and drive digital transformation across the organization.

Qualificações

• Bachelor's degree or higher in Computer Science or equivalent education/experience.
• Solid experience in data management, governance, and data warehouse/lakehouse design with SQL and NoSQL.
• Extensive experience with Python, Spark, PySpark, and Databricks platforms.
• Hands-on Python to extract data from APIs and build data pipelines.
• Strong background in data engineering, data systems, governance, and handling structured/unstructured data.
• Advanced English proficiency.

Responsabilidades

• Architect, design, and build scalable, efficient data operations.
• Collaborate with senior leadership, analysts, engineers, and scientists on data initiatives.
• Drive architecture for data integration, modeling, governance, and applications.
• Explore new tools to optimize the data platform and practices.
• Improve data engineering and analytics engineering best practices.
• Collaborate on physical data models and mappings with domain teams.
• Develop advanced digital solutions with scientists and informaticians.
• Perform code reviews and maintain high software quality through testing.
• Develop scalable data pipelines for ingesting and distributing data streams.
• Mentor data consumers on standards, pipelines, and data consumption.
• Guide planning, timelines, resources, and stakeholder communication.
• Foster a positive team environment and assist in recruitment and training.

Conhecimentos

Python

Apache Spark

PySpark

Databricks

SQL

NoSQL

Data governance

Data modeling

AWS

Docker

Kubernetes

Airflow

DS/ML tooling

Formação académica

Bachelor's degree in Computer Science or equivalent

Ferramentas

Databricks

Airflow

AWS

Docker

Kubernetes

Descrição da oferta de emprego

Collaborate with Innovative 3Mers Around the World

Choosing where to start and grow your career has a major impact on your professional and personal life, so it’s equally important you know that the company that you choose to work at, and its leaders, will support and guide you. With a diversity of people, global locations, technologies, and products, 3M is a place where you can collaborate with 60,000 other curious, creative 3Mers.

The Impact You’ll Make in this Role

3M is seeking a Principal Data Engineering to join the Corporate Research Digital Platforms (CRDP) to develop scalable Data Systems and Solutions. As part of an agile team, you will enable applications in diverse markets including energy, manufacturing, personal safety, transportation, electronics, and consumer. You will have the opportunity to design and support an Enterprise Data Mesh to empower informatics and digital technologies for users across the globe:

• Architect, design, and build scalable, efficient, and fault-tolerant data operations.
• Collaborate with senior leadership, analysts, engineers, and scientists to implement new mesh domain nodes and data initiatives.
• Drive technical architecture for accelerated solution designs, including data integration, modeling, governance, and applications.
• Explore and recommend new tools and technologies to optimize the data platform.
• Improve and implement data engineering and analytics engineering best practices.
• Collaborate with data engineering and domain nodes teams to design physical data models and mappings.
• Work with scientists and informaticians to develop advanced digital solutions and promote digital transformation and technologies.
• Perform code reviews, manage code performance improvements, and enforce code maintainability standards.
• Develop and maintain scalable data pipelines for ingesting, transforming, and distributing data streams.
• Advise and mentor 3M businesses, data scientists, and data consumers on data standards, pipeline development, and data consumption.
• Provide technical guidance and mentorship, ensure adherence to best practices, and maintain high software quality through rigorous testing and code reviews.
• Guide project planning and execution, manage timelines and resources, and facilitate effective communication between team members and stakeholders.
• Foster a positive team environment, assist in recruitment, and provide training opportunities to address skill gaps.

Your Skills and Expertise

To set you up for success in this role from day one, 3M requires (at a minimum) the following qualifications:

• Bachelor's degree or higher in Computer Science, or equivalent education/experience.
• Solid and Professional experience in data management, data governance, and data warehouse/Lakehouse design and development with proficiency across SQL and NoSQL data management systems.
• Extensive experience and proficiency with Python, Apache Spark, PySpark, and Data Warehouse/Lakehouse (preferably Databricks)
• Must have hands-on experience in Python to extract data from APIs, build data pipelines.
• Exceptional background in data engineering, data systems, and data governance and having comfort working with structured and unstructured data and analyses.
• Advanced English.

Additional qualifications that could help you succeed even further in this role include:

• Familiarity with data mesh style architecture design principles.
• Proficiency in building data pipelines to integrate business applications and procedures.
• Solid understanding preferred of advanced Databricks concepts like Delta Lake, MLFlow, Advanced Notebook Features, Custom Libraries and Workflows, Unity Catalog, etc.
• Experience with AWS cloud computing services and infrastructure developing data lakes and data pipelines leveraging multiple technologies such as AWS S3, AWS Glue, Elastic MapReduce, etc. and awareness of considerations for building scalable, distributed computational systems on Spark.
• Experience with stream-processing systems: Amazon Kinesis, Spark, Storm, Kafka, etc.
• Hands-on experience with relational SQL and NoSQL databases.
• Data quality and validation principles experience, security principles data encryption, access control, authentication & authorization.
• Deep experience in definition and implementation of feature engineering.
• Experience with Docker containers and Kubernetes, experience developing or interacting with APIs.
• Experience in using data orchestration workflows using open-source tools like Temporal.io, Apache Airflow is a plus.
• Knowledge of data visualization tools like Dash Apps, Tableau, Power BI, etc.
• Good experience with agile development processes and concepts with leveraging project management tools like JIRA and Confluence.
• Devise and implement data engineering best practices across teams, optimize and redesign existing data engineering solutions to improve efficiency or stability, as well as monitoring of and consulting with domain node teams.
• Excellent interpersonal, collaborative, team building, and communication skills to ensure effective collaborations with matrixed teams.

Converse com Max