Data Engineer

Machine Learning Technologies, LLC; Hartford (CT); Presencial; USD 120 000 - 180 000; Tempo integral

Resumo da oferta

Machine Learning Technologies, LLC is seeking a senior data engineer to design and deliver robust data pipelines on GCP, including BigQuery, Pub/Sub, and Cloud Storage. You will build real-time processing with Kafka, manage MongoDB schemas, and implement Airflow-based workflows, collaborating across onshore/offshore teams.

You will write Python code, optimize SQL, and ensure data quality through profiling and validation.

Qualificações

• Design and implement robust data pipelines using Google Cloud Platform (GCP) services such as BigQuery, Cloud Storage, and Pub/Sub.
• Develop and manage workflows using Cloud Composer (Apache Airflow) for efficient scheduling and orchestration.
• Write clean, efficient, and scalable code in Python, leveraging advanced programming techniques.
• Craft complex SQL queries in BigQuery, including window functions, CTEs, and performance tuning strategies.
• Build and maintain real-time data processing systems using Apache Kafka.
• Model and manage NoSQL databases, particularly MongoDB, with a focus on scalable schema design.
• Utilize Shell scripting and perform Linux system administration tasks to support data infrastructure.
• Conduct data profiling and implement validation techniques to ensure data quality and integrity.
• Develop and maintain API integration scripts for seamless service automation and data exchange.
• Troubleshoot and resolve data-related issues with strong analytical and problem-solving skills.
• Create and maintain data flow diagrams to clearly communicate architecture and pipeline logic to stakeholders.
• Develop and manage workflows using Cloud Composer 3

Responsabilidades

• Design and implement data pipelines on GCP using BigQuery, Cloud Storage, and Pub/Sub.
• Create and manage automated workflows with Cloud Composer for scheduling.
• Write Python code that is clean, efficient, and scalable.
• Develop complex BigQuery SQL including window functions and CTEs.
• Set up real-time data processing with Apache Kafka.
• Design NoSQL schemas in MongoDB for scalable data access.
• Use Bash shell scripting and Linux admin tasks to support data infrastructure.
• Profile data and validate quality to ensure trustworthiness.
• Create API integration scripts for automated data exchange.
• Troubleshoot data issues with strong problem-solving skills.
• Produce data flow diagrams to illustrate architecture and logic to stakeholders.
• Maintain Cloud Composer 3 workflows.

Conhecimentos

AI/ML experience

GCP data services

Python programming

SQL queries

Distributed systems

Communication skills

Agile/SAFe

offshore collaboration

Ferramentas

BigQuery

Cloud Storage

Pub/Sub

Cloud Composer

Apache Kafka

MongoDB

Shell scripting

Linux administration

GitHub

RTC

Descrição da oferta de emprego

• Required - Must have experience AI - GPT, Gemini. Must be able to demonstrate how they use it. Using it as business use cases on resume. Must be highlighted clear on the resume
• Required - minimum 7 years of proven hands-on experience with GCP data services.
• Preferred - GCP Google Cloud Professional Data Engineer Certification.
• Required Strong understanding of distributed systems and data engineering best practices.
• Required Excellent communication and documentation skills.
• Required Ability to work independently and collaboratively in a fast-paced environment.
• Required experience in Agile methodology with preferred experience in SAFe methodology
• Required good communication and collaborative skills
• Required experience working in onshore/offshore support model collaborating work with offshore teams.
• Preferred experience working with GitHub, RTC, automation tools.

Required:

• Design and implement robust data pipelines using Google Cloud Platform (GCP) services such as BigQuery, Cloud Storage, and Pub/Sub.
• Develop and manage workflows using Cloud Composer (Apache Airflow) for efficient scheduling and orchestration.
• Write clean, efficient, and scalable code in Python, leveraging advanced programming techniques.
• Craft complex SQL queries in BigQuery, including window functions, CTEs, and performance tuning strategies.
• Build and maintain real-time data processing systems using Apache Kafka.
• Model and manage NoSQL databases, particularly MongoDB, with a focus on scalable schema design.
• Utilize Shell scripting and perform Linux system administration tasks to support data infrastructure.
• Conduct data profiling and implement validation techniques to ensure data quality and integrity.
• Develop and maintain API integration scripts for seamless service automation and data exchange.
• Troubleshoot and resolve data-related issues with strong analytical and problem-solving skills.
• Create and maintain data flow diagrams to clearly communicate architecture and pipeline logic to stakeholders.
• Develop and manage workflows using Cloud Composer 3

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.