Senior GCP Data Engineer | AI-Driven Pipelines

Machine Learning Technologies, LLC; Hartford (CT); On-site; USD 120,000 - 180,000; Full time

Job summary

Machine Learning Technologies, LLC is seeking a senior data engineer to design and deliver robust data pipelines on GCP, including BigQuery, Pub/Sub, and Cloud Storage. You will build real-time processing with Kafka, manage MongoDB schemas, and implement Airflow-based workflows, collaborating across onshore/offshore teams.

You will write Python code, optimize SQL, and ensure data quality through profiling and validation.

Qualifications

• Design and implement robust data pipelines using Google Cloud Platform (GCP) services such as BigQuery, Cloud Storage, and Pub/Sub.
• Develop and manage workflows using Cloud Composer (Apache Airflow) for efficient scheduling and orchestration.
• Write clean, efficient, and scalable code in Python, leveraging advanced programming techniques.
• Craft complex SQL queries in BigQuery, including window functions, CTEs, and performance tuning strategies.
• Build and maintain real-time data processing systems using Apache Kafka.
• Model and manage NoSQL databases, particularly MongoDB, with a focus on scalable schema design.
• Utilize Shell scripting and perform Linux system administration tasks to support data infrastructure.
• Conduct data profiling and implement validation techniques to ensure data quality and integrity.
• Develop and maintain API integration scripts for seamless service automation and data exchange.
• Troubleshoot and resolve data-related issues with strong analytical and problem-solving skills.
• Create and maintain data flow diagrams to clearly communicate architecture and pipeline logic to stakeholders.
• Develop and manage workflows using Cloud Composer 3

Responsibilities

• Design and implement data pipelines on GCP using BigQuery, Cloud Storage, and Pub/Sub.
• Create and manage automated workflows with Cloud Composer for scheduling.
• Write Python code that is clean, efficient, and scalable.
• Develop complex BigQuery SQL including window functions and CTEs.
• Set up real-time data processing with Apache Kafka.
• Design NoSQL schemas in MongoDB for scalable data access.
• Use Bash shell scripting and Linux admin tasks to support data infrastructure.
• Profile data and validate quality to ensure trustworthiness.
• Create API integration scripts for automated data exchange.
• Troubleshoot data issues with strong problem-solving skills.
• Produce data flow diagrams to illustrate architecture and logic to stakeholders.
• Maintain Cloud Composer 3 workflows.

Skills

AI/ML experience

GCP data services

Python programming

SQL queries

Distributed systems

Communication skills

Agile/SAFe

offshore collaboration

Tools

BigQuery

Cloud Storage

Pub/Sub

Cloud Composer

Apache Kafka

MongoDB

Shell scripting

Linux administration

GitHub

RTC

Job description

Machine Learning Technologies, LLC is seeking a senior data engineer to design and deliver robust data pipelines on GCP, including BigQuery, Pub/Sub, and Cloud Storage. You will build real-time processing with Kafka, manage MongoDB schemas, and implement Airflow-based workflows, collaborating across onshore/offshore teams.

You will write Python code, optimize SQL, and ensure data quality through profiling and validation.