Data Engineer Lead-GCP & BigQuery Specialist

Nexus Staffing LLC; Hyderabad; On-site; INR 350,000 - 550,000; Full time

Job summary

Nexus Staffing LLC seeks a Data Engineer Lead with 15+ years in data engineering and strong GCP/BigQuery expertise. You will design scalable pipelines, manage data warehouses, and collaborate with data scientists and business teams to deliver insights.

You will guide a team, implement streaming solutions with Dataflow and Apache Beam, and ensure governance, security, and compliance across large datasets in a fast-paced environment.

Qualifications

• 15+ years of experience in data engineering, with a focus on cloud-based solutions.
• Expertise in GCP services, including but not limited to: BigQuery, Dataflow, Pub/Sub, Cloud Storage, Cloud Composer (Airflow), Cloud Functions.
• Strong proficiency in SQL and BigQuery SQL for complex queries and performance optimization.
• Hands-on experience with Python or Java for building data pipelines and automation.
• Deep understanding of data modeling, data warehousing, and schema design.
• Experience with streaming data processing and tools like Apache Beam or Kafka.
• Familiarity with CI/CD pipelines and version control systems like Git.
• Strong knowledge of data security and compliance standards (e.g., GDPR, HIPAA).
• Proven experience in migrating on-premise data systems to GCP.
• Excellent problem-solving skills and the ability to work in a fast-paced, dynamic environment.
• Strong communication and collaboration skills, with the ability to work effectively with both technical and non-technical stakeholders.

Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows using GCP services.
• Architect and implement data warehouse solutions using BigQuery, ensuring optimal performance and cost efficiency.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights.
• Optimize and manage large-scale datasets, ensuring data quality, integrity, and security.
• Develop and enforce best practices for data governance, data modeling, and data lifecycle management.
• Implement streaming data solutions using tools like Dataflow, Pub/Sub, and Apache Beam.
• Monitor and troubleshoot data pipelines, ensuring high availability and reliability.
• Lead the migration of on-premise data systems to GCP, ensuring seamless integration and minimal downtime.
• Mentor and guide junior engineers, fostering a culture of continuous learning and innovation.
• Stay updated with the latest advancements in GCP and data engineering technologies, and recommend improvements to existing systems.

Skills

GCP BigQuery

Dataflow

Pub/Sub

Cloud Storage

Cloud Composer

Cloud Functions

SQL

BigQuery SQL

Python

Java

Data modeling

Data warehousing

Data governance

Data security

CI/CD

Git

Apache Beam

Kafka

GDPR

HIPAA

Tools

Apache Beam

Kafka

Git

Job description

Data Engineer Lead-GCP & BigQuery Specialist

Role Overview: We are seeking a highly experienced Senior Data Engineer with over 15 years of expertise in designing, building, and managing scalable data solutions. The ideal candidate will have extensive hands-on experience with Google Cloud Platform (GCP) services, particularly BigQuery, and a strong background in data pipelines, ETL/ELT processes, and data architecture.This role requires a strategic thinker who can lead complex data engineering projects, mentor junior team members, and collaborate with cross-functional teams to deliver high-quality data solutions.

Key Responsibilities:

• Design, develop, and maintain scalable and efficient data pipelines and ETL/ELT workflows using GCP services.
  
• Architect and implement data warehouse solutions using BigQuery, ensuring optimal performance and cost efficiency.
  
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights.
  
• Optimize and manage large-scale datasets, ensuring data quality, integrity, and security.
  
• Develop and enforce best practices for data governance, data modeling, and data lifecycle management.
  
• Implement streaming data solutions using tools like Dataflow, Pub/Sub, and Apache Beam.
  
• Monitor and troubleshoot data pipelines, ensuring high availability and reliability.
  
• Lead the migration of on-premise data systems to GCP, ensuring seamless integration and minimal downtime.
  
• Mentor and guide junior engineers, fostering a culture of continuous learning and innovation.
  
• Stay updated with the latest advancements in GCP and data engineering technologies, and recommend improvements to existing systems.
  

Requirements

What we are looking for:

• 15+ years of experience in data engineering, with a focus on cloud-based solutions.
  
• Expertise in GCP services, including but not limited to: BigQuery, Dataflow, Pub/Sub, Cloud Storage, Cloud Composer (Airflow), Cloud Functions.
  
• Strong proficiency in SQL and experience with BigQuery SQL for complex queries and performance optimization.\
  
• Hands-on experience with Python or Java for building data pipelines and automation.
  
• Deep understanding of data modeling, data warehousing, and schema design.
  
• Experience with streaming data processing and tools like Apache Beam or Kafka.
  
• Familiarity with CI/CD pipelines and version control systems like Git.
  
• Strong knowledge of data security and compliance standards (e.g., GDPR, HIPAA).
  
• Proven experience in migrating on-premise data systems to GCP.
  
• Excellent problem-solving skills and the ability to work in a fast-paced, dynamic environment.
  
• Strong communication and collaboration skills, with the ability to work effectively with both technical and non-technical stakeholders.