GCP Data Engineer

Fixity Technologies; Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

Fixity Technologies is seeking an experienced Senior Data Engineer to design, develop, and optimize scalable data pipelines for large-scale processing. You will build cloud-based data engineering solutions on Google Cloud Platform (GCP) using BigQuery, DataProc, Airflow, and Pub/Sub, and you will implement batch and real-time processing with Kafka and GCP Pub/Sub.

Proficiency in Spark/Hadoop/Hive and Python is required.

Qualifications

• 8+ years of experience in software development and/or Big Data Engineering.

Responsibilities

• Design, develop, and optimize scalable data pipelines for large-scale data processing and transformation.

Skills

GCP

BigQuery

Apache Airflow

Cloud Composer

DataProc

Pub/Sub

Apache Spark

Hadoop

Hive

Python

Shell Scripting

SQL

Jenkins

Git

GitHub

CI/CD

distributed systems

SDLC

Agile/Scrum

Tools

GCP services

BigQuery

Airflow/Cloud Composer

DataProc

Pub/Sub

Kafka

Job description

• Design, develop, and optimize scalable data pipelines for large-scale data processing and transformation.
• Develop cloud-based data engineering solutions using GCP services, including BigQuery, DataProc, Airflow, and Pub/Sub.
• Design and optimize data models, databases, and SQL queries for performance and scalability.
• Build batch and real-time/event-driven data processing pipelines using Apache Kafka and/or GCP Pub/Sub.
• Develop data processing solutions using Apache Spark, Hadoop, and Hive.
• Write efficient and maintainable code using Python and Shell Scripting.
• Apply strong Object-Oriented Programming concepts and software design patterns.
• Design and develop solutions for distributed and multi-tiered systems.
• Build and maintain CI/CD pipelines using Jenkins, Git, and GitHub.
• Implement automated testing frameworks and quality practices for data engineering solutions.
• Follow software development lifecycle processes and Agile/Scrum methodologies.
• Collaborate effectively with product teams, architects, developers, and other cross-functional teams.
• Analyze complex data engineering problems and develop effective, scalable solutions.
• Identify opportunities for automation, process improvement, and continuous improvement.
• Learn and adopt enterprise frameworks and emerging technologies to build modern Big Data solutions.

Required Skills

• 8+ years of experience in software development and/or Big Data Engineering.
• Strong hands‑on experience with Google Cloud Platform (GCP).
• Strong experience with BigQuery.
• Experience with Apache Airflow / Cloud Composer.
• Experience with Google Cloud DataProc.
• Experience with GCP Pub/Sub and/or Apache Kafka.
• Strong knowledge of Apache Spark, Hadoop, and Hive.
• Strong programming experience in Python.
• Experience with Shell Scripting.
• Strong SQL and relational database experience.
• Experience designing and optimizing data models and data pipelines.
• Experience with Jenkins, Git, and GitHub.
• Knowledge of CI/CD pipelines and automated testing frameworks.
• Strong understanding of distributed systems, algorithms, and software design patterns.
• Good understanding of Object-Oriented Programming concepts.
• Experience working in Agile/Scrum environments and understanding of SDLC processes.
• Strong analytical, communication, and problem-solving skills.

Preferred Qualifications

• Google Cloud Professional Data Engineer Certification is a plus.
• Experience with enterprise Big Data frameworks and architecture.
• Knowledge of infrastructure automation and configuration management.
• Experience with continuous integration and deployment practices.
• Strong curiosity and willingness to learn new technologies.