Custom Software Engineer

Hackajob; Pune District, Mumbai, Bengaluru; Hybrid; INR 2,500,000 - 4,200,000; Full time

Job summary

Hackajob in Pune is seeking an experienced Data Engineer to design and optimize data pipelines on Google Cloud Platform. The role involves building batch and streaming pipelines, data transformations in Dataform with BigQuery, and event-driven processing using Pub/Sub and Cloud Run.

You will manage data across GCS, BigQuery, and Spanner, write clean Python code leveraging GCP SDKs, and contribute to internal tooling. A strong focus on testing, CI/CD, and IaC is essential.

Qualifications

• Minimum 5 years of data engineering experience.
• At least 2 years on Google Cloud Platform.
• Strong Python, clean code, OOP and packaging.
• Hands-on BigQuery performance optimisation and data modelling.
• Experience with Pytest unit testing and CI/CD.
• Terraform for GCP provisioning and IaC.

Responsibilities

• Data pipeline development across batch and streaming workloads.
• Design and implement scalable data pipelines with Python and GCP.
• Build data transformation workflows in Dataform integrated with BigQuery.
• Develop event-driven/real-time processing using Pub/Sub and Cloud Functions/Run.
• Manage large-scale data in GCS, BigQuery, and Cloud Spanner.
• Leverage Python & Google Cloud SDKs for data workloads.
• Develop reusable Python modules and internal tooling SDKs.
• Ensure code quality with unit tests and code reviews.
• Provision GCP infrastructure using Terraform (IaC).
• Maintain CI/CD pipelines in GitLab for testing and deployment.
• Monitor, alert, and observe pipelines and GCP resources.
• Containerise workloads for Cloud Run with Dockerfiles.

Skills

Data engineering

Python programming

GCP experience

BigQuery

Unit testing (pytest)

Tools

GCP services

Pub/Sub

Cloud Run

Terraform

GitLab CI/CD

Docker

Dataflow/Beam

Job description

Key Responsibilities

• Data Pipeline Development


• Design and implement robust, scalable batch and streaming data pipelines using Python and GCP services.


• Build and maintain data transformation workflows in Dataform (SQLX) integrated with BigQuery.


• Develop event-driven and real-time data processing solutions using Pub/Sub and Cloud Functions / Cloud Run.


• Manage large-scale structured and unstructured data stored in GCS, BigQuery, and Cloud Spanner.


• Python & Google SDK


• Write clean, well-structured Python code leveraging GCP client libraries (google-cloud-bigquery, google-cloud-storage, google-cloud-pubsub, google-cloud-spanner, etc.).


• Develop reusable Python modules, utilities, and SDKs for internal data platform tooling.


• Ensure all code meets quality standards through thorough unit testing with pytest and code reviews.


• Infrastructure & DevOps


• Provision and manage GCP infrastructure using Terraform, following infrastructure-as-code (IaC) best practices.


• Maintain CI/CD pipelines in GitLab for automated testing, linting, and deployment of data workloads.


• Understanding of monitoring, alerting, and observability for data pipelines and GCP resources.


• Manage containerised workloads on Cloud Run, including Dockerfiles and container registries.

Skills

• 5+ years of experience in data engineering with at least 2 years on GCP.


• Strong Python programming skills clean code, OOP, packaging, and performance optimization.


• Hands-on experience with BigQuery: query optimization, partitioning, clustering, and data modelling.


• Experience with Pub/Sub for event-driven architectures and streaming ingestion.


• Proficiency in Cloud Run and/or Cloud Functions for serverless and containerized workloads.


• Experience writing unit tests with pytest mocking, fixtures, coverage reporting.


• Hands-on Terraform experience for GCP resource provisioning (projects, IAM, networking, data services).


• Experience with GitLab CI/CD pipeline authoring, environment management, secrets handling.


• Solid understanding of GCS for data lake patterns file formats (Parquet, Avro, JSON), lifecycle policies.

Nice to Have

• Experience with Dataform for SQL-based transformation pipelines in BigQuery.


• Familiarity with Cloud Spanner for globally distributed relational workloads.


• Knowledge of data mesh or data lakehouse architecture patterns.


• Knowledge/Experience with Apache Beam / Dataflow for large-scale streaming or batch processing.


• GCP Professional Data Engineer or Cloud Architect certification.


• Familiarity with Apache Airflow / Cloud Composer, or similar orchestration tools.

Additional Information

- The candidate should have minimum 3 years of experience in Google Cloud Platform Architecture.