GCP Data Engineer

Artech L.L.C.; Telangana; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Artech L.L.C. in Telangana seeks an experienced Data Engineer to design, develop, and optimize scalable data pipelines using Python and Apache Beam within GCP environments.

The role requires strong SQL skills, data modeling, and collaboration with data scientists, analysts, and DevOps to deliver robust data solutions that support business decisions while ensuring data security and governance. Experience with CI/CD and containerization is a plus.

Qualifications

• Expertise in Python with Apache Beam for building data pipelines.
• Strong expertise in GCP environments, including BigQuery, Dataflow, Pub/Sub, and Cloud Storage.
• Advanced SQL skills to write and optimize complex queries for large datasets.
• Strong analytical skills with experience in interpreting data and providing actionable insights.
• Solid understanding of distributed systems and ETL/ELT workflows.
• Experience with CI/CD pipelines, containerization (Docker), and orchestration tools (Kubernetes, Airflow).
• Familiarity with relational and NoSQL databases and data modeling concepts.
• Excellent problem-solving skills in a fast-paced environment.

Responsibilities

• Contribute to design, development, and implementation of end-to-end data pipelines using Python and Apache Beam.
• Architect and optimize data workflows for GCP environments, ensuring scalability, reliability, and performance.
• Write complex SQL queries for data extraction, transformation, and analysis, ensuring data accuracy and consistency.
• Collaborate with cross-functional teams, including data scientists, analysts, and DevOps, to understand data requirements and deliver solutions.
• Provide technical contributions for a squad of engineers, ensuring best practices in coding, testing, and deployment.
• Develop and maintain data models, ETL/ELT processes, and data integration solutions.
• Perform data analysis to identify trends, anomalies, and insights to support business decision-making.
• Ensure data security, compliance, and governance across all environments.
• Monitor and troubleshoot data pipelines, addressing performance bottlenecks and failures.
• Stay up to date with the latest trends and technologies in data engineering, cloud computing, and distributed systems.

Skills

Python

Apache Beam

GCP

SQL

Data analysis

Tools

BigQuery

Apache Iceberg

Dataflow

Pub/Sub

Cloud Storage

Docker

Kubernetes

Airflow

Job description

Experienced with a strong background in Data Engineering to join our team. The ideal candidate will have expertise in Python, Apache Beam, and experience working with Google Cloud Platform (GCP) environments. Additionally, strong SQL and analytical skills are mandatory for this role. You will play a critical role in designing, developing, and optimizing scalable data pipelines and solutions to support our business needs.

Job Expectations:

• Contribute to design, development, and implementation of end-to-end data pipelines using Python and Apache Beam.
• Architect and optimize data workflows for GCP environments, ensuring scalability, reliability, and performance.
• Write complex SQL queries for data extraction, transformation, and analysis, ensuring data accuracy and consistency.
• Collaborate with cross-functional teams, including data scientists, analysts, and DevOps, to understand data requirements and deliver solutions.
• Provide technical contributions for a squad of engineers, ensuring best practices in coding, testing, and deployment.
• Develop and maintain data models, ETL processes, and data integration solutions.
• Perform data analysis to identify trends, anomalies, and insights to support business decision-making.
• Ensure data security, compliance, and governance across all environments.
• Monitor and troubleshoot data pipelines, addressing performance bottlenecks and failures.
• Stay up to date with the latest trends and technologies in data engineering, cloud computing, and distributed systems.

Desired Qualifications:

• Expertise in Python with Apache Beam for building data pipelines.
• Strong expertise in GCP environments, including services like BigQuery, Apache IceBerg, Dataflow, Pub/Sub, and Cloud Storage.
• Advanced SQL skills, with the ability to write and optimize complex queries for large datasets.
• Strong analytical skills, with experience in interpreting data and providing actionable insights.
• Solid understanding of distributed systems, data processing frameworks, and ETL/ELT workflows.
• Experience with CI/CD pipelines, containerization (Docker), and orchestration tools (Kubernetes, Airflow).
• Familiarity with relational and NoSQL databases, as well as data modeling concepts.
• Excellent problem-solving skills and the ability to work in a fast-paced, dynamic environment.

Additional Skills (Good to have):

• Experience with other data processing frameworks like Spark or Flink.
• Knowledge of data governance, security, and compliance best practices.
• Certification in GCP (e.g., Professional Data Engineer) is a plus.
• Experience with hybrid cloud solutions and migrating workloads from On-Premises to GCP.