Gcp Data Engineer

Reyika; United States; Remote; USD 90,000 - 130,000; Full time

Job summary

A leading technology company seeks Data Engineers to enhance their data infrastructure and support health services via data solutions. The ideal candidate will have expertise in GCP, SQL, and Python, contributing to data pipeline design and implementation. You'll work collaboratively to solve complex data challenges and improve service reliability and efficiency.

Qualifications

• 4-6 years of experience in data engineering.
• GCP certified Data Engineer preferred.
• Experience with SQL and Python required.

Responsibilities

• Design and build data pipelines across on-prem and cloud platforms.
• Collaborate with team members to translate business requirements into technical solutions.
• Lead the resolution of operational issues and mentor junior engineers.

Skills

Problem solving

Analytical skills

Team collaboration

Communication skills

Education

Bachelor's degree in Computer Science or related field

Tools

GCP

SQL

Python

Terraform

Apache Airflow

Job description

Role & responsibilities

Data Engineer I

4-6 Years of Experience

Here's the impact you'll make and what well accomplish together

As a member of the Data Engineering team you will utilize DevOps principles to evaluate current and new solutions, discover, evaluate, and provide Data engineering support to EMR platforms. Your contributions will directly impact our ongoing pursuit to enhance our infrastructure and data initiatives to provide real time health alerts of our service to be used by our frontline and support teams to quickly identify problems and in turn help reduce mean time to repair and provide aner action reports which will enhance the understanding of the health status of our services.

Here's how

• Willingness to learn and become an expert in new technologies
• Leading, inspiring and supporting team members to drive high performance and team engagement
• Collaborating and fostering strong working relationships with internal and external stakeholders
• Providing support to our internal customers through the design and implementation of solutions.
• Working closely with the ours and other teams to ensure requirements are understood and delivered
• Helping develop and drive automation opportunities, both on various databases and AWS side
• Continuousl imrovin rocesses and a relentless ursuit of reliabilit and efficienc
• You're the missing piece of the puzzle
• Commended for being a fast learner and resourceful
• Known for effective time management and ability to work independently and with a team
• Research, analytical and problem solving skills
• Recognized for your initiative and owning complex issues through to resolution
• Experience with DB technologies like Oracle, MySQL, MSSQL, MongoDB

Expertise with the internals of a distributed compute engine (Spark/PySpark - Preferred, Presto, or Flink/Beam - also Preferred)

• Some experience in SQL, Stored Procedures, Data Modelling
• Experience with cloud infrastructure (any one of GCP and/or AWS with preference to GCP)
• Some experience with Terraform coding

• Modern Big-Data storage technologies (Iceberg - Preferred, Hudi, Delta)
• Moderate experience with GCP services, specifically DB related, like BigQuery, Cloud Storage, Cloud Dataflow, and Dataproc
• Expertise with Data Analytics
• Strong sense of business ownership and customer focus
• Advanced oral and written English language and communication skills
• Great-to-haves
• Knowledge of and Experience with the following:
• Experience in query optimization, resource allocation and management, and data lake performance (Presto, SQL)
• Jira, Confluence, Jenkins
• Palantir Foundry
• Background and practical experience in statistics and/or computational mathematics (Bayesian and Frequentist approaches, NumPy, PyMC3, etc.)
• Some experience with AWS services, specifically DB related, like DMS, Glue, Athena, Textract • Linux RHEL and CentOS scripting.

Data Engineer II (SAAS Based)

• 4-6 Years of experience
• As a GCP Data Engineer, you'll focus on solving problems and creating value for the business by building solutions that are reliable and scalable to work with the size and scope of the company. You will be tasked with creating custom-built pipelines as well as migrating on-prem data pipelines to the GCP stack. You will be part of a team tackling intricate problems by designing and deploying reliable and scalable solutions tailored to the company's data landscape.
• Required Skills:
• 5+ years of industry experience in software development, data engineering, business intelligence, or related field with experience in manipulating, processing, and extracting value from datasets. • Extensive experience in doing requirement discovery, analysis and data pipeline solution design. • Design, build and deploy internal applications to support our technology life cycle, collaboration and spaces, service delivery management, data and business intelligence among others. • Building Modular code for multi usable pipeline or any kind of complex Ingestion Framework used to ease the job to load the data into Datalake or Data Warehouse from multiple sources. • Work closely with analysts and business process owners to translate business requirements into technical solutions.
• Coding experience in scripting and languages ( Python, SQL, PySpark).

• Expertise in Google Cloud Platform (GCP) technologies in the data warehousing space (BigQuery, GCP Workflows, Cloud Scheduler, Secret Manager, Batch, Cloud Logging, Cloud SDK, Google Cloud Storage, IAM).
• Exposure of Google Dataproc and Dataflow.
• Maintain highest levels of development practices including: technical design, solution development, systems configuration, test documentation/execution, issue identification and resolution, writing clean, modular and self-sustaining code, with repeatable quality and predictability.
• Understanding CI/CD Processes using Pulumi, Github, Cloud Build, Cloud SDK, Docker
• Experience with SAS/SQL Server/SSIS is an added advantage.
• Qualifications:
• Bachelor's degree in Computer Science or related technical field, or equivalent practical experience.
• GCP Certified Data Engineer (preferred)
• Excellent verbal and written communication skills with the ability to effectively advocate technical solutions to other engineering teams and business audiences.
• Willingness to work in the afternoon shift from 3 PM to 12 AM IST shift.

DATA ENGINEER III

4-6 Years of experience

As a Data Engineer, you will design, develop, and support data pipelines and related data products and platforms. Your primary responsibilities include designing and building data extraction, loading, and transformation pipelines across on-prem and cloud platforms. You will perform application impact assessments, requirements reviews, and develop work estimates. Additionally, you will develop test strategies and site reliability engineering measures for data products and solutions, participate in agile development "scrums" and solution reviews, mentor junior Data Engineering Specialists, lead the resolution of critical operations issues, and perform technical data stewardship tasks, including metadata management, security, and privacy by design.

Required Skills:

• Design, develop, and support data pipelines and related data products and platforms. Design and build data extraction, loading, and transformation pipelines and data products across on-prem and cloud platforms.
• Perform application impact assessments, requirements reviews, and develop work estimates. Develop test strategies and site reliability engineering measures

for data products and solutions. Participate in agile development "scrums" and solution reviews.

• Mentor junior Data Engineers.
• Lead the resolution of critical operations issues, including post-implementation reviews. Perform technical data stewardship tasks, including metadata management, security, and privacy by design. Design and build data extraction, loading, and transformation pipelines using Python and other GCP Data Technologies
• Demonstrate SQL and database proficiency in various data engineering tasks.
• Automate data workflows by setting up DAGs in tools like Control-M, Apache Airflow, and Prefect. Develop Unix scripts to support various data operations.
• Model data to support business intelligence and analytics initiatives.
• Utilize infrastructure-as-code tools such as Terraform, Puppet, and Ansible for deployment automation. Expertise in GCP data warehousing technologies, including BigQuery, Cloud SQL, Dataflow, Data Catalog, Cloud Composer, Google Cloud Storage, IAM, Compute Engine, Cloud Data Fusion and Dataproc (good to have).
• Qualifications:
• Bachelor's degree in Software Engineering, Computer Science, Business, Mathematics, or related field. 4+ years of data engineering experience.
• 2 years of data solution architecture and design experience.
• GCP Certified Data Engineer (preferred).

DATA ENGINEER IV

4-6 years of experience

GCP Data Engineer - On-Premises to Cloud SQL Migration

Job Description:

• As a Data Engineer with a focus on migrating on-premises databases to Google Cloud SQL, you will play a critical role in solving complex problems and creating value for our business by ensuring reliable, scalable, and efficient data migration processes. You will be responsible for architecting,designing and implementing custom pipelines on the GCP stack to facilitate seamless migration.
• Required Skills:
• 5+ years of industry experience in data engineering, business intelligence, or a related field with experience in manipulating, processing, and extracting value from datasets.

• Expertise in architecting, designing, building, and deploying internal applications to support technology life cycle management, service delivery management, data, and business intelligence.
• Experience in developing modular code for versatile pipelines or complex ingestion frameworks aimed at loading data into Cloud SQL and managing data migration from multiple on-premises sources. • Strong collaboration with analysts and business process owners to translate business requirements into technical solutions.
• Proficiency in coding with scripting languages (Shell scripting, Python, SQL).
• Deep understanding and hands-on experience with Google Cloud Platform (GCP) technologies, especially in data migration and warehousing, including Database Migration Service (DMS), Cloud SQL, BigQuery, Dataflow, Data Catalog, Cloud Composer, Google Cloud Storage (GCS), IAM, Compute Engine, Cloud Data Fusion, and optionally Dataproc.
• Adherence to best development practices including technical design, solution development, systems configuration, test documentation/execution, issue identification and resolution, and writing clean, modular, self-sustaining code.
• Familiarity with CI/CD processes using GitHub, Cloud Build, and Google Cloud SDK. Qualifications:
• Bachelor's degree in Computer Science or a related technical field, or equivalent practical experience.
• GCP Certified Data Engineer (preferred).
• Excellent verbal and written communication skills with the ability to effectively advocate technical solutions to research scientists, engineering teams, and business audiences.Bachelor's degree in Computer Science or a related technical field, or equivalent practical experience

Preferred candidate profile

Must Have:

- Minimum 4+ years of experience

- Minimum 4+ relevant exp in GCP along with Sql, Python & ETL background

- 0 to 30 days joining period

- Shift : 3 PM to 12 AM