Senior Engineer, Data Engineering

Storyful; Bengaluru; On-site; INR 2,800,000 - 4,200,000; Full time

Job summary

Storyful is seeking a Senior Data Engineer in Bengaluru to develop and maintain scalable ETL/ELT pipelines on Google Cloud, leveraging BigQuery, Cloud Composer and Dataproc. You will collaborate with cross-functional teams to define data warehouse needs and ensure robust data governance and security.

The role requires 4.5+ years of hands-on data engineering experience, strong Python/SQL skills, and experience with CircleCI, Terraform and GitHub for CI/CD and IaC.

Qualifications

• 4.5+ years hands-on data engineering experience.
• Strong Python, SQL, ETL abilities and BigQuery proficiency.
• Experience with CI/CD and IaC tools (CircleCI, Terraform, GitHub).
• Knowledge of data governance, quality, security and compliance.

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines in GCP.
• Collaborate on data architecture and warehouse requirements.
• Automate pipelines using IaC and configuration management tools.
• Deep dive into BigQuery optimization, partitioning and performance.
• Deliver production-quality Python and SQL code.
• Implement CI/CD for data pipelines and apps.
• Stay updated with new data methods and apply where relevant.

Skills

Python

SQL

ETL

BigQuery

Cloud Composer

Airflow

Terraform

CircleCI

Github

Dataproc

Data Governance

Education

Advanced Degree in Computer Science, Data Engineering, Data Analytics

Tools

CircleCI

Terraform

Github

Dataproc

BigQuery

Cloud Composer

Job description

Job Description

We are looking for a Senior Data Engineer with experience in excellent analytical abilities in GCP BigQuery, ETL/BI, Python, Composer/Airfl ow, SQL, GCP Services - with the ability to operate in an agile environment and keen to detail while understanding the larger data warehouse.

Skills & Competencies

• A minimum of 4.5+ years of working experience as a hands-on data engineering is a must
• Advanced Degree in the fi eld of Computer Science, Data Engineering, Data Analytics or equivalent discipline
• Strong programming experience with Python, SQL, ETL and effi ciently querying big datasets using SQL
• Prior experience with CircleCI CI/CD, Terraform and Github version control is a must
• Solid understanding of Data Collection, Data Ingestion, Data Integration, and Data Operations & Maintenance
• Implement data governance practices, including data quality, security, and compliance, in line with organizational standards.
• Domain knowledge on News, Media & Communications is an added advantage.

R E S P O N S I B L I T I E S

• Design, build, and maintain scalable ETL/ELT pipelines in Google Cloud, using tools like Cloud Composer (Airfl ow), Dataproc and BigQuery.
• Collaborate with cross-functional teams to defi ne data architecture and data warehouse requirements
• Automate data pipelines, ETL/ELT processes, and data integration workfl ows using infrastructure-as-code (IaC) and confi guration management tools
• Deep understanding of BigQuery, including SQL, optimization techniques, partitioning, clustering, and performance tuning.
• Deliver production quality, modular, reproducible code in Python, SQL
• Implement continuous integration and continuous deployment (CI/CD) pipelines for data pipelines and data-related applications
• Keep abreast of the latest developments in the fi eld through continuous learning and proactively champion promising new methods relevant to the problems at hand.

Our Tech Stack

• Google Cloud Platform is IaaS
• BigQuery, DataProc, Python, Pyspark, SQL
• Composer / Airfl ow - orchestration tools
• Terraform, CircleCI, CI/CD, GIthub, Jira, Confl uence

Equal Opportunity Employer

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, age, national origin, protected veteran status, disability status or any other protected characteristic.

Reasonable Accommodation

We are committed to providing reasonable accommodation for qualified individuals with disabilities in our job application and/or interview process. If you need assistance or accommodation in completing your application or participating in an interview due to a disability, email us at [e-mail]. Please put "Reasonable Accommodation" in the subject line and provide a brief description of the type of assistance you need. This inbox will not be monitored for application status updates. Please refer to the privacy notice at the bottom of this page for submitting any data access, deletion, or other data subject rights requests, where permitted under your local laws and regulations.

News Corp (Nasdaq: NWS, NWSA; ASX: NWS, NWSLV) is a global, diversified media and information services company focused on creating and distributing authoritative and engaging content and other products and services. The company comprises businesses across a range of media, including: information services and news, digital real estate services and book publishing. Headquartered in New York, News Corp operates primarily in the United States, Australia and the United Kingdom, and its content and other products and services are distributed and consumed worldwide. More information is available at: http://newscorp.com.