Senior Engineer, Data Engineering

News Corp; Bengaluru; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

News Corp in Bengaluru is seeking a Lead Data Engineer to unify data across categories, countries and organizations into a single customer view to enhance engagement, measurement and targeting.

You will design and maintain high-performance data pipelines, collaborate with product and engineering teams, and mentor a team to continuously improve data processing capabilities and data quality.

Qualifications

• BS in Computer Science or similar technical field.
• 8+ years designing and developing big data processing systems.

Responsibilities

• Design, implement and maintain high-performance big data infrastructure and pipelines.
• Mentor and guide a team of data engineers to improve technical skills.
• Ensure data quality, security, and governance across platforms.

Skills

AWS

Python

Spark

Kubernetes

Docker

SQL

NoSQL

Education

BS in Computer Science

Tools

Git

Jira

Airow

DynamoDB

Job description

Job Description

Lead Data Engineer One of the most innovative and high-prole teams in News Corp is looking for a lead data engineer to help accelerate its vision. The role will involve combining all of News Corp’s data across categories, countries and organizations into a singular view of a customer enabling engagement, measurement and targeting. The Dow Jones Lead Data Engineer will partner closely with product, data & engineering teams to help make this data available and accessible to our businesses, teams and partners in order to drive revenue and improve the customer experience.

Responsibilities

• Design, implement and maintain high performance big data infrastructure/systems & big data processing pipelines scaling to billions of structured and unstructured events daily
• Design, implement and maintain deep integration with up-stream systems
• Implement data pipelines to automate the ingestion, transformation, and augmentation of data sources, and provide best practices for pipeline operations.
• Advanced SQL writing and experience in data mining (SQL, ETL, data warehouse, etc.) and using databases in a business environment with complex datasets
• Monitor and optimize the performance of the data platform to ensure reliability, scalability, and efiency.
• Ensure data quality across systems (accuracy, consistency, completeness) by providing transparency and proactive monitoring.
• Support products with the overall roadmap and ensure updates to senior leadership are 100% technically correct.
• Data analysis, understanding of business requirements and translation into logical pipelines & processes
• Conduct timely and effective research in response to specific requests (e.g. data collection, summarization, analysis, and synthesis of relevant data and information)
• Evaluate and prototype new technologies in the area of data processing
• Think quickly, communicate clearly and work collaboratively with product, data, engineering, QA and operations teams
• Data Engineering & Data Modelling: Creating data architectures and logical/physical data models for databases and data warehouses.
• Providing training on data engineering concepts and techniques to other members of the team
• Writing documentation for data pipelines and data models
• Mentor and guide a team of data engineers, fostering a culture of continuous improvement, innovation, and knowledge sharing to enhance technical expertise and collaboration across the team.
• Promote a high-performance team environment, ensuring strong work ethic, accountability, and enthusiasm to drive successful project delivery.

Technologies we use

• AWS (EMR, EC2, Glue, Athena, Redshift, Lambda, S3, DynamoDB, SNS)
• Python, Spark (PySpark)
• Kubernetes, Docker
• Airow
• Git for source code management
• Jira

Qualifications

• BS in Computer Science or other technical discipline
• 8+ years of experience designing and developing big data processing systems using distributed computing
• 6+ years experience in SQL
• 5+ years experience in Python and Spark
• 6+ years of experience working with cloud platforms such as AWS
• 4+ years experience in Apache Airow
• Expert knowledge in optimizing complex SQL queries and Hands on experience with NoSQL (DynamoDB)
• An affinity for automation
• Familiarity with networking and network application programming, including HTTP/HTTPS, JSON, and REST APIs
• Experience with at least one object oriented language (ex: Java)
• Strong OO design, data structure, and algorithm design skills
• Strong interest in emerging technologies: Spark, EKS

Nice to Have

• Experience in the digital advertising and marketing industry
• Contribution to Open Source projects

Equal Opportunity Employer

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, age, national origin, protected veteran status, disability status or any other protected characteristic.

Reasonable Accommodation

We are committed to providing reasonable accommodation for qualified individuals with disabilities in our job application and/or interview process. If you need assistance or accommodation in completing your application or participating in an interview due to a disability, email us at [e-mail]. Please put \"Reasonable Accommodation\" in the subject line and provide a brief description of the type of assistance you need. This inbox will not be monitored for application status updates.

Please refer to the privacy notice at the bottom of this page for submitting any data access, deletion, or other data subject rights requests, where permitted under your local laws and regulations.

This job is posted with NTS Technology Services Pvt. Ltd.