Big Data Practice Lead – Spark / Databricks / AWS

Mogi I/O : OTT/Podcast/Short Video Apps for you; United States; Remote; USD 150,000 - 200,000; Full time

Benefits offered by this job

Bonus based on performance

Job summary

A global consulting and technology services organization is looking for a Big Data / Spark Principal Architect to lead the design and implementation of large-scale data systems. This role requires deep expertise in Spark and consulting within the Media & Entertainment domain. The successful candidate will architect data pipelines and guide technical teams. Compensation is between USD 150,000 and 200,000, based on experience and performance. This position is remote and full-time.

Qualifications

• 10–18 years of experience in Big Data Architecture, Engineering, and Consulting.
• Strong hands-on expertise in Apache Spark and Databricks.
• Proven track record in solutioning and consulting as an SME in Big Data.

Responsibilities

• Define and implement scalable big data architectures using Spark and Databricks.
• Lead solution design engagements acting as SME for big data technologies.
• Build and optimize batch and streaming data pipelines leveraging Spark.

Skills

Apache Spark (DataFrame, SQL, PySpark/Scala Spark)

Databricks

AWS Glue

Kafka

ETL design

Data governance

Streaming technologies

Job description

Job Overview

A global consulting and technology services organization is seeking a Big Data / Spark Principal Architect to lead the design and implementation of large-scale data systems. This senior-level role combines deep technical expertise in Spark, Databricks, and cloud-based data platforms with a strategic understanding of solutioning and consulting across enterprise environments. The successful candidate will architect high-performance data pipelines, guide technical teams, and act as a subject matter expert (SME) for big data and streaming solutions within the Media & Entertainment domain.

Key Responsibilities

• Architecture & Design: Define and implement scalable, high-performance big data architectures using Spark (DataFrame, Spark SQL, PySpark/Scala Spark) and Databricks.
• Solutioning & Consulting: Lead solution design engagements, act as SME for big data technologies, and provide consulting expertise for enterprise data strategies.
• Pipeline Development: Build and optimize batch and streaming data pipelines leveraging Spark, AWS Glue, and Kafka (or equivalent streaming tools).
• Data Governance & Quality: Establish frameworks for data quality, lineage, and compliance across multi-source data ecosystems.
• Cloud Enablement: Architect and deploy solutions on AWS or Azure, ensuring performance, scalability, and cost efficiency.
• Practice Development: Contribute to the growth of the Big Data practice through mentoring, pre-sales support, and best practice documentation.
• Collaboration: Work closely with data engineers, analysts, and business stakeholders to translate requirements into robust technical solutions.
• Continuous Improvement: Stay current with emerging technologies and promote adoption of modern data engineering practices.

Must-Have Requirements

• 10–18 years of total experience in Big Data Architecture, Engineering, and Consulting.
• Strong hands-on expertise in Apache Spark (DataFrame, SQL, PySpark/Scala Spark) and Databricks.
• Proven track record in solutioning and consulting as a Subject Matter Expert (SME) in Big Data.
• Experience with AWS Glue or Azure Data Services for data ingestion and transformation.
• Familiarity with streaming technologies such as Kafka, Kinesis, or Flink.
• Deep understanding of ETL design, data governance, and data quality frameworks.
• Strong coding skills and willingness to be hands-on with development.
• Prior experience in Media & Entertainment or related domains.
• Must be a U.S. Citizen or Green Card Holder (mandatory).
• Demonstrated job stability and long-term project ownership.

Nice-to-Have Skills

• Exposure to AI/ML integration within big data ecosystems.
• Experience leading practice development or solution selling for enterprise clients.
• Knowledge of modern data mesh or lakehouse architectures.

Location: USA – Remote (Full-Time)

Work Type: Permanent Remote

Experience Required: 10 – 18 Years

Compensation: USD 150,000 – 200,000 (plus bonus, based on experience and performance)