Data Engineer (Mid-Level) US

Cornerstone Global Partners (CGP Group); United States; Remote; USD 120,000 - 170,000; Full time

Benefits offered by this job

Competitive salary

Medical insurance

401(k) with company match

Generous PTO

Company-paid holidays

Remote work

Job summary

Cornerstone Global Partners (CGP Group) is seeking a Data Engineer to build AI-enabled data ingestion pipelines for a new threat and risk management platform. You will implement batch and streaming ingestion on Databricks, apply medallion architecture, and collaborate with Data Scientists to productionize extraction pipelines.

US-based team, remote work, and overlapping hours with India team are required. This role emphasizes CI/CD, governance, and scalable data pipelines across GIS, SCADA, and

Qualifications

• Hands-on experience building production pipelines on Databricks including Spark/PySpark.
• Hands-on experience with Delta Lake, medallion architecture, and lakehouse engineering.
• Strong SQL skills.
• 3-5 years of experience in data engineering, ETL development, or cloud data platform engineering.
• Hands-on CI/CD experience for data pipelines.
• Experience using AI-assisted coding tools in development work.
• Git-based development in a code-reviewed team.

Responsibilities

• Build and maintain ingestion pipelines for structured and semi-structured sources (GIS, SCADA, maintenance systems).
• Implement batch and streaming ingestion using Databricks Workflows, Spark, PySpark, SQL.
• Apply medallion architecture for transformation, standardization, and enrichment.
• Implement CDC, SCD, schema evolution, and data-validation rules.
• Normalize third-party data feeds into the shared data model.
• Automate normalization of units, schemas, and semantics across data.
• Work with Data Scientists to productionize document-extraction pipelines and review low-confidence extractions.
• Configure and manage Delta Lake tables, partitioning, and Unity Catalog."
• Build connectors to customer systems of record and support geospatial processing."
• Implement data-quality tests, profiling, drift monitoring, and lineage for regulatory traceability.
• Build, schedule, and monitor data workflows; own alerting and failure handling.
• Implement CI/CD for pipeline code with version control and tests.
• Troubleshoot production incidents and optimize performance and cost.
• Participate in architecture reviews and document pipelines and runbooks.

Skills

Databricks

Spark/PySpark

Delta Lake

SQL

Python

CI/CD

Data modeling

Data governance

Git

Tools

Databricks

SQL

Job description

About Our Client

Our client is a leading SaaS company with nearly three decades of experience helping energy, utility, telecom, and infrastructure companies protect their critical network infrastructure. They are now expanding their platform with new data-driven and AI-powered capabilities.

About Our Client

Our client is a leading SaaS company with nearly three decades of experience helping energy, utility, telecom, and infrastructure companies protect their critical network infrastructure. They are now expanding their platform with new data-driven and AI-powered capabilities.

About The Role

Our client is building a new AI-driven threat and risk management platform for pipeline asset integrity. It brings together a governed, cross-product data platform built on Databricks and Azure, an AI-powered ingestion layer that normalizes, repairs, and enriches customer data, and a reusable analytical layer that runs risk models against that data. As a Data Engineer, you will build the pipelines that make this platform real. This is a new project, so you will help build it from the ground up, implementing patterns set by the Data Architect and working closely with Data Scientists on AI-assisted data ingestion. The team is US-based and works daily with colleagues in India.

Location: Remote (United States)

Schedule: Monday-Friday, with working hours that overlap with the India team, roughly 8:00-11:00 AM Eastern Time.

Employment Type: Full-time

Key Responsibilities

• Build and maintain ingestion pipelines for structured and semi-structured sources, including GIS, inline inspection data, SCADA, maintenance systems, and enterprise systems of record.
• Implement batch and streaming ingestion using Databricks Workflows, Spark, PySpark, SQL, and declarative pipeline tooling.
• Apply medallion architecture patterns (Bronze, Silver, Gold) for transformation, standardization, and enrichment.
• Implement change data capture (CDC), slowly changing dimensions (SCD), schema evolution, and data-validation rules.
• Normalize third-party and public data feeds (weather, soil, satellite-derived, and one-call ticket data) into the shared data model.
• Build pipelines that automate normalization of units, schemas, and semantics across inconsistent customer data, including AI-assisted mapping of new customer schemas to the target schema.
• Implement automated data-quality repair workflows, with clear provenance for every synthesized value.
• Work with Data Scientists to productionize document-extraction pipelines, and build human-in-the-loop review workflows for low-confidence extractions.
• Configure and manage Delta Lake tables, partitioning, and optimization routines, and implement lineage and cataloging with Unity Catalog.
• Build connectors to customer systems of record and support geospatial processing, including spatial joins.
• Implement data-quality tests, profiling, drift monitoring, access-control policies, and lineage for regulatory traceability.
• Build, schedule, and monitor data workflows, and own alerting and failure handling for your pipelines.
• Implement CI/CD for pipeline code, including version control, automated testing, and environment promotion.
• Troubleshoot production incidents, recover failed runs, and optimize performance and cost.
• Participate in architecture, design, and code reviews, and document pipelines, data dictionaries, and runbooks.

Requirements

• Hands‑on experience building production pipelines on Databricks, including Spark/PySpark.
• Hands‑on experience with Delta Lake, medallion architecture, and lakehouse engineering.
• Strong SQL skills.
• 3-5 years of experience in data engineering, ETL development, or cloud data platform engineering.
• Hands‑on CI/CD experience for data pipelines.
• Experience using AI-assisted coding tools (e.g. Cursor, GitHub Copilot, Claude Code) in your development work.
• Git‑based development in a code‑reviewed team (GitHub, GitLab, or Bitbucket).
• Experience with at least one major cloud platform (Azure preferred).
• Familiarity with data modeling, data quality, schema evolution, and pipeline troubleshooting.
• Experience with workflow orchestration and scheduling frameworks.
• Understanding of core data‑security practices, including access control, encryption, and credential management.
• Availability to work hours that overlap with 8:00-11:00 AM Eastern Time.

Nice-to-Have

• Microsoft Azure experience.
• Unity Catalog, Microsoft Purview, or comparable metadata and lineage tooling.
• Building pipelines that ingest unstructured or semi-structured documents.
• Geospatial data processing and common GIS data formats.
• Infrastructure as code (IaC) for data workloads.
• Preparing data for machine learning or probabilistic model consumption.
• Cloud or Databricks certifications.
• Experience with oil and gas or utility asset data, asset integrity concepts, or related regulatory reporting.
• Migrating customers from legacy or spreadsheet-based systems to modern data platforms.

Benefits

• Competitive salary
• Medical, dental, and vision insurance
• 401(k) plan with company match
• Generous paid time off (PTO)
• Company-paid holidays
• Remote work