Sr. Data Engineer – Data & Intelligence

Sophus IT Solutions; Frisco (TX); Presencial; USD 140.000 - 190.000; Jornada completa

Descripción de la vacante

Sophus IT Solutions in Frisco, TX seeks a Senior/Lead Data Engineer to architect and lead enterprise data pipelines for finance and revenue data using Snowflake, Databricks, Python/PySpark and advanced SQL. You will define standards for batch, streaming, and CDC ingestion, and ensure scalable, fault-tolerant designs on Azure.

You will provide technical leadership across multiple teams, guide data modeling and governance, and implement CI/CD and observability.

Formación

• Experience architecting enterprise-scale ELT/ETL pipelines for finance and revenue data.

Responsabilidades

• Architect, design, and oversee enterprise-scale ELT/ETL pipelines for finance and revenue data (billing, revenue, GL, opex).
• Define and enforce standards for batch, incremental, and streaming ingestion patterns (CDC, watermarking, event-driven ingestion).
• Ensure idempotent, fault-tolerant, and highly scalable pipeline design across platforms.
• Establish frameworks for error handling, retry strategies, dead-letter queue patterns, and operational resiliency.
• Provide technical leadership for multi-source, high-volume data integration pipelines.
• Lead architecture and adoption of Snowflake and Databricks platforms for large-scale data processing and analytics.

Conocimientos

Snowflake

Databricks

PySpark

Python

Advanced SQL

Azure

CI/CD

Airflow

Azure Data Factory

Terraform

Bicep

Great Expectations

dbt

Kafka

Herramientas

Airflow

Azure Data Factory

Terraform

Bicep

Great Expectations

dbt

Kafka

Descripción del empleo

Location : Frisco, TX ( onsite)

Must have

Must-have skills: Snowflake, Databricks, PySpark, Python, advanced SQL, Azure, and CI/CD.

Nice to have

Exp with telecom/finance/revenue/billing data

Summary

Senior Data Engineer / Technical Lead who can architect and lead enterprise finance-data pipelines using Snowflake + Databricks + Python/PySpark + SQL, with strong Azure/cloud, streaming, CI/CD, data quality, and finance/revenue data experience.

Additional skills

Snowflake + Databricks + Python/PySpark + SQL, with strong Azure/cloud, CI/CD

Comments

Looking for -

Senior/Lead Data Engineer

Senior/Lead Data Engineer with Snowflake + Databricks + PySpark + Azure, who has built high-volume enterprise data pipelines and preferably worked with telecom/finance/revenue/billing data

DATA PIPELINE DEVELOPMENT

• Architect, design, and oversee development of enterprise-scale ELT/ETL pipelines for finance and revenue data (billing, revenue, GL, opex).
• Define and enforce standards for batch, incremental, and streaming ingestion patterns (CDC, watermarking, event-driven ingestion).
• Ensure idempotent, fault-tolerant, and highly scalable pipeline design across platforms.
• Establish frameworks for error handling, retry strategies, dead-letter queue patterns, and operational resiliency.
• Provide technical leadership for multi-source, high-volume data integration pipelines.

PLATFORM & TOOLING

• Lead architecture and adoption of Snowflake and Databricks platforms for large-scale data processing and analytics.
• Snowflake (Snowpipe, streams, tasks, query optimization, cost efficiency)
• Databricks (PySpark, Delta Live Tables, Unity Catalog, job optimization)
• Establish and govern orchestration frameworks using Airflow / Azure Data Factory, including DAG standards, dependency design, and monitoring.
• Evaluate and drive tooling strategy and platform standardization across teams.

CLOUD INFRASTRUCTURE

• Architect and optimize cloud-native data platforms on Azure (ADLS Gen2, Event Hub, ADF, Key Vault) or AWS equivalents.
• Define standards for infrastructure-as-code (Terraform, Bicep) and environment provisioning.
• Drive cost optimization strategies (compute sizing, storage design, partitioning, workload isolation).
• Ensure platforms are scalable, secure, and production-ready.

LANGUAGES & FRAMEWORKS

• Advanced SQL (query tuning, execution optimization, complex transformations)
• Guide teams on best practices, reusable frameworks, and performance optimization.
• Oversee development standards for Spark, Scala (where applicable), and automation scripting.
• Architect real-time and near real-time data processing solutions using Kafka / Event Hub and Spark Structured Streaming.
• Define patterns for stateful processing, watermarking, checkpointing, and fault tolerance.
• Lead implementation of real-time finance/revenue use cases such as reconciliation, anomaly detection signals, and operational reporting.

DATA QUALITY & TESTING

• Establish enterprise frameworks for data quality, validation, and observability.
• Automated testing (unit, integration, regression)
• Data quality tools (dbt tests, Great Expectations, custom frameworks)
• Ensure SLA monitoring, alerting, and data freshness tracking across all pipelines.
• Drive proactive data quality and governance practices across teams.

DATA MODELING SUPPORT

• Interpret and implement architect-defined enterprise data models (star, snowflake, data vault).
• SCD (Type 1/2) strategies
• Partitioning, clustering, and performance optimization
• Collaborate with architects to evolve scalable and reusable data models.
• Support semantic layer enablement for analytics and reporting.

DEVOPS & ENGINEERING PRACTICES

• Define and enforce CI/CD standards for data engineering (GitHub Actions, Azure DevOps).
• Establish code quality, versioning, and deployment best practices (branching strategies, PR reviews, release pipelines).
• Drive adoption of engineering excellence practices including reusable frameworks and templates.

SECURITY & GOVERNANCE

• Lead implementation of enterprise-grade security and governance controls:
• PII and CPNI compliance (TISS-310)
• Define standards for secrets management and secure pipeline design.
• Ensure data lineage, auditability, and compliance readiness across platforms.

FINANCE DOMAIN KNOWLEDGE

• Deep understanding of finance and revenue data domains, including:
• Billing and revenue systems
• GL structures and financial reporting
• Revenue recognition and reconciliation
• Period-end close cycles
• Guide engineering teams on accurate implementation of finance logic.
• Ensure high data integrity standards for regulated financial data.

SOFT SKILLS & COLLABORATION

• Act as a technical leader and escalation point across engineering teams.
• Partner with architects, product managers, analysts, and business stakeholders.
• Drive cross-team alignment and solution consistency.
• Communicate complex technical topics clearly to both technical and non-technical audiences.
• Lead incident reviews and ensure continuous improvement.