Data Architect / Data Engineering Consultant

Zodiac Solutions, Inc; Irving (TX); On-site; USD 150,000 - 230,000; Full time

Job summary

Zodiac Solutions, Inc. is seeking a senior Data Architect / Data Engineering Consultant in Irving, TX to lead end-to-end data pipelines and platform optimization.

You will design, implement, and troubleshoot distributed data processing using Python, PySpark, Spark, Hive/Impala, and SQL, ensuring data quality and regulatory compliance across financial services data environments. The role emphasizes hands-on development, performance tuning, and collaboration with engineering and business teams to

Qualifications

• 12–15 years in data architecture, data engineering, or enterprise architecture.
• Experience with Databricks and Snowflake.
• Strong expertise in data warehousing, Lakehouse architecture, ETL/ELT, data modeling, and event-driven integration.
• Experience in regulated financial services.

Responsibilities

• Design, develop, enhance, and troubleshoot ETL/ELT data pipelines.
• Develop and maintain data processing solutions using Python and PySpark.
• Work extensively with Apache Spark, including performance tuning and optimization.
• Analyze Spark jobs to identify performance bottlenecks related to partitions, shuffles, joins, data skew, caching, serialization, and resource utilization.
• Work with distributed data processing concepts and large-volume datasets.
• Develop complex SQL queries for data transformation, validation, reconciliation, and troubleshooting.
• Perform source-to-target reconciliation and investigate data discrepancies.
• Work with Hive and Impala for querying and processing large datasets.
• Troubleshoot production ETL failures, data quality issues, performance problems, and batch-processing failures.
• Perform root-cause analysis and implement permanent fixes for recurring production issues.
• Understand and troubleshoot end-to-end data flows, from source systems through ETL processing to downstream consumers.
• Work with Linux environments, shell commands, batch processing, and job scheduling.
• Collaborate with engineering, application, and business teams to resolve complex data issues.
• Participate in technical design discussions and provide recommendations for scalable and maintainable data solutions.
• Document technical designs, data flows, troubleshooting procedures, and production resolutions.

Skills

Python

PySpark

Spark optimization

SQL

ETL/ELT

Data modeling

Data warehousing

Event-driven integration

Distributed data processing

End-to-end data flows

Production troubleshooting

Linux commands

Data reconciliation

Tools

Databricks

Snowflake

Hive

Impala

Apache Spark

Shell scripting

Job description

Location- Irving, TX.

Required Qualifications

• 12-15 years in data architecture, data engineering, or enterprise architecture.
• Experience with Databricks and Snowflake.
• Strong expertise in data warehousing, Lakehouse architecture, ETL/ELT, data modeling, and event-driven integration.
• Experience in regulated financial services.

And/or-

We are looking for a hands-on Data Architect / Data Engineering Consultant with strong expertise in Python, PySpark, Spark optimization, distributed data processing, Hive/Impala, SQL, and production ETL troubleshooting.

The consultant will be responsible for understanding end-to-end data flows, developing and troubleshooting data pipelines, optimizing distributed processing workloads, performing SQL-based data reconciliation, and supporting production data platforms.

Key Responsibilities

• Design, develop, enhance, and troubleshoot ETL/ELT data pipelines.
• Develop and maintain data processing solutions using Python and PySpark.
• Work extensively with Apache Spark, including performance tuning and optimization.
• Analyze Spark jobs to identify performance bottlenecks related to partitions, shuffles, joins, data skew, caching, serialization, and resource utilization.
• Work with distributed data processing concepts and large-volume datasets.
• Develop complex SQL queries for data transformation, validation, reconciliation, and troubleshooting.
• Perform source-to-target reconciliation and investigate data discrepancies.
• Work with Hive and Impala for querying and processing large datasets.
• Troubleshoot production ETL failures, data quality issues, performance problems, and batch-processing failures.
• Perform root-cause analysis and implement permanent fixes for recurring production issues.
• Understand and troubleshoot end-to-end data flows, from source systems through ETL processing to downstream consumers.
• Work with Linux environments, shell commands, batch processing, and job scheduling.
• Collaborate with engineering, application, and business teams to resolve complex data issues.
• Participate in technical design discussions and provide recommendations for scalable and maintainable data solutions.
• Document technical designs, data flows, troubleshooting procedures, and production resolutions.