Senior Onsite Data Architect & Engineer (Python, Spark)

IT Minds LLC; Irving (TX); On-site; USD 140,000 - 180,000; Full time

Job summary

IT Minds LLC is seeking a hands-on Data Architect / Data Engineering Consultant for 100% onsite work in Irving, TX. The role focuses on end-to-end data flows, building and troubleshooting data pipelines, and optimizing distributed processing workloads.

Ideal candidates have extensive data architecture and engineering experience, with proficiency in Python, PySpark, Spark optimization, SQL, and production ETL troubleshooting, plus familiarity with Databricks and Snowflake in regulated financial

Qualifications

• 12-15 years in data architecture, data engineering, or enterprise architecture.
• 7+ years of hands-on development experience.
• Experience with Databricks and Snowflake.
• Strong expertise in data warehousing, Lakehouse architecture, ETL/ELT, data modeling, and event-driven integration.
• Experience in regulated financial services.

Responsibilities

• Design, develop, enhance, and troubleshoot ETL/ELT data pipelines.
• Develop and maintain data processing solutions using Python and PySpark.
• Work extensively with Apache Spark, including performance tuning and optimization.
• Analyze Spark jobs to identify performance bottlenecks (partitions, shuffles, joins, data skew, caching, serialization, resources).
• Work with distributed data processing concepts and large-volume datasets.
• Develop complex SQL queries for data transformation, validation, reconciliation, and troubleshooting.
• Perform source-to-target reconciliation and investigate data discrepancies.
• Work with Hive and Impala for querying and processing large datasets.
• Troubleshoot production ETL failures, data quality issues, performance problems, and batch-processing failures.
• Perform root-cause analysis and implement permanent fixes for recurring production issues.
• Understand end-to-end data flows from source systems through ETL to downstream consumers.
• Work with Linux environments, shell commands, batch processing, and job scheduling.
• Collaborate with engineering, application, and business teams to resolve complex data issues.
• Participate in technical design discussions and provide recommendations for scalable data solutions.
• Document technical designs, data flows, troubleshooting procedures, and production resolutions.

Skills

Python

PySpark

Spark optimization

Distributed data processing

SQL

ETL troubleshooting

Tools

Databricks

Snowflake

Hive

Impala

Job description

IT Minds LLC is seeking a hands-on Data Architect / Data Engineering Consultant for 100% onsite work in Irving, TX. The role focuses on end-to-end data flows, building and troubleshooting data pipelines, and optimizing distributed processing workloads.

Ideal candidates have extensive data architecture and engineering experience, with proficiency in Python, PySpark, Spark optimization, SQL, and production ETL troubleshooting, plus familiarity with Databricks and Snowflake in regulated financial