Senior PySpark Data Architect - Cloud ETL & Big Data

LTM; Tampa (FL); Sur place; USD 120 000 - 180 000; Plein temps

Avantages offerts par ce poste

Comprehensive Medical Plan

401(k) Plan with Company match

Life Insurance

Vacation Time, Sick Leave, Paid Holidá

Résumé du poste

LTIMindtree is seeking a Data Architect with strong PySpark skills to design, develop and optimize large-scale data processing solutions for analytics and data engineering initiatives. You will build scalable data pipelines, ETL/ELT processes, and collaborate with data scientists, analysts and other engineers while leveraging AWS, Azure, or GCP and Databricks platforms.

The role requires Databricks certification and hands-on experience with Spark, SQL, Hadoop, and modern data warehousing

Qualifications

• Strong experience in Python and PySpark development.
• Hands-on experience with Apache Spark, Spark SQL, DataFrames, and Spark Streaming.
• Strong knowledge of SQL and relational databases.
• Experience with ETL development and data warehousing concepts.
• Familiarity with Hadoop ecosystem technologies (HDFS, Hive, Yarn).
• Experience with version control systems such as Git.
• Understanding of distributed computing concepts and performance tuning.
• Knowledge of workflow orchestration tools such as Airflow, Oozie, or Azure Data Factory.
• Strong analytical and problem-solving skills.

Responsabilités

• Design, develop, and maintain scalable data pipelines using PySpark.
• Build and optimize ETL/ELT processes for structured and unstructured data.
• Process large datasets using Apache Spark in distributed environments.
• Collaborate with analysts and data scientists to understand data requirements.
• Integrate data from databases, APIs, and cloud storage.
• Develop data quality checks and monitoring mechanisms.
• Work with cloud platforms such as AWS, Azure, or GCP for data engineering.
• Participate in code reviews and adhere to development standards.

Connaissances

Python

PySpark

Spark SQL

DataFrames

Spark Streaming

SQL

ETL

Data Warehousing

Hadoop

Git

Airflow

Azure Data Factory

Databricks

AWS

GCP

Delta Lake

Hive

Formation

Bachelor's or Master’s in Computer Science/IT/Engineering

Outils

Databricks

Azure Data Factory

Azure Synapse

AWS Glue

Kafka

Airflow

Oozie

Hadoop

Hive

Git

Description du poste

LTIMindtree is seeking a Data Architect with strong PySpark skills to design, develop and optimize large-scale data processing solutions for analytics and data engineering initiatives. You will build scalable data pipelines, ETL/ELT processes, and collaborate with data scientists, analysts and other engineers while leveraging AWS, Azure, or GCP and Databricks platforms.

The role requires Databricks certification and hands-on experience with Spark, SQL, Hadoop, and modern data warehousing