PySpark Data Engineer - Build Scalable Pipelines

LTM

Tampa (FL)

On-site

USD 110,000 - 140,000

Full time

14 days+

Get more replies from employers

Send a job-specific resume in minutes.

Benefits offered by this job

Medical plan

Disability coverage

401(k) with company match

Life insurance

Parental leave

Paid vacation and holidays

Job summary

LTIMindtree in Tampa, FL is seeking a PySpark Developer to design, build and optimize large-scale data pipelines using PySpark and Spark SQL. You will work with Azure Databricks, ADF, Synapse and Hadoop ecosystems, integrate with DataOps practices, and collaborate with data engineers, analysts and DevOps in an Agile environment.

Strong SQL, Linux, ETL fundamentals and Python scripting are required, plus a banking analytics background is a plus.

Qualifications

Hands-on PySpark and Apache Spark experience.
Strong SQL and relational database knowledge.
Experience in Linux/Unix environments.
ETL and large-scale data processing.
Exposure to Azure Databricks, ADF, Synapse or similar.
Hadoop ecosystem (Hive, HDFS).
Airflow orchestration knowledge.
Python scripting beyond Spark usage.
Agile and DevOps delivery practices.
Banking/BFSI analytics exposure is a plus.
Strong problem-solving and ownership mindset.
Cross-functional collaboration and communication skills.

Responsibilities

Design, develop and optimize data pipelines using PySpark for large-scale processing.
Transform, validate and aggregate data with Spark SQL and DataFrames.
Write and optimize SQL queries for data extraction and reporting.
Perform data quality checks and profiling.
Troubleshoot production pipelines and perform root-cause analysis.
Collaborate with data engineers, analysts, QA and DevOps in Agile teams.
Enhance Spark job performance for scalability and efficiency.
Maintain documentation, runbooks and technical workflows.
Adhere to enterprise coding, deployment and change-management standards.
Self-direct learning of new big-data technologies and cloud platforms.