Senior Data Engineer: Spark & Python on Azure Cloud

Socket.dev; Poland; Hybrid; PLN 180,000 - 280,000; Full time

Benefits offered by this job

International projects

Hybrid or remote flexibility

Medical healthcare

Well-being program

Learning reimbursement

Team events

Top-tier equipment

Job summary

Exadel invites a Senior/Lead Data Engineer to design and implement data pipelines using PySpark on Databricks and Spark SQL. You will optimize performance, manage Delta tables, and collaborate with cross-functional teams to deliver scalable AI-driven solutions for a global client portfolio.

You will also establish coding standards, review code, and communicate with stakeholders to ensure robust, maintainable software. Strong Python skills and cloud familiarity are essential.

Qualifications

• Hands-on software development experience.
• Experience migrating SQL to Spark SQL or PySpark environments.
• Extensive PySpark on Databricks experience (Delta tables, cluster management, and workflow automation).
• Proven track record of Spark performance optimization in production."
• Strong Python skills for data manipulation with Polars or Pandas.
• Solid understanding of Parquet format and Delta Tables."],
• job_responsibilities_description':['Define and enforce best practices and coding standards across the project','Conduct thorough code reviews to ensure adherence to guidelines and maintain high code quality','Work both independently and in collaboration with teammates','Communicate clear instructions to team members and help manage day-to-day operations','Engage with the client regularly to align on deliverables','Design, develop, and maintain robust Spark applications','Write clean, maintainable code following best practices','Optimize code for performance and scalability with efficient data handling','Collaborate with cross-functional teams to deliver high-quality software solutions','Identify and resolve technical issues ensuring reliability of applications','Create and maintain comprehensive documentation for code, processes, and workflows'],
• CoT_job_summary_short":["COMPANY NAME: Exadel","KEY POINTS: Data engineering, PySpark, Spark performance, Databricks"],
• job_summary_short":
• contract_type
• fulltime
• location_type
• remote
• remote_scope":null,
• perks":[{"name":"International projects"},{"name":"Hybrid or remote flexibility"},{"name":"Medical healthcare"},{"name":"Well-being program"},{"name":"Learning reimbursement"},{"name":"Team events"},{"name":"Top-tier equipment"}],
• stated_min":null,
• stated_max":null,
• stated_absolute":null,
• salary_estimation_cot":null,
• estimated_low":180000,
• estimated_medium":230000,
• estimated_high":280000,
• currency":"PLN","frequency":"yearly","bonus":null,
• tax":"gross","application_email":null,
• job_end_date":null,
• contact_person":null,
• hiring_department_name":null}]}} /* Note: This field is a placeholder to indicate end of JSON object. */} } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } }
• application_email_missing_reason :=

Responsibilities

• Define and enforce best practices and coding standards across the project
• Conduct thorough code reviews to ensure adherence to established guidelines and maintain high code quality
• Working both independently and in close collaboration with others in the team
• Communicating clear instructions to team members and helping manage the flow of day-to-day operations
• Communicating with the client regularly
• Design, develop, and maintain robust and scalable Spark applications
• Write clean, maintainable, and efficient code following best practices and coding standards
• Optimize code for performance and scalability, ensuring efficient data handling
• Work closely with cross-functional teams to deliver high-quality software solutions
• Identify and resolve technical issues, ensuring the reliability and performance of applications
• Create and maintain comprehensive documentation for code, processes, and workflows

Skills

PySpark

Spark

Python

Databricks

SQL

Delta Tables

Parquet

Microservices

Azure Cloud

Docker

Kubernetes

FastAPI

Tools

Databricks

Spark SQL

Delta Lake

Docker

Kubernetes

Job description

Exadel invites a Senior/Lead Data Engineer to design and implement data pipelines using PySpark on Databricks and Spark SQL. You will optimize performance, manage Delta tables, and collaborate with cross-functional teams to deliver scalable AI-driven solutions for a global client portfolio.

You will also establish coding standards, review code, and communicate with stakeholders to ensure robust, maintainable software. Strong Python skills and cloud familiarity are essential.