Senior Data Engineer

Eliassen Group; Chicago (IL); Hybrid; Confidential; Full time

Benefits offered by this job

Medical, Dental, Vision

401k with company matching

Life insurance

Job summary

Eliassen Group in Chicago, IL is seeking a Senior Data Engineer to design, build, and operate large-scale data processing for attribution, measurement, forecasting, and privacy-preserving analytics. You will develop Scala and Spark solutions on cloud platforms and partner with cross-functional teams to deliver secure, reliable data products.

Ideal candidates have strong experience with Scala, Apache Spark, Python, and modern data warehouses, plus a proven track record in governance and privacy

Qualifications

• 5+ years data engineering with Scala and Spark on cloud platforms.
• Proficient in Python for pipelines and automation.
• Experience with data privacy controls and governance.

Responsibilities

• Develop and optimize large-scale data pipelines with Spark/Scala on cloud.
• Build trusted data pipelines across AWS/GCP/Azure.
• Collaborate across teams on privacy-preserving features.
• Design batch and streaming workflows with Airflow or equivalent.
• Mentor junior engineers and conduct code reviews.
• Ensure security, governance, and data lineage in data products.

Skills

Scala

Apache Spark

Python

SQL

Cloud AWS/GCP

Airflow

Databricks

Security & Privacy

Tools

Databricks

Delta Lake

Git

Kubernetes

Job description

Description:

Hybrid At least 2 days per week in office in Chicago, IL

Our client seeks a Senior Data Engineer to design, build, and operate large-scale data processing for attribution, measurement, forecasting, and privacy-preserving analytics. You will develop Scala and Spark solutions on cloud platforms, implement governance and privacy controls, and partner with cross-functional teams to deliver secure, reliable data products.

We can facilitate w2 and corp-to-corp consultants. For our w2 consultants, we offer a great benefits package that includes Medical, Dental, and Vision benefits, 401k with company matching, and life insurance.

Rate: $70.00 to $80.00/hr. w2

Responsibilities:

• Develop and optimize large-scale data processing solutions using Scala, Spark, and SQL on modern data platforms.
• Build and operate trusted data pipelines across secure cloud environments such as AWS, GCP, or Azure.
• Partner with Product, Data Science, Security, Privacy, and Platform Engineering to deliver privacy-preserving features.
• Build, schedule, and maintain scalable batch and streaming data workflows with orchestration frameworks.
• Implement data classification, access controls, and privacy-preserving techniques aligned to compliance requirements.
• Contribute to clean-room and trusted data-sharing environments with approved aggregated outputs.
• Create observability, monitoring, and operational tooling for reliability and compliance.
• Troubleshoot complex performance and pipeline issues across distributed systems.
• Contribute to technical design, best practices, and operational excellence.
• Mentor junior engineers and perform thorough code reviews.
• Continuously improve attribution, measurement, forecasting, and privacy-preserving analytics capabilities.
• Operate pipelines within trusted environments, clean rooms, or secure data-sharing platforms across cloud and on-premises.
• Apply access controls, data classification, lineage, and governance for PII, PCI, and confidential signals.
• Follow data handling standards with Security, Privacy, and Compliance teams to keep sensitive data within trust boundaries.
• Enforce aggregation, anonymization, tokenization, and approved outputs for data leaving trusted environments.
• Build monitoring and alerting to detect anomalous data movement and policy violations.
• Apply privacy-preserving computation when outputs cross trust boundaries, including aggregation-before-export, pseudonymization, tokenization, differential privacy concepts, and privacy-aware reporting.
• Implement encryption, key management, and secure handling with cloud-native security services.
• Document trust boundaries, data contracts, lineage, and permitted data movement.
• Support audits, compliance requirements, governance reviews, and secure data-sharing initiatives.
• Participate in architecture and design reviews to embed governance, privacy, lineage, and trust-boundary requirements.
• Contribute to engineering standards for secure data processing and trusted platform operations.

Experience Requirements:

• 5+ years of data engineering with strong Scala and Apache Spark on AWS and/or GCP.
• Strong Python for pipelines, tooling, automation, and infrastructure modules.
• Advanced SQL across RDBMS, cloud data warehouses, and lakehouse platforms with TB-scale datasets.
• Designing and maintaining batch and streaming data pipelines.
• Data warehousing, dimensional modeling, data quality, partitioning, and performance optimization.
• Distributed processing and modern lakehouse architectures such as Databricks, Delta Lake, or Apache Spark.
• Operating distributed data platforms at scale.
• Workflow orchestration with Airflow, Databricks Workflows, AWS Step Functions, or equivalent.
• Source control with Git and test automation frameworks.
• Cloud-native development on AWS and/or GCP.
• Software engineering practices including CI/CD, code reviews, observability, and production support.
• Ownership of features and pipelines with cross-team collaboration and mentoring.
• Trusted environment execution with clean rooms or secure data-sharing platforms handling PII and regulated data.
• Fine‑grained access controls, governance policies, and policy‑based enforcement for sensitive datasets.
• Privacy‑preserving techniques such as tokenization, pseudonymization, aggregation‑before‑export, and differential privacy concepts.
• Experience with clean‑room, measurement, attribution, audience analytics, or privacy‑preserving reporting solutions.
• Understanding of trust boundaries, secure data‑sharing patterns, and zero‑trust principles.
• Encryption, key management, and secure handling of sensitive data with cloud‑native services.
• Observability and alerting to detect anomalous data movement and potential leakage events.
• Strong understanding of cloud‑native security and governance.
• Good to have: Databricks, AWS Clean Rooms, advertising measurement platforms, collaboration with Security/Privacy/Risk/Compliance, ELK/Grafana/OpenTelemetry, Docker and Kubernetes, lineage and governance tooling, and documenting data contracts and flows.
• Strong written and verbal English communication skills.
• Experience with Agile or SCRUM in cross‑functional product teams.