Lead Data Engineer

Softcrylic; United States; Remote; USD 80,000 - 720,000; Full time

Job summary

A leading company in data solutions is looking for a Lead/Senior Data Engineer. This full-time remote position requires extensive experience in data engineering, specifically in migration processes involving BigQuery and Databricks. You will be a key player, ensuring data quality and overseeing the pipeline development. If you have a passion for transforming data into actionable insights and enjoy working in a team-focused environment, apply now.

Qualifications

• 8+ years of experience in Data Engineering.
• Strong focus on data migration and quality assurance.
• Experience migrating datasets from BigQuery to Databricks using Python.

Responsibilities

• Migrate data pipelines and develop Python scripts.
• Ensure quality of migrated data through rigorous principles.
• Review and comprehend the data pipeline framework.

Skills

Data migration processes

Python scripting

Data pipeline tools

Data quality management

Tools

Airflow

Airbyte

Dagster

Job description

Direct message the job poster from Softcrylic

Who We Are

For more than 20 years, we have been working with organizations large and small to help solve business challenges through technology. We bring a unique combination of engineering and strategy to Make Data Work for organizations.

Our clients range from the travel and leisure industry to publishing, retail and banking. The common thread between our clients is their commitment to making data work as seen through their investment in those efforts.

In our quest to solve data challenges for our clients, we work with large enterprise, cloud based and marketing technology suites. We have a deep understanding of these solutions so we can help our clients make the most of their investment in an efficient way to have a data driven business.

Softcrylic joined forces with Hexaware to Make Data Work in bigger ways!

Why Work at Softcrylic?

Softcrylic provides an engaging, team-focused, and rewarding work environment where people are excited about the work they do and passionate about delivering creative solutions to our clients.

We are looking to add a Lead/Senior Data Engineer to our team! This is a Full-Time position and a Remote role.

Job Description:

• 8+ years of experience in Data Engineering, with a strong focus on data migration, pipeline development, and quality assurance.
• Experience in migrating existing datasets from BigQuery to Databricks using Python scripts.
• Review and comprehend the data pipeline framework implemented by the client.
• Strong experience in developing Python scripts to migrate data from BigQuery to Databricks.
• Migrate data pipelines from Airflow to Airbyte/Dagster using the framework developed.
• Ensure quality of data and pipelines migrated through rigorous data quality principles.

Required Skills

• Strong understanding of data migration processes, including moving data from BigQuery to Databricks.
• Proficiency in developing Python scripts for data migration and automation.
• Experience with data pipeline tools such as Airflow, Airbyte, and Dagster.
• Expertise in implementing and adhering to data quality controls and testing frameworks.
• Ability to design and develop approaches for data migration, client onboarding, and data quality management.
• Familiarity with core architecture, platform design, client onboarding processes, and business goals.

Seniority level

• Seniority level

  Mid-Senior level

Employment type

• Employment type

  Full-time

Job function

• Job function

  Information Technology

• Industries

  Software Development

Referrals increase your chances of interviewing at Softcrylic by 2x

Get notified about new Data Engineer jobs in United States.

United States $170,000.00-$720,000.00 1 week ago

United States $170,000.00-$720,000.00 1 week ago

United States $170,000.00-$720,000.00 4 days ago

Irving, TX $80,000.00-$100,000.00 1 week ago

McLean, VA $80,000.00-$100,000.00 1 week ago

United States $93,000.00-$116,000.00 1 day ago

United States $120,000.00-$130,000.00 1 week ago

Data Engineer (Data Science and AI) [REMOTE]

Fort Walton Beach, FL $77,814.00-$132,283.00 3 days ago

Orange County, CA $75,000.00-$85,000.00 2 days ago

New York, NY $120,000.00-$132,000.00 5 days ago

United States $184,500.00-$217,100.00 1 week ago

New York, NY $139,500.00-$271,566.94 1 week ago

North Carolina, United States $125,000.00-$145,000.00 4 weeks ago

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.