Databricks Engineer

Concentrix; Austin (TX); Remote; USD 92,000 - 120,000; Full time

Benefits offered by this job

Medical, dental, and vision insurance

401(k) retirement plan

Paid time off and holidays

Paid learning days

Employee assistance program

Job summary

Concentrix is hiring a Databricks Engineer to build, maintain, and support data pipelines on the Databricks Lakehouse platform across Azure, AWS, or GCP. The role emphasizes Python, PySpark, SQL, and data modeling with strong CI/CD and governance practices.

You will own end-to-end delivery from development to production support, collaborating with stakeholders and IT to ensure performance and data quality standards are met.

Qualifications

• 6+ years of professional experience in data engineering or data warehousing.
• 3+ years working with Databricks in a data engineering capacity.
• 3+ years of Python with PySpark DataFrames, Spark SQL or pandas API on Spark.
• 2+ years of data engineering on Microsoft Azure (ADLS Gen2, Data Factory, Key Vault).

Responsibilities

• Build and maintain batch and streaming ETL/ELT pipelines into Databricks lakehouse on Azure.
• Ingest data using Auto Loader, Lakeflow Connect, and Azure Data Factory.
• Develop pipelines with Lakeflow Spark Declarative Pipelines (Delta Live Tables).
• Manipulate data with PySpark and Spark SQL for analytics readiness.
• Configure, schedule, and monitor data workflows with Lakeflow Jobs.
• Implement medallion architecture and governance with Unity Catalog.
• Write unit tests and deploy via CI/CD using Git and Declarative Automation Bundles.
• Monitor, troubleshoot, and resolve production pipeline issues.
• Contribute to code reviews and team documentation.
• Collaborate with business and IT stakeholders to meet performance goals.

Skills

Python

SQL

PySpark

Data modeling

CI/CD

Testing

Data quality

Education

Bachelor's degree in CS / IS / Data Engineering

Tools

Databricks

Lakeflow

Auto Loader

Azure Data Factory

Unity Catalog

Delta Lake

Job description

Title: Databricks EngineerLocation: Remote in USAWorking hours in ESTRequired certification: Active Databricks Certified Data Engineer Associate certification.Role SummaryThe Data Engineer is responsible for building, maintaining, and supporting data solutions on the Databricks Data Intelligence Platform running on Microsoft Azure, AWS or GCP. The role leverages Databricks, Python, PySpark, and SQL to deliver data ingestion, transformation, modeling, and pipeline orchestration.Working within the standards and designs set with the Senior Data Engineer, this role independently owns the delivery of assigned pipelines from development through production support.ResponsibilitiesBuild and maintain batch and streaming data pipelines for extracting, transforming, and loading (ETL/ELT) data from multiple sources into the Databricks lakehouse on Azure.Implement data ingestion using Auto Loader, Lakeflow Connect, and Azure Data Factory.Develop pipelines with Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables), including data quality expectations.Perform data manipulation and transformation using PySpark and Spark SQL, ensuring data is prepared for analytics and reporting.Configure, schedule, and monitor data workflows with Lakeflow Jobs (formerly Databricks Workflows).Implement data models within the lakehouse following medallion architecture (bronze/silver/gold) and team modeling standards.Apply Unity Catalog permissions and governance standards to the data assets you build.Write unit and data quality tests and deploy pipelines through CI/CD using Git folders and Declarative Automation Bundles (formerly Databricks Asset Bundles).Monitor, troubleshoot, and resolve pipeline failures and data quality issues in production.Participate in code reviews and contribute to team documentation and engineering standards.Collaborate with cross-functional teams, including business stakeholders and IT, to ensure data solutions meet business and performance requirements aligned to project goals.Required QualificationsExperience6+ years of professional experience in data engineering, data & analytics or data warehousing.3+ years of hands-on experience working with Databricks in a data engineering capacity.3+ years of experience with Python, including manipulating large datasets using PySpark DataFrames, Spark SQL, and/or the pandas API on Spark.2+ years of data engineering experience on Microsoft Azure (e.g., ADLS Gen2, Azure Data Factory, Key Vault).Skills and KnowledgeProficiency in Python programming for data engineering tasks.Strong knowledge of SQL for data modeling, transformation, and querying.Working knowledge of Delta Lake fundamentals (e.g., MERGE, schema evolution, time travel).Working knowledge of Unity Catalog concepts, including catalogs, schemas, and permissions.Experience with Git-based version control and familiarity with CI/CD practices for data pipelines.Understanding of testing and deployment strategies for data pipelines, including productionizing ETL pipelines and Databricks SQL queries.EducationBachelor’s degree in computer science, Information Systems, Data Engineering, or a closely related field, or equivalent professional experience.Preferred QualificationsExperience implementing streaming data ingestion in Databricks (Structured Streaming, Auto Loader, Event Hubs).Experience with healthcare data (e.g., claims, EHR/clinical data, HL7/FHIR) and HIPAA/PHI handling.Exposure to Spark and Delta Lake performance tuning and compute cost optimization.Familiarity with dimensional modeling and SCD Type 1 and Type 2 patterns.Experience enabling BI consumption through Databricks SQL warehouses and tools such as Power BI.The base salary for this position is $92,250 to $120,000 plus incentives that align with individual and company performance. Actual salaries will vary based on work location, qualifications, skills, education, experience, and competencies. Benefits available to eligible employees in this role include medical, dental, and vision insurance, comprehensive employee assistance program, 401(k) retirement plan, paid time off and holidays and paid learning days.We accept applications for this position on an ongoing basis. It is currently for an existing, immediate vacancy; however, we are also considering applications for future opportunities as they arise. We encourage all interested candidates to apply.As part of the hiring process, candidates may be required to undergo background screening and identity verification, where permitted by applicable law and consistent with the requirements of the role. Certain verification processes used by the Company or its service providers may involve technologies that rely on biometric identifiers or biometric information, where permitted by law. If biometric identifiers or biometric information are collected, used, or stored, the Company will provide the legally required disclosures and obtain any required written consent prior to such collection, and will handle such information in accordance with applicable biometric privacy laws and Company policies.#Concentrix#WAH#LI-RemotePhysical and Mental Requirements:The employee is regularly required to operate a computer, keyboard, telephone/headset, and/or other office equipment as essential functions of this position. Work is generally sedentary in nature.Equal Employment Opportunity:Concentrix is an equal opportunity and affirmative action (EEO-AA) employer. We promote equal opportunity to all qualified individuals and do not discriminate in any phase of the employment process based on race, color, religion, sex, sexual orientation, gender identity, national origin, age, pregnancy or related condition, disability, status as a protected veteran, or any other basis protected by law.For more information regarding your EEO rights as an applicant, please visit the following websites:· English· SpanishAccommodation:Concentrix welcomes and encourages applications from candidates with disabilities and is committed to providing an inclusive recruitment process. If you require reasonable accommodation to participate in any stage of the application or interview process, please let us know. Requests may be made by contacting [e-mail]. All information will be treated confidentially and used solely to facilitate your participation in the recruitment process.Artificial Intelligence:As part of our recruitment process, we may use artificial intelligence (AI) tools to assist in the screening and/or assessment of job applicants. These tools could be used to evaluate resumes, applications, and other materials submitted to help us identify the best candidates for the role.Work Authorization:In accordance with federal law, only applicants who are legally authorized to work in the United States will be considered for this position. Must reside in the United States or have a valid U.S. address for residence.For further information on available work states and Equal Employment Opportunity as an applicant, please click HERE