Principal Data Engineer – Data Platform

The Planet Group; Irving (TX); Hybrid; USD 110,000 - 131,000; Full time

Job summary

Planet Group is seeking a Senior Lead Data Engineer to shape an enterprise data environment on Azure Databricks. You will develop end-to-end pipelines, refine notebooks, and set up robust data architectures with lakehouse principles.

You will evaluate existing notebooks and data workflows, identify dependencies, and guide modernization efforts. The role demands hands-on leadership and collaboration with business partners to productionize solutions.

Qualifications

• 7+ years of data engineering or data platform experience with ownership of complex productions.
• Strong hands-on experience with Azure Databricks, Spark, Python and SQL.
• Experience making architecture decisions and driving implementation and ongoing support.

Responsibilities

• Build automated pipelines that collect and process information from databases, APIs, documents, and other sources.
• Review notebooks and transformation logic with business experts to understand calculations, dependencies, and downstream uses.
• Refactor analytical code into modular, version-controlled solutions with testing, deployment, and support practices.
• Organize data into raw, validated, and business-ready layers using lakehouse and medallion principles.
• Develop reusable approaches to ingestion, transformation, data modeling, and pipeline orchestration.
• Establish checks for data accuracy and completeness, with monitoring to help teams identify and resolve failures.
• Incorporate access controls, data lineage, documentation, and recovery procedures into production solutions.
• Investigate processing bottlenecks and improve pipeline performance, reliability, and operating costs.
• Prepare well-defined data structures to support reporting, AI-assisted analysis, and conversational access to information.
• Evaluate new technologies and recommend where they add value.
• Provide technical guidance and collaborate with platform teams to move solutions from design into production.

Skills

Azure Databricks

Apache Spark/PySpark

Python

SQL

Data architecture

CI/CD

Data quality

Monitoring

RBAC / access controls

Notebook modernization

Data lakehouse

Tools

Unity Catalog

Delta Lake

Databricks Asset Bundles

dbt

Apache Iceberg

Job description

LeadDataEngineer-Databricks

Location: Dallas,TX

Hybridrole-requiresthreedaysaweekonsite

Contracttohire:(H1B'sacceptablebutMUSTbewillingtosigna90dayrelease)

Rate:$80-95/hour

ThePlanetGrouhaspartneredwithaDallasareacompanytolocateaseniordataengineerwhoenjoyssolvingcomplicateddataproblems,improvingexistingsystems,andhelpingteamsbuildbettersoftware.

Inthishands-ontechnicalleadershiprole,thesuccessfulcandidatewillhelpshapeanenterprisedataenvironmentbuiltonAzureDatabricksincludingdevelopingpipelines,strengtheningengineeringpractices,andturningexistinganalyticalcodeintoreliableapplicationsandreusablecomponents.

Animportantpartofthispositioninvolvesevaluatinganestablishedcollectionofnotebooksanddataworkflowsanddetermininghowtheywork,identifyingdependencies,andrecommendingwhichcomponentsshouldberetained,improved,consolidated,orrebuilt.

Responsibilities:

• Buildautomatedpipelinesthatcollectandprocessinformationfromdatabases,APIs,documents,andothersources.
• Reviewexistingnotebooksandtransformationlogic,workingwithbusinessexpertstounderstandcalculations,dependencies,anddownstreamuses.
• Refactoranalyticalcodeintomodular,version-controlledsolutionswithappropriatetesting,deployment,andsupportpractices.
• Organizedataintoraw,validated,andbusiness-readylayersusinglakehouseandmedallionarchitectureprinciples.
• Developreusableapproachestoingestion,transformation,datamodeling,andpipelineorchestration.
• Establishchecksfordataaccuracyandcompleteness,alongwithmonitoringthathelpsteamsidentifyandresolvefailures.
• Incorporateaccesscontrols,datalineage,documentation,andrecoveryproceduresintoproductionsolutions.
• Investigateprocessingbottlenecksandimprovepipelineperformance,reliability,andoperatingcosts.
• Preparewell-defineddatastructuresthatsupportreporting,AI-assistedanalysis,andconversationalaccesstoinformation.
• Evaluatenewtechnologiesandrecommendwheretheyofferpracticalvalue.
• Providetechnicalguidanceandcollaboratewithplatformteamstomovesolutionsfromdesignintoproduction.

RequiredSkillsandExperience:

• Atleastsevenyearsofrelevantdataengineeringordataplatformexperience,includingownershipofcomplexproductionsolutions.
• Strong,recenthands-onexperiencewithAzureDatabricks,ApacheSpark/PySpark,Python,andSQL.
• Experiencemakingarchitecturedecisionsandcarryingthosedecisionsthroughimplementationandongoingsupport.
• Atrackrecordofimprovinginheritednotebooks,prototypes,orlegacydataprocesses.
• Experiencebuildinglayereddataarchitecturesanddesigningdatasetsfordifferentbusinessandanalyticalneeds.
• Abilitytoworkwithrelationaldata,JSON,XML,APIs,andunstructuredsourcessuchasPDFs.
• ExperiencewithdbtandmoderntableformatssuchasApacheIceberg.
• Practicalknowledgeofworkflowscheduling,automatedtesting,CI/CD,dataquality,monitoring,androle-basedaccess.
• Strongtroubleshooting,datamodeling,andperformancetuningskill.
• Familiaritywiththedatafoundationsneededfornatural-languageanalytics,semanticmodels,andknowledgegraphs.
• Theabilitytoexplaintechnicaldecisionsclearlyandworkthroughcomplexrequirementswithbothtechnicalandbusinesspartners.
• Anindependentapproachtolearningunfamiliartools,understandingnewdatasets,andresearchingsolutions.

Preferredexperience:

• Healthcare,insuranceclaims,financialservices,consulting,orsimilarlycomplexbusinessdata.
• UnityCatalog,DeltaLake,andDatabricksAssetBundles.
• DatabricksGenieorcomparableconversationalanalyticstools.
• Knowledgegraphs,metadata-drivensolutions,orsemanticmodeling.
• ExperienceconnectingDatabricks,dbt,andopentableformatswithinanenterpriselakehouse.