W2 - 15+ Lead Data Engineer

Dice is the leading career destination for tech experts at every stage of their careers. Our client, Prohires, is seeking the following. Apply via Dice today!

Position

Lead Data Engineer

Main Skills

SQL, Databricks, ADF, Datastage (or other ETL tool), SSAS cubes, Cognos/other, Tableau/other, Thoughtspot/other

Employment typ

W2

Project duratio

9 months with possible extension

Location

San Francisco, CA

Work mode

1-2 days per week from office

Travel

No

Recruitment process

General -> Technical Interview -> Manager Interview -> Client Interview

Required Start Date

ASAP

Level

Senior/Lead level

Work authorization statu

H1B and TN visa candidates can be considered. Any applicant needs to be on your W2.

Description

Job Description

We are seeking a Lead Data Engineer to design, implement, and maintain our data infrastructure and business intelligence solutions. This role requires a technical expert who can architect scalable data solutions while collaborating across teams to drive business insights.

Required Experience

• 8-10 years of software development and deployment experience
• At least 7 years of hands-on experience with:
• SQL
• Databricks
• Azure Data Factory (ADF)
• Datastage (or other ETL tools)
• SSAS cubes
• BI tools (Cognos, Tableau, ThoughtSpot, etc.)
• Strong Database knowledge (SQL & No-SQL preferred)
• Advanced SQL and Python skills
• Experience working with APIs for data collection/ingestion

Key Responsibilities

Technical Development

• Write SQL for processing raw data, Kafka ingestions, ADF pipelines, data validation, and QA
• Lead, design and implement innovative analytical solutions using Tableau, SQL, and Big Data technologies
• Experience with storing, joining, filtering, and analyzing data using SQL, Spark, Hive, etc.

Data Architecture & Design

• Design comprehensive data architecture and data modeling strategies for BI systems
• Create logical and physical data models using best practices to ensure high data quality and reduced redundancy
• Collaborate with stakeholders to translate business requirements into robust, scalable data models
• Perform detailed analysis of business problems and technical environments to design quality solutions

Integration & Implementation

• Integrate data from various sources (databases, warehouses, external systems)
• Develop strategies for ETL processes ensuring data accuracy, consistency, and availability
• Evaluate and select appropriate BI tools based on organizational needs
• Run POCs and implement BI software, including visualization tools and analytics solutions

Optimization & Governance

• Monitor and optimize BI system performance for efficient data retrieval, processing, and reporting
• Identify and resolve performance bottlenecks through query tuning and data structure optimization
• Establish data governance policies ensuring data quality, consistency, and security
• Define access controls, privacy measures, and retention policies to meet regulatory requirements

Collaboration & Leadership

• Build partnerships with architects, technical leads, and stakeholders across functional groups
• Actively participate in code reviews and testing to ensure adherence to best practices
• Work with product and engineering teams to evaluate requirements and new features
• Build and foster a high-performance engineering culture
• Mentor team members and provide tools and motivation for success

Additional Skills & Requirements

• Experience building data ingestion pipelines and data warehouse architecture
• Ability to write design documentation, source-to-target mapping documentation, and manage Confluence pages
• Experience converting business functionalities into technical Jira stories
• Strong experience with data modeling, design patterns, and building scalable BI solutions
• Knowledge of cloud platforms (Azure, AWS or equivalent)
• Experience analyzing large datasets to identify trends, patterns, and outliers