Job Description

Job Purpose

We are looking for a talented and experienced Senior Data Engineer / Backend SQL Developer to join our Data Aggregation team. In this role, you will be a core contributor to our SQL Server backend, writing complex stored procedures, views, and aggregation logic that feeds downstream reporting and analytical workloads. Alongside traditional SQL Server development, you will extend these capabilities into the modern lakehouse using Python and Databricks, enabling scalable batch and near-real-time data processing across large datasets. The ideal candidate brings deep expertise in SQL Server (T-SQL, stored procedures, query tuning, indexing strategies) combined with hands-on experience in PySpark, Apache Iceberg, and Databricks Workflows - and is comfortable bridging the gap between on-premise SQL environments and cloud-based data platforms.

Responsibilities

• Design and implement robust ETL/ELT pipelines using PySpark and Python on Databricks
• Build and manage Apache Iceberg table formats for efficient data storage, versioning, and time-travel queries on the data lakehouse
• Develop and optimize complex SQL queries, stored procedures, and data models for analytical workloads
• Automate infrastructure provisioning, deployment, and operational tasks using PowerShell scripts
• Monitor and tune Databricks cluster performance, job scheduling, and cost optimization
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver reliable data products
• Enforce data quality, governance, and lineage best practices across the platform
• Participate in code reviews, technical design sessions, and Agile ceremonies

Knowledge and Experience

• 6+ years of experience in data engineering or similar role
• Experience working with cloud platforms (AWS / GCP)
• Familiarity with version control (Git) and CI/CD practices
• Strong understanding of data warehouse and data lakehouse concepts
• Apache Iceberg- Experience managing Iceberg table format - schema evolution, partitioning, compaction, time-travel
• Databricks- Hands-on with Databricks Workflows, Unity Catalog, Delta/Iceberg integration, cluster management
• Python - Proficient in writing clean, production-grade Python for data engineering tasks
• PySpark - Strong experience with distributed data processing, DataFrame API, optimizations
• SQL - Advanced SQL - window functions, CTEs, query optimization, schema design
• PowerShell - Scripting for automation, CI/CD pipelines, Windows/Azure environment management

Preferred Knowledge and Experience

• Experience with Azure Data Factory, dbt, or Airflow for orchestration
• Knowledge of SSIS and PowerBI/Tableau