Job Specification — Data Engineer x3 (Databricks/PySpark/Azure) - Ireland
Location: Limerick (preferred) OR UK based remote
Pay rate: €400-430 per day
6 month initial contract

Team: Data & Analytics — Supply Chain Reports to: Engineering Lead, within a SAFe Agile Release Train

1. About the Role
You will join a Data & Analytics team building a large-scale supply chain visibility and decision-intelligence platform. The platform ingests data from multiple enterprise source systems and curates it through a layered (medallion-style) data architecture to power interactive analytics for planners, supply chain leaders, and decision-makers.

The team delivers continuously on a 2-week sprint cadence with regular major release cycles, in a regulated environment where data accuracy and traceability are critical.

Data architecture (medallion-style): L0 (raw) ? L1 (standardized) ? L2 (curated) 2. Role Summary
We are looking for a Data Engineer to design, build, and maintain the data pipelines that feed the analytics layer. You will own data ingestion, transformation, and quality across the L0 ? L1 ? L2 layers, working in Databricks/PySpark and Azure Data Factory, ensuring curated, trustworthy data reaches the downstream applications.

You will work in an Agile squad alongside fellow data engineers and QA, contributing to a regulated environment where data accuracy and traceability are critical.

3. Key Responsibilities

Build & maintain data pipelines across the L0 ? L1 ? L2 layers using Databricks (PySpark) and Azure Data Factory (ADF).
Develop curated data models in the L2 layer that power supply chain KPIs (inventory health, demand/supply waterfalls, network health, plan attainment, risk logic, etc.).
Integrate new source systems into the platform, mapping raw feeds to standardized and curated schemas.
Implement data quality and validation controls — automated checks, reconciliation, and traceability from source to dashboard.
Write and maintain unit tests for your code (rather than relying on manual testing), contributing to automated test runs as part of the pipeline.
Optimize performance & cost of Spark jobs and data workflows — query tuning, indexing, and partitioning strategies; manage Unity Catalog governance.
Build and monitor ADF pipelines — orchestrate, schedule, and troubleshoot Azure Data Factory data flows.
Support releases on a regular major release cadence — participate in SIT/UAT, troubleshoot defects, and support hotfixes.
Collaborate in sprint ceremonies (planning, refinement, standups, retros) and document key design decisions.
Maintain CI/CD hygiene — pull requests, database migrations, code quality gates, and artifact management.

4. Required Skills & Experience
Core technical (must-have) Area Requirement Data processing Hands-on experience with Spark and Delta Lake for data processing and analysis (Databricks / PySpark) Languages Proficiency in Python (used for SQL-based transformations and pipeline logic) SQL platforms Experience with SQL Server and/or Synapse (formerly SQL Data Warehouse) Data modeling Solid understanding of data modeling principles and concepts Performance tuning Practical experience optimizing data processing, including indexing and partitioning for query performance Orchestration Comfortable building, monitoring, and managing Azure Data Factory (ADF) pipelines Testing Experience implementing unit testing practices for code (not relying solely on manual testing) CI/CD Understanding of CI/CD processes and their implementation Technical depth Strong technical depth across databases, testing, and performance tuning Ways of working

Comfortable in Agile/Scrum (SAFe Release Train experience a plus).
Experience supporting frequent releases and hotfix cycles.
Strong collaboration with QA and analytics consumers.

5. Nice-to-Have / Differentiators

Familiarity with Common Data Layer (CDL) concepts.
Python-based Spark development and monitoring ADF pipelines at scale.
neo4j / graph databases and Redis caching.
Supply chain / manufacturing domain experience.
Experience with AI-assisted tooling (e.g., GitHub Copilot and similar tools across the SDLC).

6. Tech Stack You'll Work With Layer Technologies Data / ETL Databricks (Spark + Delta Lake, PySpark), Azure Data Factory, Azure Blob, Unity Catalog Databases SQL Server / Synapse, neo4j Graph DB, Redis CI/CD & Quality Jenkins, Bitbucket, Artifactory, Flyway, SonarQube Testing Unit testing, automated test runs in CI Collaboration Jira, Confluence 7. What Success looks like

First Sprint — Onboarded to the L0?L1?L2 data architecture; environment access set up (Databricks, source control, CI/CD, SQL Server); shadowing a squad through a sprint, first User Story delivered.
First Release — Independently delivering data-engineering stories within a sprint; writing unit tests for your pipelines; raising clean pull requests that pass code quality gates.
Success — Owning a data domain end-to-end; contributing to a major release; proposing pipeline or quality improvements.

By applying for this role, you consent to SGI contacting you by telephone regarding recruitment services, market updates, and relevant business opportunities

We believe in equal opportunity for all and actively encourage applications from diverse backgrounds, experiences, and perspectives.
Source Group International Ltd is acting as an Employment Business in relation to this vacancy

Flexible job Project Support Programme job in Luxembourg (1)

Sign up to Job Alerts!

Senior Compliance Officer

Boost your career