Senior Data Engineer - Vitality Drive International

<4>Senior Data Engineer The Senior Data Engineer is responsible for the design, development, and optimisation of scalable, enterprise-grade data platforms on Azure, with Azure Synapse Analytics at the core. This role enables the delivery of high-quality, trusted, and reusable data assets to support analytics, reporting, and AI use cases. The incumbent will play a key role in advancing a modern data lakehouse architecture, ensuring data is governed, performant, and aligned to business value outcomes. <4>Areas of Responsibility <4>Design - Translate business requirements into technical designs adhering to Discovery Insure processes, standards and guidelines, considering performance, security, scalability, and cost. - Define and evolve medallion architecture patterns in Microsoft Fabric (Bronze/Silver/Gold), including ingestion, transformation, and curated consumption layers. - Produce and maintain modern data models: OLTP-aligned models where needed, Dimensional models (star/snowflake) for analytics, Curated “Gold” layer data products (conformed dimensions, business-ready facts). - Design ingestion and transformation strategies across Fabric components (Lakehouse/Warehouse, Data Pipelines, Dataflows Gen2, Notebooks/Spark). - Establish data quality, lineage, and governance requirements (e.g., data contracts, validation rules, auditing, retention). - Liaise with relevant parties where clarification of business requirements or resolution to technical issues is needed. - Research and recommend effective solutions to technical issues that arise (e.g., performance bottlenecks, modelling trade-offs, orchestration patterns). - Estimate development timelines based on business requirements and technical complexity; identify risks and dependencies early. <4>Development - Construct robust, maintainable, scalable, optimally performing solution components in line with technical specifications, following prescribed process, standards and procedures. - Develop and optimise: T-SQL for Fabric Warehouse/Azure SQL (as applicable), ELT/ETL transformations (SQL/Spark) aligned to the medallion pattern, Reusable frameworks for ingestion, validation, and incremental processing. - Build and maintain orchestration using Fabric Data Pipelines (scheduling, dependencies, retries, parameterization). - Implement patterns for incremental loads and change processing (e.g., CDC-like approaches, watermarking, SCD Type 1/2 where appropriate). - Apply performance engineering: Partitioning strategies, incremental refresh patterns, Query optimisation and model simplification, Efficient file formats and table design for analytical workloads. <4>Testing - Conduct unit testing and fix any defects found, aiming for high-quality releases. - Verify build stability and quality with development team before releasing to test team, aiming to release with zero defects. - Build and maintain test data sets, validation rules, and reconciliation checks across Bronze/Silver/Gold. - Assist the Business Analyst in ensuring the test pack includes relevant scenarios and test data. - Consult and assist in reviewing risk / impact of defects found in testing and assist with fixing where necessary. <4>Knowledge and Skills - Modern Data Modelling & Analytics - Modern data modelling (OLTP where needed, dimensional modelling, curated analytical models, semantic consistency) - Strong understanding of medallion architecture concepts (Bronze/Silver/Gold, data quality gates, curated data products) - Metric definition discipline (facts, dimensions, conformed entities, grain, business rules) - Microsoft / Azure Data Platform - Microsoft Fabric: Lakehouse/Warehouse concepts, Pipelines, Dataflows Gen2, Notebooks/Spark (as applicable) - T-SQL expertise (complex querying, optimisation, robust patterns) - Performance tuning and optimisation for analytical workloads (SQL/Spark and modelling considerations) - Understanding of security patterns (RBAC, least privilege, workspace access, data access controls) - Engineering Excellence - Strong SDLC practices (requirements → design → build → test → release) - Source control and peer review culture (Git; branching strategies; PR discipline) - Data quality and validation frameworks (reconciliation, anomaly checks, rule-based validations) - Documentation and maintainability: clear design artifacts, data dictionaries, lineage-ready documentation <4>Working Knowledge of - Software development within SDLC. - Unit Testing and quality practices. - Data modelling and design of database structures. - Practical understanding of cloud data solution concepts (security, orchestration, monitoring, cost-awareness). <4>Advantageous - Power BI semantic modelling (star schemas, measures, performance considerations) - Governance tooling awareness (cataloging, lineage, data classification) - Exposure to CI/CD for data solutions (deployment pipelines, environment promotion concepts) <4>Education and Experience <4>Education - Matric (Essential) - BSc Computer Science or equivalent 3-year qualification - (advantageous) <4>Minimum Experience - 5+ years consistent experience in Microsoft data development with demonstrable delivery of complex solutions (senior-level capability). - Strong experience with SQL development and performance optimisation. - Proven experience designing and implementing data models and data structures in support of operational and analytical requirements. - Experience delivering data solutions aligned to layered architecture patterns (or demonstrable capability to adopt medallion practices quickly). <4>Advantageous - Experience modernising or migrating data workloads from on-prem to cloud. - Experience building curated “Gold” layer models for enterprise reporting/analytics. <4>How to Apply Apply via the link: [Apply now »](https://careers.discovery.co.za/talentcommunity/apply/1399483433/?locale=en_GB)