AMS PSR

On behalf of The Department for Education, we are looking for an Azure Databricks Platform Engineer (Inside IR35) for an 8 Month Remote contract with very occasional visits to a DfE office.

Role Overview

We are seeking a Databricks Administrator to support the Analytical Data Access (ADA) programme, a large-scale transformation initiative focused on migrating legacy SQL Server data into a centralised Delta Lake using Azure Data Factory (ADF) and Databricks.

This role will primarily focus on Databricks platform administration, including user access management, governance, and day-to-day operational support, while also providing support for data engineering activities across the wider platform.

Key Responsibilities

Databricks Platform Administration 

•    Administer and maintain the Azure Databricks workspace, ensuring platform stability, security, and performance.

•    Manage user access and permissions (RBAC, groups, roles, and service principals) on a day-to-day basis.

•    Support user onboarding and offboarding in compliance with data governance and security requirements.

•    Monitor workspace usage, cluster performance, and resource consumption.

•    Implement and maintain access controls aligned with organisational governance standards.

•    Manage secrets, tokens, and secure configurations within Databricks.

•    Support audit activities, including access reviews, compliance reporting, and security assessments.

Data Engineering & Integration

•    Develop, maintain, and support data pipelines using Azure Data Factory (ADF).

•    Support the ingestion and transformation of data from legacy systems into the Delta Lake environment.

•    Work with structured and semi-structured data within a lakehouse architecture.

•    Assist with the optimisation of Databricks notebooks, jobs, and workflows.

•    Ensure the quality, reliability, and performance of data pipelines and integrations.

Collaboration & Stakeholder Support

•    Work closely with Product Owners, Business Analysts, Data Owners, and Governance teams.

•    Support the onboarding of new analyst teams onto the ADA platform, ensuring smooth access and setup.

•    Provide guidance and best practice recommendations for using Databricks and Delta Lake.

•    Contribute to platform documentation, including process guides, access procedures, and operational runbooks.

Governance, Security & Compliance

•    Ensure adherence to data governance, security, and compliance standards.

•    Support the implementation of Data Sharing Agreements (DSAs) and access approval processes.

•    Maintain accurate documentation of data access, permissions, and lineage where required.

•    Ensure platform operations align with Agile delivery principles and GDS standards, where applicable.

Key Skills & Experience

Essential

•    Strong experience administering Azure Databricks environments.

•    Experience managing user access, roles, permissions, and RBAC controls.

•    Hands-on experience with Azure Data Factory (ADF).

•    Understanding of Data Lake, Delta Lake, and Lakehouse architectures.

•    Experience supporting users on live data platforms.

•    Knowledge of data governance, security, and access management principles.

•    Strong analytical, problem-solving, and stakeholder management skills.

Desirable

•    Experience working within Agile delivery environments.

•    Knowledge of Azure Active Directory (Azure AD/Entra ID) and identity management.

•    Familiarity with Unity Catalog and other data access governance tools.

•    Experience with SQL, Python, and/or Apache Spark.

•    Experience supporting data migration and cloud transformation programmes.

Please be aware that this role can only be worked within the UK and not Overseas.

Disability Confident 

As a member of the Disability Confident Scheme, MOJ guarantees to interview all candidates who have a disability and who meet all the essential criteria for the vacancy. In cases where we have a high volume of candidates who have a disability who meet all the essential criteria, we will interview the best candidates from within that group. This scheme encourages candidates with a disability and/or neurodivergence to apply. In exceptional circumstances, we may also need to apply the desirable criteria in our shortlisting process which may include holding active security clearance.

Armed Forces Covenant

The Ministry of Justice guarantees to interview veterans or spouses / partners of military personnel who meet all the essential criteria for the vacancy. In cases where we have a high volume of ex-military candidates / military spouses or partners, who meet all of the essential criteria, we will interview the best candidates from within that group. In exceptional circumstances, we may also need to apply the desirable criteria in our shortlisting process which may include holding active security clearance.

In applying for this role, you acknowledge the following "this role falls in scope of the Off Payroll Working in the Public Sector legislation. Any rates of payment quoted will reflect the gross rate per day for the assignment and will be subject to appropriate taxes and statutory costs. As such the payment to the intermediary and your income resulting from this contract will be different".