Skip to main content

Complete our 3 minute feedback survey to help us improve the framework.

Data engineer

Find out what a data engineer in government does and the skills you need to do the role at each level.

Last updated 30 August 2024 — See all updates

What a data engineer does

A data engineer develops and constructs data products and services, and integrates them into systems and business processes.

Data engineer role levels

There are 4 data engineer role levels, from data engineer to head of data engineering.

The typical responsibilities and skills for each role level are described in the sections below. You can use this to identify the skills you need to progress in your career, or simply to learn more about each role in the Government Digital and Data profession.

1. Data engineer

A data engineer delivers the designs set by more senior members of the data engineering community.

At this role level, you will:

  • implement data flows to connect operational systems, data for analytics and business intelligence (BI) systems
  • document source-to-target mappings
  • re-engineer manual data flows to enable scaling and repeatable use
  • support the build of data streaming systems
  • write ETL (extract, transform, load) scripts and code to ensure the ETL process performs optimally
  • develop business intelligence reports that can be reused
  • build accessible data for analysis

This role level is often performed at the Civil Service job grade of:

  • HEO (Higher Executive Officer)
  • SEO (Senior Executive Officer)
Skill Description

Communicating between the technical and non-technical

Level: awareness

Awareness is the first of 4 ascending skill levels

You can:

  • show an awareness of the need to translate technical concepts into non-technical language
  • understand what communication is required with internal and external stakeholders

Data analysis and synthesis

Level: working

Working is the second of 4 ascending skill levels

You can:

  • undertake data profiling and source system analysis
  • present clear insights to colleagues to support the end use of the data

Data development process

Level: working

Working is the second of 4 ascending skill levels

You can:

  • design, build and test data products based on feeds from multiple systems, using a range of different storage technologies, access methods or both
  • create repeatable and reusable products

Data innovation

Level: awareness

Awareness is the first of 4 ascending skill levels

You can:

  • show an awareness of opportunities for innovation with new tools and uses of data

Data integration design

Level: working

Working is the second of 4 ascending skill levels

You can:

  • deliver data solutions in accordance with agreed organisational standards that ensure services are resilient, scalable and future-proof

Data modelling

Level: working

Working is the second of 4 ascending skill levels

You can:

  • explain the concepts and principles of data modelling
  • produce, maintain and update relevant data models for an organisation’s specific needs
  • reverse-engineer data models from a live system

Metadata management

Level: working

Working is the second of 4 ascending skill levels

You can:

  • use metadata repositories to complete complex tasks such as data and systems integration impact analysis
  • maintain a metadata repository to ensure information remains accurate and up to date

Problem resolution (data)

Level: awareness

Awareness is the first of 4 ascending skill levels

You can:

  • explain the types of problems in databases, data processes, data products and services

Programming and build (data engineering)

Level: working

Working is the second of 4 ascending skill levels

You can:

  • design, code, test, correct and document simple programs or scripts under the direction of others

Technical understanding

Level: working

Working is the second of 4 ascending skill levels

You can:

  • understand the core technical concepts related to the role, and apply them with guidance

Testing

Level: awareness

Awareness is the first of 4 ascending skill levels

You can:

  • correctly execute test scripts under supervision
  • understand the role of testing and how it works

2. Senior data engineer

A senior data engineer designs and leads the implementation of data flows to connect operational systems, data for analytics and business intelligence (BI) systems.

At this role level, you will:

  • recognise opportunities to reuse existing data flows
  • lead the build of data streaming systems
  • optimise the code to ensure processes perform optimally
  • lead work on database management

This role level is often performed at the Civil Service job grade of:

  • SEO (Senior Executive Officer)
  • G7 (Grade 7)
Skill Description

Communicating between the technical and non-technical

Level: working

Working is the second of 4 ascending skill levels

You can:

  • communicate effectively with technical and non-technical stakeholders
  • support and host discussions within a multidisciplinary team, with potentially difficult dynamics
  • be an advocate for the team externally, and can manage differing perspectives

Data analysis and synthesis

Level: working

Working is the second of 4 ascending skill levels

You can:

  • undertake data profiling and source system analysis
  • present clear insights to colleagues to support the end use of the data

Data development process

Level: practitioner

Practitioner is the third of 4 ascending skill levels

You can:

  • design, build and test data products that are complex or large scale
  • build teams to complete data integration services

Data innovation

Level: working

Working is the second of 4 ascending skill levels

You can:

  • understand the impact on the organisation of emerging trends in data tools, analysis techniques and data usage

Data integration design

Level: practitioner

Practitioner is the third of 4 ascending skill levels

You can:

  • select and implement the appropriate technologies to deliver resilient, scalable and future-proofed data solutions

Data modelling

Level: practitioner

Practitioner is the third of 4 ascending skill levels

You can:

  • produce relevant data models across multiple subject areas
  • explain which models to use for which purpose
  • understand industry-recognised data modelling patterns and standards, and when to apply them
  • compare and align different data models

Metadata management

Level: practitioner

Practitioner is the third of 4 ascending skill levels

You can:

  • design an appropriate metadata repository
  • suggest changes to improve current metadata repositories
  • understand a range of tools for storing and working with metadata
  • advise less experienced members of the team about metadata management

Problem resolution (data)

Level: working

Working is the second of 4 ascending skill levels

You can:

  • respond to problems in databases, data processes, data products and services as they occur
  • initiate actions, monitor services and identify trends to resolve problems
  • determine the appropriate remedy and assist with its implementation, and with preventative measures

Programming and build (data engineering)

Level: practitioner

Practitioner is the third of 4 ascending skill levels

You can:

  • use agreed standards and tools to design, code, test, correct and document moderate-to-complex programs and scripts from agreed specifications and subsequent iterations
  • collaborate with others to review specifications where appropriate

Technical understanding

Level: working

Working is the second of 4 ascending skill levels

You can:

  • understand the core technical concepts related to the role, and apply them with guidance

Testing

Level: working

Working is the second of 4 ascending skill levels

You can:

  • review requirements and specifications, and define test conditions
  • identify issues and risks associated with work
  • analyse and report test activities and results

3. Lead data engineer

A lead data engineer is responsible for the design and implementation of numerous complex data flows to connect operational systems, data for analytics and business intelligence (BI) systems.

At this role level, you will:

  • recognise and share opportunities to reuse existing data flows between teams
  • be responsible for the build of data-streaming systems
  • co-ordinate teams and set best practice and standards
  • apply knowledge of systems integration to your work
  • champion data engineering across government

This role level is often performed at the Civil Service job grade of:

  • G7 (Grade 7)
Skill Description

Communicating between the technical and non-technical

Level: practitioner

Practitioner is the third of 4 ascending skill levels

You can:

  • listen to the needs of technical and business stakeholders, and interpret them
  • effectively manage stakeholder expectations
  • manage active and reactive communication
  • support or host difficult discussions within the team or with diverse senior stakeholders

Data analysis and synthesis

Level: practitioner

Practitioner is the third of 4 ascending skill levels

You can:

  • understand and help teams to apply a range of techniques for data profiling
  • source system analysis from a complex single source
  • bring multiple data sources together in a conformed model for analysis

Data development process

Level: expert

Expert is the fourth of 4 ascending skill levels

You can:

  • establish enterprise-scale data integration procedures across the data development life cycle, and ensure that teams adhere to them
  • manage resources to ensure that data services work effectively at an enterprise level

Data innovation

Level: practitioner

Practitioner is the third of 4 ascending skill levels

You can:

  • identify areas of innovation in data tools and techniques, and recognise appropriate timing for adoption

Data integration design

Level: expert

Expert is the fourth of 4 ascending skill levels

You can:

  • establish standards, keep them up to date and ensure adherence to them
  • keep abreast of best practice in industry and across government

Data modelling

Level: expert

Expert is the fourth of 4 ascending skill levels

You can:

  • understand the concepts and principles of data modelling and can produce relevant data models
  • work across government and industry, recognising opportunities for the reuse and alignment of data models in different organisations
  • design the method to categorise data models within an organisation

Metadata management

Level: practitioner

Practitioner is the third of 4 ascending skill levels

You can:

  • design an appropriate metadata repository
  • suggest changes to improve current metadata repositories
  • understand a range of tools for storing and working with metadata
  • advise less experienced members of the team about metadata management

Problem resolution (data)

Level: practitioner

Practitioner is the third of 4 ascending skill levels

You can:

  • ensure that the most appropriate actions are taken to resolve problems as they occur
  • co-ordinate teams to resolve problems and to implement solutions and preventative measures

Programming and build (data engineering)

Level: practitioner

Practitioner is the third of 4 ascending skill levels

You can:

  • use agreed standards and tools to design, code, test, correct and document moderate-to-complex programs and scripts from agreed specifications and subsequent iterations
  • collaborate with others to review specifications where appropriate

Technical understanding

Level: practitioner

Practitioner is the third of 4 ascending skill levels

You can:

  • show a thorough understanding of the technical concepts required for the role, and can explain how these fit into the wider technical landscape

Testing

Level: working

Working is the second of 4 ascending skill levels

You can:

  • review requirements and specifications, and define test conditions
  • identify issues and risks associated with work
  • analyse and report test activities and results

4. Head of data engineering

A head of data engineering leads multi-functional delivery teams to deliver robust data services for their department, other government departments and private sector partners.

At this role level, you will:

  • inspire best practice for data products and services within your teams
  • build data engineering capability by providing technical leadership and career development for the community
  • work with other senior team members to identify, plan, develop and deliver data services

This role level is often performed at the Civil Service job grade of:

  • G7 (Grade 7)
  • G6 (Grade 6)
Skill Description

Communicating between the technical and non-technical

Level: expert

Expert is the fourth of 4 ascending skill levels

You can:

  • mediate between people and mend relationships, communicating with stakeholders at all levels
  • manage stakeholder expectations and moderate discussions about high risk and complexity, even within constrained timescales
  • speak on behalf of and represent the community to large audiences inside and outside of government

Data analysis and synthesis

Level: practitioner

Practitioner is the third of 4 ascending skill levels

You can:

  • understand and help teams to apply a range of techniques for data profiling
  • source system analysis from a complex single source
  • bring multiple data sources together in a conformed model for analysis

Data development process

Level: expert

Expert is the fourth of 4 ascending skill levels

You can:

  • establish enterprise-scale data integration procedures across the data development life cycle, and ensure that teams adhere to them
  • manage resources to ensure that data services work effectively at an enterprise level

Data innovation

Level: expert

Expert is the fourth of 4 ascending skill levels

You can:

  • investigate emerging trends in data-related approaches, perform horizon-scanning for the organisation and introduce innovative ways of working

Data integration design

Level: expert

Expert is the fourth of 4 ascending skill levels

You can:

  • establish standards, keep them up to date and ensure adherence to them
  • keep abreast of best practice in industry and across government

Data modelling

Level: working

Working is the second of 4 ascending skill levels

You can:

  • explain the concepts and principles of data modelling
  • produce, maintain and update relevant data models for an organisation’s specific needs
  • reverse-engineer data models from a live system

Metadata management

Level: expert

Expert is the fourth of 4 ascending skill levels

You can:

  • identify how metadata repositories can support different areas of the organisation
  • communicate the value of metadata repositories
  • set up robust governance processes to keep repositories up to date

Problem resolution (data)

Level: expert

Expert is the fourth of 4 ascending skill levels

You can:

  • anticipate problems and know how to prevent them
  • understand how problems fit into the bigger picture
  • identify and describe problems, and help others to describe them
  • build problem-solving capabilities in others

Programming and build (data engineering)

Level: expert

Expert is the fourth of 4 ascending skill levels

You can:

  • set local or team-based standards for programming tools and techniques and can select appropriate development methods
  • advise on the application of standards and methods and ensure compliance
  • take technical responsibility for all stages and iterations in a software development project, providing method-specific technical advice and guidance to project stakeholders

Technical understanding

Level: expert

Expert is the fourth of 4 ascending skill levels

You can:

  • anticipate and advise on future technology changes that present opportunities for the product or programme

Testing

Level: working

Working is the second of 4 ascending skill levels

You can:

  • review requirements and specifications, and define test conditions
  • identify issues and risks associated with work
  • analyse and report test activities and results
Role Shared skills
Analytics engineer

Communicating between the technical and non-technical

Data analysis and synthesis

Data innovation

Metadata management

Problem resolution (data)

Programming and build (data engineering)

Testing

Data architect

Communicating between the technical and non-technical

Data analysis and synthesis

Data innovation

Data modelling

Metadata management

Application operations engineer

Technical understanding

Testing

Data governance manager

Communicating between the technical and non-technical

Data innovation

End user computing engineer

Technical understanding

Testing

Updates

Published 7 January 2020

Last updated 30 August 2024

30 August 2024

  • The skill level descriptions for 'metadata management' have been updated to improve clarity and ensure consistency across the framework. No change was made to the meaning of skill level descriptions.

31 May 2024

  • The indicative job grades for 3 role levels have been updated. Data engineer has been updated from 'EO and HEO' to 'HEO and SEO'. Senior data engineer has been updated from 'HEO and SEO' to 'SEO and G7'. Lead data engineer has been updated from 'SEO and G7' to 'G7'. This change is based on the latest data on the most common grades for these role levels across government.

31 March 2023

  • The ‘data modelling’ skill description has been updated at practitioner level.

30 August 2022

  • The ‘communication skills (data)’ skill has been renamed ‘communicating between the technical and non-technical’ to ensure consistency across the DDaT Profession Capability Framework.

7 January 2020

  • First published.