Pilot Self-Assessment Systems

POSITION PAPER

23POS01

24 March 2023

Pilot Self-Assessment Systems

NOTE

This paper supersedes 19POS18 of the same name.

BACKGROUND

Airlines have introduced software and tools produced by third parties that enable flight crews to assess their own performance after every flight, based on Flight Data derived data. There are differences in the way each of these tools work, but the general idea conveyed by airlines is that such systems enable flight crews to review their last flight, or a trend of flights, and identify possible areas of improvement, which in turn leads to safety gains.

Whilst IFALPA always welcomes initiatives designed to enhance flight safety, there has been little evidence that these initiatives do so. Having had time to assess the implementation of these tools, the Federation has serious concerns that these tools may, on balance, be judged to negatively impact safety.

Of particular concern is the potential for misuse of data through:

• Competition - a competitive crew may compare their data to determine who had a smoother landing, landed in less distance, or got closest to some criteria such as fully configured and stable by 1,000ft, or who used the least amount of fuel.
• Ranking systems - these “competitions” are exacerbated by tools which produce the data in the form of a ranking system without any contextual information, bringing perceived pressure on pilots.
• Bullying - there is potential for bullying based on pilot self-assessment statistics or around a pilot’s unwillingness to participating in a fabricated competitive environment. There have also been cases of snapshots of this data being shared in ways that can identify pilots.
• Training - since the data is given to the pilot, an otherwise well-intentioned instructor may ask pilots to produce their data in an effort to focus training to their deficiencies. These are areas best left to a fully developed Flight Data Analysis Programme (FDAP) or Flight Operational Quality Assurance programme (FOQA).
• The benefits of a formal, established Flight Data Analysis Programme (FDAP) or Flight Operational Quality Assurance programme (FOQA) is that any contact made with pilots is made using validated data, a committee of safety-focused individuals will have considered the impact of raising the matter with the pilot and only do so with training or feedback that is suitably qualified. This system develops trust, which in turn protects the validity of the data being fed back to an airline’s Safety Management System (SMS), encouraging pilots to operate according to their safe judgement rather than flying in a way that benefits their statistics or is likely to avoid perceived punitive action.

Use of standalone pilot self-assessment systems risks releasing unvalidated data to pilots who are expected to self-debrief, draw their own conclusions, and execute their own self-training package. It is unlikely that individuals are best placed to make competency decision about their own flying. Expecting them to do so is at odds with current training standards as well as the multi-crew airline environment in which they operate. Take our example of a smooth landing, a pilot may use self-assessment data to improve their touchdown, but in doing so may teach themselves to land long or with excess thrust.

POSITION

IFALPA is concerned about misuse of data in pilot self-assessment systems, especially when these systems are poorly designed or implemented. Critical areas of such systems include the data collection process, data ownership, data protection, and the use of these tools for non-technical purposes, including any form of comparison between pilots and/or the establishment of “ranking” systems that would evaluate pilots, based on economic or safety aspects such as fuel use or stable approach criteria.

Self-assessment programs without the benefit of a developed FDAP/FOQA have the potential to provide unvetted data to a pilot who could make independent decisions on how to self-correct perceived deficiencies which could result in unintended consequences. A cooperatively developed FDAP/FOQA is the best way to fully analyze data, recommend corrective training and mitigate the risks of unintended consequences.

IFALPA therefore believes that careful thought must be given to the unintended negative effects any such system may have. If, following this review, an airline chooses to implement pilot self-assessment software/tools, IFALPA recommends that this be held wholly within an airline’s fully established FDAP/FOQA, which retain existing restrictions around data sharing and whose safety committee continuously oversees and develop such systems with a view to prioritizing the airline’s safety culture.

Data used in pilot self-assessment systems must follow the same procedures for data collection, validation, ownership, and storage that is used in the airline’s existing FDAP/FOQA. This will include the ability for each individual pilot to delete all selfassessment data presented to them and the ability for pilots to opt out of such systems without penalty or judgement. Once a reasonable pilot review period expires, the data should also be automatically deleted from the FDAP/FOQA system unless retained in line with the airline’s FDAP/FOQA for purposes other than pilot self-assessment.

Self-assessment software/tools must never be advertised by the airline as a “safety enhancement”. Safety is based on standards set and performance achieved across an airline/fleet/base, and not on individual data sent to individual pilots. Further, pilot selfassessment data should never be used to assess pilot competency, for disciplinary measures, or to assess training requirements. Pilots shall not be expected to assess their own operation without training system input.

Additionally, where data related to specific flights is sent directly to pilots, the following requirements must be met:

• Data has been validated by the airlines flight safety department;
• Data does not include any event that has been flagged by FDAP/FOQA for follow-up actions;
• Pilots have received suitable training to understand the use and limitations of such data;
• Prior consent of all pilots concerned has been explicitly obtained for each sector or a specific request of the pilot(s) concerned has been received.

©2023 The International Federation of Air Line Pilots’ Associations. This publication is provided for information purposes only, in all cases pilots should follow their company’s guidance and procedures. In the interest of flight safety, reproduction of this publication in whole or in part is encouraged. It may not be offered for sale or used commercially. All reprints must credit IFALPA.