12 Sep 2019 - Final report published into West Atlantic 737-400 G-JMCR electrical failures in-flight on 12 Oct 2018
On 12 Oct 2018 West Atlantic 737-400, G-JMCR (25372/1992), was operating a night flight to East Midlands Airport, with the left engine generator disconnected, and had just commenced its descent when the crew faced an unusual array of electrical failures on the flight deck. Despite the loss and degradation of a number of systems, the aircraft landed safely at East Midlands. The electrical failures were caused by the right engine Generator Control Unit (GCU) which had been incorrectly secured in its mounting tray and had disconnected in flight. The investigation also uncovered a number of contributory factors including: the management of defects and Acceptable Deferred Defects (ADD), recording of maintenance, and a number of weaknesses in the operator’s Safety Management System with regards to managing risk. Six Safety Recommendations are made to the operator regarding its safety management system and one to the Civil Aviation Authority.
On 6 Sep 2019 the AAIB published their final report into the incident.
It concluded that:
Extracts of the report are given below
*** Updated 18 Apr 2019 ***
The aircraft was operating a night flight to East Midlands Airport, with the left engine generator disconnected, and had just commenced its descent when the crew faced an unusual array of electrical failures on the flight deck. Despite the loss and degradation of a number of systems, the aircraft landed safely at East Midlands. The electrical failures were caused by the right engine Generator Control Unit (GCU) which had been incorrectly secured in its mounting tray and had disconnected in flight. The investigation also uncovered a number of contributory factors including: the management of defects and Acceptable Deferred Defects (ADD), recording of maintenance, and a number of weaknesses in the operator’s Safety Management System with regards to managing risk. Six Safety Recommendations are made to the operator regarding its safety management system and one to the Civil Aviation Authority.
GCU 2 protruding forward (AAIB report)
History of the Flight
The crew reported for work at Leipzig Halle Airport, Germany, on the evening of 11 October 2018. They were rostered to operate a three-sector day from Leipzig to Amsterdam Schiphol Airport, then to East Midlands Airport and finally to Aberdeen Airport. On arrival at the aircraft, the crew met with the pilots who had flown the aircraft into Leipzig and briefly discussed that the aircraft was operating with an ADD for an inoperative Gen 1. The aircraft was permitted to operate under Minimum Equipment List (MEL) 24-1b providing the APU, and its generator, were run during the flight. In this condition the No 1 electrical system was powered by the APU generator and the No 2 system by the engine-driven generator on the right engine (Gen 2).
At 2243 hrs, the aircraft departed from Leipzig and the flight was without incident until the landing at Amsterdam when the co-pilot’s flight instruments, which are powered by the No 2 electrical system, intermittently blanked and several electrical warning lights on the overhead panel illuminated intermittently. The crew were unable to determine the cause of the problem and concluded that Gen 2 had failed, leaving the APU generator providing the only electrical power to the AC busses. They attempted to select the APU generator to provide power to the No 2 electrical system, but it would not connect. The aircraft was taxied to the parking stand and shut down. The crew were aware that the MEL did not allow the aircraft to dispatch with only a single generator functioning and, therefore, the crew contacted the operator’s Line Maintenance Control (LMC) who arranged for an engineer in Amsterdam to attend the aircraft. After around 30 minutes, the engineer arrived at the aircraft and was briefed by the commander. He was seen to open the cowlings on the right engine in order to examine Gen 2; he also checked the relevant circuit breakers and Panel M238 on the sidewall of the cockpit. The engineer informed the crew that he had reset a circuit breaker and was confident that this was the cause of the problem but would require the right engine to be run in order to ensure that the engine generator was working correctly. The engine run was performed satisfactorily and the generator on the right engine and the No 2 electrical system worked normally. The engineer cleared the entry in the aircraft technical log and as part of their pre-flight preparation the crew discussed the actions they might take in the event they lost the remaining engine generator. The aircraft departed Amsterdam with the original ADD for an inoperative Gen 1.
The flight was without incident until the aircraft was approximately 60 nm from East Midlands, with the co-pilot as PF, when during the descent the autopilot disconnected, the co-pilot’s screens lost power and his flight instruments failed. The commander took control and disconnected the autothrottle as he was flying the aircraft manually. Numerous lights on the overhead panel and system annunciation panels illuminated and flashed, and multiple aural warnings were generated by the Terrain Avoiding Warning System (TAWS). As both crew members were visual with the runway, the commander instructed the co-pilot to make a PAN call and ask for vectors straight onto the ILS at East Midlands. During the next 20 minutes, and until the aircraft landed, the flight instruments on the co-pilot’s side came on and off numerous times. The commander manually flew an ILS approach onto Runway 27. The aircraft controls, flaps and gear worked normally although the distracting flashing warning lights and aural callouts continued throughout the approach. On landing, numerous aircraft systems failed including the autobrakes (although manual braking remained available), half the exterior lights and the commander’s speed indications on his electronic attitude display indicator. On reaching the stand, the crew were unable to connect the electrical ground power to the aircraft system. While the flaps were retracted, the flap indication showed them still deployed. No electrical power was available to the cargo door, cargo bay and multiple items on the flight deck.
The crew briefed the ground engineers and completed the technical log before continuing to Aberdeen on a replacement aircraft. The engineers later discovered that the GCU for Gen 2, which is located in the flight deck behind the right pilot seat, was not correctly fitted in its housing.
MEL for Boeing 737-300/400
The EASA has not issued a Master Minimum Equipment List (MMEL) for the Boeing 737‑300/400, instead the operator used a MEL, approved by the CAA, which was based on the FAA Boeing 737 MMEL5. For an inoperative engine generator, the MMEL specified that for dispatch the aircraft required one engine generator and the APU generator, which must operate normally and be used throughout the flight.
Rectification Interval Extension
Where a deferred defect cannot be cleared within the MEL time limits, the operator’s procedures allow a one-time Rectification Interval Extension (RIE). The Operation Manual sets out the procedures for authorising an RIE and states that it should only be used in ‘exceptional circumstances.’ The RIE must be approved by one of the three managers specified in the Operation Manual and must only be approved when ‘…it was not reasonably practical for the repairs to be made. “Reasonably practical” means the availability of spares, time and personal.’
On 11 October 2018 an RIE was approved for the extension of MEL 24-1b on G-JMCR for Gen 1. The reason for the extension was given as: ‘Extensive wiring checks and component replacements have been carried out to isolate the fault on generator 1 system which is tripping TRU 1 CB when selected. The fault finding so far has not managed to isolate the root cause, further trouble shooting required’
The ADD had originally been approved when the aircraft was on scheduled maintenance at East Midlands between 5 to 8 October 2018. The aircraft then returned to the operator’s main operating base at East Midlands on three further occasions before the RIE was approved when the aircraft was in Oslo. During this period, the aircraft landed at a number of other bases where troubleshooting and rectification was started but could not be completed before the aircraft was dispatched.
Following the event on 12 October 2018, an investigation into the cause of the electrical failures was carried out by the operator’s maintenance staff at East Midlands who identified a fault in the left engine electrical generating system and the incorrect racking of GCU 2.
Left engine electrical generation system
The fault in the left engine electrical generation system was traced to an open phase on one of the three power feeder cables that run from Gen 1 to its generator circuit breaker. The cause of the open phase was a burnt pin on connecter C at the wing / pylon disconnect. The operator reported that there was no evidence of arcing between the pin and either the adjacent pins or the body of the connector; both the socket and pin appeared to be formed correctly at the crimp. Due to the extensive damage to the pin, it was not possible to establish the cause of the damage.Comments by the aircraft manufacturer
The aircraft manufacturer advised that previous occurrences of burnt pins had usually been caused by the connector having not been correctly torqued, or because the wire spacer was missing, which created excessive load on the connector pins. When incorrectly torqued, vibration can cause the connector shell to move in relation to the receptacle connector shell. Relative motion between the connector shells would allow similar motion between the pin and socket contacts. This promotes wear and fretting corrosion that degrades the pin to socket contact interface, resulting in increased resistance and a rise in temperature. The aircraft manufacturer issued a Service Letter to operators with suggested actions to ensure the best possible connection of these pins and connectors. This was a known problem.
Generator Control Unit 2
After the incident flight, GCU 2 was found to be protruding from the equipment shelf by approximately 3 cm. This would have been sufficient for the contacts at the rear of the electrical unit to disengaged with the contacts in the shelf (Figure 6). The handle lever, which was in the locked position, was found to be serviceable and the locking hook and fork intact. There was no visible damage to the electrical shelf, tray or its connectors. The electrical unit was re-racked and the right engine electrical generating system was tested and found to be serviceable.
Comments by the aircraft manufacturer
The aircraft manufacturer advised that there had been no reports during the previous four years of electrical units having been incorrectly racked. There was also no model-wide or Boeing 737 fleet trend of units separating from the racks during flight.
The Boeing 737-400 uses a Quick Reference Handbook (QRH) for abnormal and emergency situations, which includes a section on aircraft electrics. None of the checklists in the QRH matched the flight deck indications directly, with the closest match being the ‘Loss of all Engine Driven Generators’. The crew did not consider this checklist as they did not believe they had lost all engine generators. Using this checklist would have involved the crew reading through the first part without completing any actions as it did not apply to their situation as the APU was already running. Eventually it would have instructed them to connect the APU generator to AC Bus 2 to power the part of the electrical system that most needed the electrical power from the APU. However, it is unclear if this would have been effective, given the inability to connect the APU onto the No 1 AC Bus at Amsterdam, and with the fault having been caused by the disconnection of GCU 2. There was no other checklist that would have provided any assistance to the crew.
Additional abnormal and emergency procedures
The operator recognises that crews can be faced with complex and challenging problems when operating an aircraft. The use of a strategy to manage the resources available to the crew and to assist them in dealing with a problem is recommended in the company Operation Manual. The strategy recommended in the manual is the decision-making tool DODAR. This mnemonic is a circular tool in that the last action is to review the actions and decisions the crew have made, thereby encouraging them to continually reassess whether their course of action is still the most valid. The letters of DODAR correspond to: D – Diagnose (what is the problem) O – Options (hold, divert, immediate landing etc) D – Decide (which option) A – Act/Assign (carry out selected option and assign tasks) R – Review (can involve the addition of new information, and/or the ongoing result(s) of selected options)
The Operations Manual provides more information on each step of the DODAR mnemonic. Before departing on the incident flight the crew discussed what actions they might take in the event of the loss of the working engine generator. However, during this event the crew did not carry out a DODAR or use any other tool to assess the situation.
In the twelve days leading up to this serious incident there had been a number of electrical power faults on the aircraft which had resulted in engineering activity taking place at several locations across Europe.
Maintenance carried out at Amsterdam
At 0040 hrs, after landing at Amsterdam, the co-pilot contacted LMC at East Midlands while taxiing to the stand. From the LMC telephone recordings the co-pilot can be heard explaining that they had lost Gen 2 after landing. The APU generator was still running and connected, but various warning lights and instruments were “blinking”. At 0054 hrs, the commander then informed LMC that he was parked on the stand and confirmed that both engine generators had failed and the voltages and frequency on the gauges all indicated zero.
At 0120 hrs, LMC contacted an EASA Part 145 organisation at Amsterdam and spoke to the senior engineer on shift and requested assistance. LMC briefed the engineer that the crew had reported that they had lost both Gen 1 and Gen 2. There was already an ADD for Gen 1. The engineer was asked to see if they could “Get the number 2 back and reset the system so that they could get the aircraft back to East Midlands”. The Part 145 organisation responded to the call and recorded the work to be carried out in the billing invoice, which stated ‘Both GEN’S INOP’. No other documentation between the two organisations was raised.
At 0124 hrs, LMC contacted the commander, informed him that engineers were on their way and asked for a full description of the electrical problems on the aircraft to record on their system (FSR). The commander gave a very detailed brief during which clarification as to what the crew had experienced was sought by LMC. LMC commented that there was a serious electrical problem on the aircraft and they had been unable to identify the root cause. He advised the commander to wait and see what the engineers at Amsterdam found.
A licensed engineer (B1) with a type rating for the Boeing 737-300/400 was tasked to attend the aircraft and as the shift was relatively quiet was accompanied by the senior engineer. On arriving at the aircraft, the engineer noted that the Ground Power Unit (GPU) was connected and the APU was not running. In this configuration the GPU should have powered both Gen Bus 1 and 2. However, the indications showed that only Gen Bus 1 was powered, which was not what the engineer expected to see. Following a brief from the commander, the engineer believed that Gen 2 disconnected from Gen Bus 2 in flight; however, this was not the case. He also noted from his conversation with the commander and from reading the technical log that there was an ADD for Gen 1. However, the technical log entry made by the commander, which said ‘Gen 2 failed on landing, with both bus off & transfer bus off lights flashing, also all FO’s instruments flashing and blanking’, confused the engineer who could not understand why both generator busses were off when the APU generator would still have been on-line. The engineer was not aware that the gen 1 off Bus light on the Bus switching panel had been removed as part of the MEL and, therefore, could not have illuminated during the flight.
The engineer’s first action was to visually check the position of the ‘AC’ and ‘DC’ circuit breakers on the front of GCU 2. Both circuit breakers appeared to be fully in. He did not touch the GCU and was also not aware of it protruding forward out of the rack. He then checked the fault lights on Panel M238 and found that for Gen 2, the ‘FF’ lamp was illuminated; the lamps for ‘MT’, ‘HV’ and ‘LV’ were all extinguished.
Based on the briefing from the commander and the illuminated lamp ‘FF’, the engineer followed the trouble shooting chart for ‘The feeder fault light on the annunciator panel comes on’. As part of this action the engineer opened the cowling on the right engine and checked the drive on the CSDU (which had not disconnected) and the feeder cables from the generator to the Differential Protection Current Transfer, which visually appeared to be normal. Once the cowling had been closed, the commander, at the request of the engineer, started the right engine using the APU generator to power Gen Bus 1. When selected, Gen 2 would not come on-line. The engineer then cleared the ‘FF’ code on Panel M238 by pressing the erase button on the panel. When selected, Gen 2 connected to Gen Bus 2. With Gen Bus 1 powered by the APU, Gen 2 was cycled several times and it connected to Gen Bus 2 every time.
At 0155 hrs, the commander contacted LMC and told them the problem had been fixed and then passed the telephone to the engineer. The engineer told LMC that there had been a “frequency fault” which had been cleared by resetting the circuit breaker. The engineer mentioned that from the technical log there seemed to have been a number of electrical problems on the aircraft and LMC responded that they were going to conduct a further investigation over the weekend. The engineer cleared the entry in the aircraft technical log and the commander accepted the aircraft for flight.
The senior and licensed engineers both said that it was not a particularly busy night and that there was no pressure on them to complete the work and return the aircraft to service. They did not have access to the operator’s on-line portal which provided a technical history of the aircraft, but instead relied on the brief from the commander and the limited information in the technical log. The task in clearing the fault, from initial callout to clearing the entry in the technical log, took approximately 35 minutes.
The AAIB investigation could identify no evidence to show that the aircraft would have been grounded when it reached East Midlands for further investigation work on the electrical systems to be carried out. Instead, the same crew had been scheduled, after a short stop at East Midlands, to continue their flight in this aircraft to Aberdeen.
Cause of the electrical failures
he electrical failures that occurred during the landing at Amsterdam and on the subsequent flight to East Midlands were caused by GCU 2 moving forward in its rack far enough to cause the electrical connectors to disconnect. The flickering lights and screens indicate that initially there was a partial connection that was intermittent, but on landing at East Midlands the GCU appears to have moved forward sufficiently for the connector to fully disconnect. This would have resulted in the loss of: Gen 2; Gen Bus 2; Transfer Bus 2; 115V AC Electronic Bus 2; 28V DC Bus 2; and 28V DC Electronic Bus 2. Electrical System 1 would still have been powered by the APU generator through Gen Bus 1.
The Boeing 737-400 is not designed to operate with the GCU disconnected. While there was no record in the aircraft technical log or worksheets for the previous 12 days of GCU 2 having been disturbed, messages on the company’s FSR stated that it had been disconnected on three occasions during this period as part of the fault finding to clear the ADD on the left engine generator.
Flight crew performance
At no stage during the event did the flight crew consult the QRH or attempt to analyse the fault. They did not use either the operator’s suggested decision-making tool DODAR nor any other tool. Given the good weather conditions, the point in the flight when the failure occurred, and the availability of a long runway at East Midlands, the crew were able land the aircraft safely with few issues. Had the crew performed some kind of analysis and discussed options for a safe landing, it is likely that they would have decided that landing at East Midlands was still the safest option.
If the aircraft had been in cloud, with poor weather conditions on the ground, the crew would have faced a challenging recovery with little automation available and with the co‑pilot unable to monitor the flight path of the aircraft. The use of a suitable decision-making tool, such as DODAR, would have aided the crew in analysing the problem and agreeing a suitable solution. It would also have prompted them to consider what systems may have been inoperative, and its effect on the approach and aircraft’s landing performance. Although the time available to the crew was only around 15 minutes, this was long enough for such an analysis to be performed without delaying the approach and landing.
Use of the MEL and RIEs
The operator did not appear to use the MEL in the spirit of EASA’s Acceptable Means of Compliance or its own procedures. Rather than using the MEL to allow the aircraft to return to its main operating base where the faults could be rectified, it appears to have been used to enable the aircraft to meet operational commitments. Fault finding, and rectification was frequently stopped before the root cause had been identified and on a number of occasions the aircraft was dispatched from a location where the work could have been carried out.
The burnt pins on the feeder cable was a known fault. On 10 October 2018, an engineer correctly identified that there was a FF on Gen 1 and inspected the connector between the engine and pylon but ran out of time to check the connector between the pylon and wing where the burnt pin was located.
The RIE for the defect on Gen 1 should only have been granted in exceptional circumstances. However, while resources were available to identify and fix the fault within the specified time, the RIE was approved to enable the operator to meet operational commitments.
There also seemed to be confusion with operations and engineering staff within the LMC and the Part M organisation as to what constituted a main operating base. It was commonly believed that a number of locations across their operating network that had Part 145 organisations could be considered as a main operating base and that it was acceptable for aircraft to be dispatched from East Midlands with an ADD operating in accordance with the limitations in the MEL. This was, however, contrary to the operator’s Operation Manual.
The confusion as to what constituted a main operating base and the routine deviation from the operator’s procedures on the use of the MEL and RIE might have partly been due to the operator’s policy and procedures not being suitable for its routine operations. Therefore, the following Safety Recommendation is made:
Management of defects
The operator recognised that the management of defects and rectification across their fleet was challenging due to the nature of their operations. The aircraft were rarely in the same place on consecutive days and there were frequently changes to the flying programme, which made the provision of spares, specialist engineers and equipment difficult. The operator’s staff were also conscious of the tight turnaround times that their customers expected and whilst there was no evidence of external pressure having been applied to any individuals, there may have been an element of self pressure to ensure that aircraft were not delayed. Fault finding was frequently stopped part way through and on three separate occasions the GCU were swapped without the aircraft documentation having been completed in accordance with Commission Regulation (EU) No 1321/2014, (continuing airworthiness). The following Safety Recommendation is made:
The management of defects was primarily carried out by staff in the LMC. These individuals may be required to manage a number of issues on separate aircraft during their shift. Their main aim is to ensure that the company meets its operational commitments during their period of duty. The main oversight was undertaken during the 0600 hrs morning conference which involved representatives from LMC and the Part M organisation using the updates provided on the operator’s messaging system. Despite numerous entries on FSR highlighting concerns with the electrical system on G-JMCR, and the difficulty in completing the fault finding during the tight turnaround times, there was no evidence of a plan to ensure that the aircraft was given sufficient downtime to rectify the faults and clear the ADD. Instead, the issue drifted on with an RIE approval and a number of engineers at different locations repeating similar fault-finding tasks until eventually the GCU was incorrectly secured and disconnected in flight.
The operator has addressed the situation by establishing the post of Defect Controller who reports through the Part M organisation. However, this individual is not available outside normal office hours or during periods of holiday or sickness. Moreover, the morning conference calls only take place during the normal working week which means that frequently only the operations supervisor and the LMC staff are in a position to undertake a dynamic risk assessment of the ongoing airworthiness of individual aircraft. While these individuals have the authority to prevent an aircraft flying if they believe it is unsafe to do so, it might not be apparent to them that this dynamic oversight is a key part of their job. The following Safety Recommendation is made:
Communicating with other Part 145 organisations
The electrical fault that occurred during the landing at Amsterdam was unusual. Lights and screens that can only be on or off were flashing which indicated that there was an intermittent fault within the No 2 electrical system that eventually caused the circuit breaker for GCU 2 to trip. The Part 145 engineers did not have access to the operators FSR and would not have known the history of the electrical problems on the aircraft, which LMC described to the commander as serious. While the commander gave a detailed explanation to LMC as to the problems he had experienced, this was not relayed to the engineer who was tasked with rectifying the problem with Gen 2 and resetting the system so that the aircraft could return to East Midlands. No written tasking document, recent history of the aircraft or the concerns from LMC that there was a serious electrical problem on the aircraft were provided to the engineer. The engineer reset the system as requested and reported back to LMC who did not ask him to undertake any further work. The total time from the engineer being tasked to travelling to the aircraft and completing the work was 35 minutes.
In completing the trouble shooting as laid out in the Maintenance Manual, the engineer had satisfactorily completed the task he was given, which was to investigate why the two serviceable generators were inoperative. But the circuit breaker that was found to have tripped could not have caused the intermittent electrical supply to the flight deck instruments. Significantly, no one appeared to address the potential increase in risk to the safe operation of the aircraft should the fault reoccur in flight while operating with one generator already inoperative in accordance with MEL 21-1b.
The commander initially felt uneasy at the fault being cleared but was reassured when the engineer discussed what he had done with LMC: the engineer felt that his conversation with LMC was more to do with when the aircraft could be returned to service. In turn, the LMC was reassured by the commander, who was new to the company, and the engineer that the aircraft was now serviceable. However, the engineer in Amsterdam did not have knowledge of the ongoing electrical problems on the aircraft and none of the three parties discussed the impact of the fault on Gen Bus 2 reoccurring during the next flight. In summary, none of the three individuals involved had the full picture on the condition of the aircraft and a risk assessment was not carried out to determine if the aircraft was in a safe condition to continue flying with one generator inoperative. The following Safety Recommendation is made:
Safety management system
This investigation identified safety issues across a number of areas that had not been identified or addressed by the Operator’s SMS. Therefore, the following Safety Recommendations are made:
This serious incident was caused by the incorrect racking of GCU 2 which moved forward in flight initially causing an intermittent and then total disconnection of the electrical connector. The aircraft was not designed to operate with the GCU disconnected and the crew were presented with an unusual situation that was not covered in the QRH.
The activities surrounding the management of the faults on G-JMCR during the previous 12 days, and the actions of the crew in handling the emergency, indicates a weakness in the operator’s policies and procedures for the management of risk. Engineers were not always given sufficient time to investigate the faults, with the result that fault finding was often repeated and not finished. Work at a number of locations was not recorded as having been carried out in the aircraft documentation. The aircraft was dispatched from its main operation base with an ADD and flew through a number of locations where it could have been cleared, which was contrary to the procedures in the Operation Manual.
Communication between LMC, the commander and the Part 145 organisation at Amsterdam was ineffective in highlighting the underling technical problems on the aircraft. The engineer was unaware of the full history of the faults and the concerns that LMC conveyed to the commander that there was a “serious electrical fault on the aircraft”. The engineer was tasked with resetting the generators and spent less than 30 minutes at the aircraft. Despite the ongoing concerns with the electrical systems previously raised by a number of engineers and crews, and the unusual set of failures that occurred during the landing at Amsterdam, LMC did not carry out any form of risk assessment or ensure a deeper investigation was carried out before the aircraft departed Amsterdam. While the commander had the ultimate decision on accepting the aircraft, he was new to the company and may have relied on the advice of the engineers without being aware that the engineer had only been tasked with resetting the generators.
The operator had previously identified that there was a need to restructure LMC, introduce the post of Defect Controller and provide staff with further training to improve their competency.