Email Stuck in Exchange On-premises Transport Queues - Solution

 Microsoft  have addressed the issue causing messages to be stuck in transport queues of on-premises Exchange Server 2016 and Exchange Server 2019. The problem relates to a date check failure with the change of the new year and it not a failure of the AV engine itself. This is not an issue with malware scanning or the malware engine, and it is not a security-related issue. The version checking performed against the signature file is causing the malware engine to crash, resulting in messages being stuck in transport queues.

We have now created a solution to address the problem of messages stuck in transport queues on Exchange Server 2016 and Exchange Server 2019 because of a latent date issue in a signature file used by the malware scanning engine within Exchange Server. Customer action is required to implement this solution. When the issue occurs, you’ll see errors in the Application event log on the Exchange Server, specifically event 5300 and 1106 (FIPFS), as illustrated below:

Log Name: Application
Source: FIPFS
Logged: 1/1/2022 1:03:42 AM
Event ID: 5300
Level: Error
Computer: server1.contoso.com
Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long.

Log Name: Application
Source: FIPFS
Logged: 1/1/2022 11:47:16 AM
Event ID: 1106
Level: Error
Computer: server1.contoso.com
Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.

Using the Automated Solution

• Download the script here: https://aka.ms/ResetScanEngineVersion 

• Before running the script, change the execution policy for PowerShell scripts by running Set-ExecutionPolicy -ExecutionPolicy RemoteSigned.

• Run the script on each Exchange mailbox server that downloads antimalware updates in your organization (use elevated Exchange Management Shell).

Edge Transport servers are unaffected by this issue. You can run this script on multiple servers in parallel. After the script has completed, you will see the following output:

[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>.\Reset-ScanEngineVersion.ps1
EXCH1 Stopping services...
EXCH1 Removing Microsoft engine folder...
EXCH1 Emptying metadata folder...
EXCH1 Starting services...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Exchange Transport (MSExchangeTransport)' to start...
EXCH1 Starting engine update...
Running as EXCH1-DOM\Administrator.
--------
Connecting to EXCH1.CONTOSO.com.
Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate
--------
[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>Get-EngineUpdateInformation

Engine                : Microsoft
LastChecked       : 01/01/2022 08:58:22 PM -08:00
LastUpdated        : 01/01/2022 08:58:31 PM -08:00
EngineVersion         : 1.1.18800.4
SignatureVersion      : 1.355.1227.0
SignatureDateTime     : 01/01/2022 03:29:06 AM -08:00
UpdateVersion         : 2112330001
UpdateStatus          : UpdateAttemptSuccessful

Using the Manual Solution

In lieu of using the script, customers can also manually perform steps to resolve the issue and restore service. To manually resolve this issue, you must perform the following steps on each Exchange mailbox server in your organization that downloads antimalware updates. Edge Transport servers are unaffected by this issue.

Remove existing engine and metadata
1. Stop the Microsoft Filtering Management service.  When prompted to also stop the Microsoft Exchange Transport service, click Yes.
2. Use Task Manager to ensure that updateservice.exe is not running.
3. Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
4. Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.

Update to latest engine
1. Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
2. Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.

Verify engine update info
1. In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
2. Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001.

After updating the engine, we also recommend that you verify that mail flow is working and that FIPFS error events are not present in the Application event log.

FAQ

Is the solution for this problem automated?
Implementation of the solution requires customer actions, and it will take some time to make the necessary changes, download the updated files, and clear the transport queues. Actions can be automated with the scan engine reset script from https://aka.ms/ResetScanEngineVersion or they can be performed manually. Whether you perform the steps automatically or manually, they must be performed on every on-premises Exchange 2016 and Exchange 2019 server in your organization. If you use the automated script, you can run it on multiple servers in parallel.

How long will running of automated script take?
Depending on the size of your organization, the script might take some time to run; please be patient.

How long will it take to clear up the queues after the script has been run?
Depending on the number of messages that were queued up and the amount of new messages transport has to process, the time might vary. Please be patient and monitor those queues are draining (number of messages are decreasing) by using Get-queue command.

We are in Exchange Hybrid environment. What do we need to do?
If you are using your on-premises Exchange server to send email (for example using Centralized Mailflow or sending messages from on-premises devices), please follow this blog post and use the script to change configuration on your on-premises servers used for email transport. If you are using Exchange on-premises only for management of Exchange recipients, you do not need to take any action.

What are the services that the script is stopping?
The following services will be restarted: Microsoft Filtering Management and Microsoft Exchange Transport.

We have temporarily disabled Antimalware. Should it be enabled after following this blog post?
If you have temporarily disabled the antimalware service, you should enable it after you have followed this blog post (use the Enable-AntimalwareScanning.ps1 script). The solution described in this post is a full solution for this problem and will result in transport queues clearing and Antimalware engine working as expected.

The version of the updated scan engine starts with 2112330001; is this right? Should we be concerned that it seems to reference a date that does not exist?
The newly updated scanning engine is fully supported by Microsoft. While we need to work on this sequence longer term, the scanning engine version was not rolled back, rather it was rolled forward into this new sequence. The scanning engine will continue to receive updates in this new sequence.

What if my Exchange servers do not have access to the Internet?
If your Exchange mailbox servers do not download antimalware updates from the Internet, you do not need to perform any manual action. In that case, the servers have not been downloading antimalware updates to begin with, and the problem described here will not exist.