We’re currently in the final round of testing of the next major release of Event Espresso, 4.9. In this release, there was a significant refactor of the messages system so this post is intended to give a heads up about all the changes coming.

[notification type=”alert-info” close=”false” ]This post gives an overview of the changes to the messages system.  For more details you can read an Overview of the Messages System and a Code Flow diagram.[/notification]

Introduction of a Message Queue System

This new system tracks and prioritizes when messages are generated and when they are sent. When messages are triggered, they are no longer generated immediately and sent on the same request. Instead, they enter into the queue and all processing happens on separate requests.

Most messages are thus persisted to the database because of the queueing system. Some still aren’t if they are browser/pdf specific (i.e. invoice/receipt/tickets) but this could change in future iterations.

This new system resulted in a rewrite of much of the controllers and business logic of the messages system which of course introduced a number of new classes. One of the notable changes is the introduction of the EE_Message entity which replaces the old way of passing around a stdClass object to represent a generated message. You can read more about this and the other classes introduced in the new documentation

Changes to Triggers and Message Priority

In the messages system, a trigger is a way of referring to something that kicks off the message process. That could be when the checkout has approved a registration or when a user clicks a special link in the list table (i.e. resent message link found in the Registrations List Table).

The new system makes no changes to how existing triggers function in terms of expected results, however there are significant changes to how a trigger is processed.

All messages that are not for messengers flagged “send now” start off as a “MIC” status message object and are “queued” for generation, then saved to the db.

In the more detailed documentation, we outline all the various stati for the EE_Message object, basically every message starts off as MIC (or incomplete) except for those flagged “send now”.

The system has a method for messengers to indicate that the message is to be generated and sent immediately on the same request. messengers that currently set this flag to true are html and pdf messengers as we want those mesengers to trigger immediately.

If a message is not for a send_now messenger, then the next thing the messages look for is the priority on the message type. Currently there are three possible priorities a message type can have:

• EEM_Message::priority_high: indicates a message should be generated and sent as soon as possible.
• EEM_Message::priority_medium: indicates a message should be generated as soon as possible but can be queued for sending.
• EEM_Message::priority_low: indicates a message should be queued for generating.

Currently, all payment message types are EEM_Message::high_priority. All registration message types are EEM_Message::medium_priority and the newsletter message type is the only current EEM_Message::low_priority message type.

The important thing to remember about message type priorities is that everything happens on a separate request, the priority just indicates how SOON it happens on a separate request.

So that means if a message type is EEM_Message::priority_high then the messages system will save the MIC message (with the bare amount of info needed for generation) and will initiate a separate non-blocking request to begin the generation. Thus it will be generated immediately but on a separate request from the trigger. Then on that separate request, the priority check indicates that another non-blocking request is to be done to SEND that message immediately. The end result is that the end user that triggered the message should see EEM_Message::priority_high messages show up fairly rapidly (with caveats, you’ll see that in the next point about scheduling).

If a message type is EEM_Message::priority_low then that means that the MIC EE_Message is saved to the db, however there is no separate request initiated and it’s just left there for the next batch schedule to kick in… which brings me to the next point.

Batch Schedules

The new messages system sets up two wp-cron schedules on activation. Both schedules are currently set to a 10 minute interval. However, this can be changed using a filter:

The first cron schedule will trigger a query to retrieve all non-generated messages sorted by priority up to a set “batch” amount (which currently is at 50). Then the messages system will generate those messages change the status to MID (Idle, ready for sending), save them, and depending on the priority level, either initiate another separate request immediately to start a batch send OR just exit leaving them queued for the next batch send schedule to fire.

The second cron schedule will trigger a query to retrieve all non-sent messages (ordered by priority) then proceed with sending them with the appropriate messenger. Currently this schedule will do 50 messages at a time (it’s filterable, and something we can adjust if needed).

The important thing to remember about how these schedules work is this:

• they don’t do ALL messages for the given type, they only do a set limit (batch limit) for each scheduled request. This limit is filterable.
• While a schedule is executing, there is a lock set (for that specific schedule, generation or sending) to prevent any other scheduled message requests from executing while processing is happening. This means if a batch generation request is running, then any other incoming requests for batch generations will be prevented from executing due to the lock. The locks themselves have an expiry set (defaults to one hour, but is filterable), so if something goes wrong and the lock isn’t removed (server crash or something), then this prevents a complete lockout.
• The above is an important point, because this means that although when a EEM_Message::high_priority message is triggered, it immediately initiates a request to generate/send IF there is a lock due to a scheduled batch request running, then it will not get generated/sent and instead wait until the next batch. The opposite is true as well, if there is a request initiated right away and it is running, and then a scheduled batch request fires, it will not complete because of the lock and will resume on the next firing. What’s important to remember though is that if a EEM_Message::high_priority message triggers an immediate generation/sending on a separate request, and there is no current lock, it executes the same BATCH process as a regular schedule. So that means that up to the batch limit will be processed for that request.
• Batches retrieved from the db are always ordered by priority. That means higher priority messages will always be generated/sent out before lower priority messages.
• The batch send method not only has a limit on the batch retrieved from the db, but also has a rate limit for how many can be sent during a one hour time period and this rate limit currently applies to all messengers (in a future iteration I’m likely going to have it set per messenger). Currently the rate limit is 200/hour. That means that in any given hour period, the maximum messages that will get sent is 200/hour and if that limit is reached, then no messages will get sent until the next hour period hits. This number was arrived at from researching what a number of web hosts have set as their email sending rate limits and set a number that is conservative for the average. Keep in mind this is filterable so people using a really good web host (or an email service like mandrill) can up this rate limit if needed. I think for most average users tho, 200/hour is plenty.

Messages Admin changes

Message Activity

The messages refactor brings a brand new list table that displays all saved messages and their status. Currently, the only time you will see html or pdf messenger related messengers in this list table is if there is an error (resulting in MFL status) so for the most part you will only see ‘email’ messenger messages (this will likely change in a future iteration though because we may find we want to save generated invoices/receipts)

The Message Activity List table serves the following purposes:

• as an archive of all messages that are in various stages (ready to be generated, all the way to sent).
• failed messages show up here and the system makes every attempt to save a useful error message with the failed message to assist with troubleshooting why message generation/sending failed.
• admins can trigger immediate generation and/or sending via this list table.
• admins can resend specific messages from this list table.
• the table is used for filtered results by registration, transaction, event, contacts etc. This helps admin’s answer questions like, “What are all the messages that have been sent for John Smith?”

Please note:

Each row in the message activity list table has the status indicated via the colored strip in the leftmost column. This will help you understand what stage a message is at. Here’s a screenshot of the legend that shows what each color represents. You can also get the status by hovering your cursor over the status column.

Admin message trigger links

The following behaviour should be expected:

• All existing trigger links (i.e. resend registration message from registration list table) should work as they currently do and result in a new generated message. The only thing that changes is that the message will not be generated/sent on the same request and follows the new priority rules (as mentioned earlier). I also changed the success messages that show after executing that trigger. You can see the status of messages sent this way via the new Message Activity list table.
• Currently on the transaction list table, registration list table, and event list table there is a new action for each row for viewing all messages related to the record the action was triggered from:

The above is an example from the transaction list table. When you click the “megaphone icon”, it will take you to the Message Activity Table but only showing the messages related to the transaction row you clicked the link in. If you are clicking from a registration list table, then the resulting messages shown in the messages activity list table will be related to the registration you clicked, if from the event list table then the messages activity list table will show all messages sent related to the event row you clicked from.

For resending already sent messages without regenerating the message, that will always be done from a Message Activity list table. Currently the only place that table is found is on the Messages Admin Page route, but in other tickets I’ll be adding a list table to the contact details, registration details, and transaction details pages. That way event admins can resend already generated message(s) for those contexts at will.