Summary

The orchestration engine, which powers the delivery of campaigns in the new Studio, is designed to reduce noise, increase engagement, and prioritize notifications to drive action. To do this, it is constantly weighing multiple variables in order to determine the best way to deliver a given campaign to a user, and to determine if a delivery is even necessary.

We provide real-time feedback about the overall effectiveness of your campaign through the campaign-level scores, performance, and fatigue. You can improve these scores by altering your priority, duration, and other settings on the Orchestrate tab.

Engine Settings

Optimize

The Optimize engine plans notifications for you. The engine will determine if, when, and how to reach a user via notification. The individualized notification plan is based on campaign configuration (especially priority and duration), a user’s current campaign queue, and the user’s history of engagement. In the absence of data about a single user, we’ll rely on community and platform history of engagement. For each member of the target audience, the Optimize engine determines if a notification is needed, and then the best time and channel to reach them. It also determines the best time and channel to re-engage with the user (if the user has not already seen the content).

If an individual user has many campaigns competing for their attention, to minimize fatigue the engine will send notifications for the highest scoring campaigns (priority, recently published, etc.) and some campaigns may not be sent as a notification and instead rely on the user seeing them in the member experience.

To leverage the full power of the Orchestration Engine, we recommend keeping the engine selection set to Optimize and leaving a long duration (say 7+ days). When enabled, the optimization setting will evaluate all the possible campaign plans for an individual within the configured duration. Each possible plan is given a score, and we implement the highest-scoring plan per user.

Hybrid

The Hybrid engine mixes the function of Optimize and Override. By selecting Hybrid, you control the first notification (all users receive a notification at publish time). Re-engagement is then determined by the Orchestration Engine based on the same variables as Optimize - has the user seen the content, what other campaigns are competing for their attention, etc.

Override

You can use the Override engine to schedule the date and time of a communication. Override will not attempt to re-engage with the user and a duration cannot be selected, as this is a one-time communication attempt. If a member of the audience chooses not to view the campaign, the engine will not attempt to engage them again. With Override, if you select one channel (email, assistant, or push notification), the engine will only publish to the audience via the selected channel. If multiple channels are selected, the engine will determine the best channel to reach each member of the audience on. Since Override overrides campaign optimization, it should be used sparingly in order to allow other campaigns to optimize efficiently. 

How is a Plan Score Calculated

Because each notification delivery plan is individualized, different users may see notifications in different channels, at different times, on different days, and receive a different number of follow-up reminders. This is exactly what you want - to reach your users when and how they prefer to be engaged.

How, exactly, do we score campaign plans? We score all possible campaign plans based on cost, value, and likelihood of engagement:

• Cost - Some channels are more disruptive than others, so we weigh them in this order of low to high cost: Assistant, email, push. Also, on a given day, each additional communication is more expensive. To account for the increased expense of multiple notifications, cost takes into account how many other campaigns are already scheduled for the user to receive.
• Value - We look at both the priority and how new the campaign is - we assume it’s more important to receive a notification shortly after publishing. Value is impacted by your campaign configuration.
• Likelihood - We weigh the likelihood of an individual user engaging via Assistant, email, or push at any given hour of the day. In the absence of individual data, we make assumptions based on community or platform history.

Note: It takes about 5 days of solid engagement data to really personalize the likelihood score for a user. This data is evaluated week by week, so if habits change (say, during travel), it could take up to a week to shift.

How Does the Optimize Engine Determine How to Reach a User?

For an Optimize campaign with ALL channels enabled, specifically, the engine weighs all of the following variables when determining how, and if, to deliver campaigns: 

• Best time to send - When is the recipient most likely to engage, based on past behavior or community behavior (in the absence of reliable past behavior)?
• Best channel - What’s the best place (i.e. email, push, Assistant, etc.) to reach the recipient, based on past behavior and the importance of this communication?
• Priority - What is the importance of this campaign compared to other campaigns targeted to this recipient?
• Duration - How much time does the engine have to deliver this campaign?
• Campaign queue - How many other campaigns are already targeted to the recipient in the duration window? Are they already receiving enough notifications to draw them into the member experience, and this campaign doesn't have a high enough score to qualify for a notification?

Since the orchestration engine generates a delivery plan for the campaign at the time of publish for each individual user in the audience, the ‘predictive’ plan outlined on the Orchestrate and Review page can and will change, both pre-publish and post-publish of the campaign.

For example, if last week you created and planned out three communications for this week, publishing on Monday, the plans for your scheduled/published campaigns will be different on Monday than they were last week, especially if on Monday an unplanned critical update must be published to the same users. The engine must account for the critical update when adjusting the plans for your three communications, to drive toward the right outcomes for all campaigns, and protect end users from overload.  

Based on the ever-changing nature of the communication queue for any user at any time, the orchestration engine aims to strike the balance between reaching users with the most important information targeted to them, while still protecting their time and attention. 

This implies that the more freedom of choice the orchestration engine has in terms of channels and time via the duration of the campaign, the more options it has to strike that balance of right information and right amount of attention demand. 

Limiting the choices of channels, or selecting a short duration, reduces the engine’s options and window of opportunity to reach your target audience. The purpose of publishing any communication is to have it be read and acted upon by those who need to know the information, and to make that information available to the right users. Allowing every communication to exist in either a topic(s) or the For You section is very important for this purpose, because then every campaign can still be discovered by a user in the web or mobile experience, even when the engine may determine that it is not valuable for the user to also receive a delivery via email, push, or Assistant. This scenario can occur when a campaign has priority, channel, and duration settings that make it so the engine prioritizes delivery of other campaigns first. Even without guaranteeing a delivery of the campaign, the user still has the opportunity to view the content on their own without a delivery to prompt them to do so. 

Given all of the above, we generally recommend the following for the best possible campaign outcomes under Optimize:

• Choose a priority for the campaign that adequately matches the importance of the information and the importance of a delivery being made to the audience.
  • Note: 'Nice to Know' is unlikely to result in notifications, unless the individual has a very light campaign queue and the campaign has a long duration. Only use this setting if the post is truly “nice to know” and notifications are not needed for all members of the audience.
    
• Choose a duration that is longer than or equal to the duration suggestion on the Review page. Nice to Know and Should Know campaigns should generally have longer durations so that the engine has more flexibility in planning initial deliveries and re-engagement attempts around higher priority campaigns and Hybrid and Override campaigns
• Allow the orchestration engine to choose the best channel(s) to leverage per user for your campaign, i.e. if possible, do not restrict the channels for the campaign
  • Not all users in your audience may be reachable or highly engaged on one channel versus another. Limiting available channels could mean that the engine chooses not to deliver your campaign in favor of other competing comms with settings that warrant more intrusive channels. 
• Allow the campaign content to exist in a topic(s) or in For You

Re-engagement

When set to Optimize or Hybrid, the Orchestration engine will attempt re-engagement with users within the assigned Duration. Re-engagement attempts will end once the user has seen the campaign (through an email open, push open, or viewing the campaign directly in a feed).

Note, the re-engagement plan for an individual user is re-evaluated frequently during the campaign duration. Publishing a campaign sets the plan into motion. Once a campaign is scheduled and published, a replan will be triggered every 24 hours, at minimum. Additional events can trigger a replan (see below). Replanning ends when the campaign duration ends.

Events That Trigger a Replan

1. Whether or not the user opens/engages with the campaign pre-delivery (email, push, Assistant) via the topic or in For You
2. An update is made to the campaign itself (everything from the content to the settings)
3. New competing campaigns are published
4. Any competing campaigns are updated
5. The user engages with any competing campaigns
   

What Do Performance and Fatigue Mean?

Performance

Performance is a measure of the overall likelihood of a particular audience engaging given a particular campaign configuration. The higher the performance score the more people we predict will engage with your message. There's always a chance the end of campaign metrics will show better or worse performance than the score initially indicated.

Aim for a high-performance score. The Orchestration Engine in Optimize mode will maximize performance for a campaign by selecting the best possible plan for each user given the campaign settings. If you select Hybrid or Override, you may see performance drop as you customize the settings. For example, if a sizable chunk of the audience has a higher likelihood of opening push notifications than emails and you turn off push notifications as a possible channel, the performance score will likely drop.

Fatigue

Fatigue is a measure of information overload. Fatigue averages the cost of a campaign and takes into account reachability. Some audiences are harder to get in touch with than others. We determine this via a reachability score based on how many users have an email, have downloaded the app, etc. If a user is reachable via one channel only, this makes it far more likely that they feel overwhelmed by notifications on that channel.

Aim for a low fatigue score. The Orchestration Engine in Optimize mode will minimize noise for a campaign by selecting the best possible plan for each user given the campaign settings. If you select Hybrid or Override, you may see fatigue increase as you customize the settings. If you narrow down the delivery time and target channels, you may increase the fatigue of your campaign by colliding with other scheduled campaigns or by preventing the engine from potentially choosing a less costly channel, even if it's more efficient for a particular user.

Note: We take a small sample of the audience in order to generate the channel insights.

How Does the Orchestration Engine Work with Email-Only?

In general, for email-only, there is no change to how the Orchestration engine works, only a restriction of which channels it evaluates as ways to reach the member. However, to ensure email delivery, we recommend that email-only customers always use Hybrid or Override.

Frequently Asked Questions

Where can I see the orchestrated campaigns?

On the Orchestrate tab, there is a hover to explain competing campaigns.

Why do some users receive multiple emails at one time?

Delivering at the same time prioritizes our goal of engaging users for “2 minutes a day”. You can also think of this as minimizing randomization and enabling focused work.

Note: We have considered the value of “batching” these emails into one email. However, if we did batch the emails, we would have to impose our own template to the content - likely having to link to the full content. You would not get the ‘what you see is what you get’ experience of composing the email in Studio and the recipient seeing exactly what you created.

What do I do if I have time-sensitive content?

You can use the duration setting to mark the end of when the campaign would be relevant, and you can schedule archiving if the campaign truly needs to be stopped at a certain date and time.

What do I do if I have an urgent communication that MUST go out via email? (or other channel)

If you need to know that a particular communication has occurred at a specific time, then you should use the Override or Hybrid setting. Just navigate to Orchestrate > Rules and choose Override or Hybrid. Then you can select or deselect the targeted channels as necessary. Reminder: Using Override means that the system will not send re-notifications. 

Note: keeping the engine set to Optimize while setting a campaign to Critical with short duration does NOT guarantee that communications will go out at a certain time to the recipients. Once Optimize is engaged, the engine determines whether and when to send a notification based on the recipient's queue. If there are a lot of "critical" communications competing for attention, some may not qualify for a notification to that particular user.

How do I cap re-notifications? Why do some posts get a lot of re-attempts and others get very few?

We re-attempt notifications if the user has not seen the content. This is true of all campaigns, including ones marked as needing “acknowledge”.

Regarding how many re-attempts, that number is based on both the campaign value and their personal queue. High-priority campaigns get more notifications than low-priority campaigns. Quiet weeks for the user may see more re-notifications than busy weeks when there is a lot of new content competing for “first notification”.