Experimenting with Push Notifications for User Retention: Tested on a Million-Strong Audience

What is more critical in a push notification — the text or the picture? How does it depend on the mobile platform? When can sending push messages more frequently get better results? What mistakes can be made while you’re figuring it all out? Learn about it in this piece by the team developing the push notification system for FunCorp entertainment apps.
 

We do our best to deliver memes and funny videos directly to users. And push notifications to remain an effective channel for returning users to iFunny and other products of our team.

Here’s an example: users who have disabled push notifications (blue curve) continue to use the app, but their retention rate after a month is 30% lower than those who haven’t done it (green curve).

We’ve got millions of active users around the globe, and we’ve accumulated a wealth of experience with notifications. Here are some of our most striking and memorable insights and experiments.

When and how often should push notifications be sent?

Some say that the best time to ping the user is in the evening; others argue it’s the morning. We decided not to rely on someone else’s opinion but instead to do an experiment.

We decided to send push notifications throughout the daily activity of our audience; in recent days, notifications showcasing the best content were going out as often as seven times a day. It may seem this is too much; however, many users open the app several times a day, so we took a chance and bet on frequent reminders.

The number of notifications and the sending time were simply an arbitrary starting point. However, subsequent experiments with reducing the frequency of push notifications turned out to be reducing retention rate and viewing depth.

We decided on the number of push notifications and stuck to this figure for a while. Here’s what happened to the retention rate when the number of notifications was reduced:

The green curve is the group of users who received seven daily push notifications. The blue one is those who received five push notifications per day.

We stuck to our magic number for a long time. Then, one day we decided to pull a trick we saw at our competitors: display each push separately in the notification center instead of bundling them. This way, the content of each one was visible, and no notification would go unnoticed. The users who’d been busy during the day would find a whole collection of memes on their phones by the evening.

It proved to be great for increasing CTR! However, when we decided to expand the test cohort (we test everything, remember?), we were hit by notification disabling at a record rate. We shouldn’t be sending push notifications so often to everyone, after all.

So we decided to segment the audience and reduce the number of notifications sent segment by segment. We tried to reduce the frequency of push notifications for those who never tapped on them but saw a decrease in DAU. Despite not tapping on the messages, such users reacted and simply opened the app.

Then, we tried to split users by activity. However, the inactive ones started forgetting about the app altogether without seeing push notifications. The highly active ones would get so used to seeing the icon in the notification center that they’d also need clarification when they didn’t. Apparently, with these frequent push notifications, we made ourselves a problem. Then, we reminded ourselves that push notifications are what help bring back lost users, not a shortcut to the app. Then, we found a solution.

The trigger for sending a push notification was a missed visit to the app: most of our audience look at memes at breakfast, lunch and dinner — it’s easy to track. You opened the app? Good job! We don’t bother you, then. Didn’t open it at dinner? Ding dong!

This worked. 25% fewer push notifications, 10% fewer unsubscribes.

Evolution of content selection for push notifications

User choice

Back in the beginning, we’d simply send the content with the most likes in the app. However, only 1 out of 10 push notifications with content that users were guaranteed to like would do the job:

With this approach, CTR looked like this.

It didn’t take us long to find the answer: the meme in the push notification often looked nothing like its full-screen version. It was often hard to see humor in 70 characters and a tiny picture; the teaser did not always turn out well either (later on, we fixed this issue using the picture in an image search algorithm).

Then, we decided to set up separate content competitions. We selected 10% of our audience that was the most active in reacting to push notifications (it amounted to tens of thousands of users) and split them into 30 equally-sized groups. Each group received the content with the most likes in the last few hours. The rest of the audience received the push notification that scored the highest CTR based on test results across the 30 groups.

Our CTR skyrocketed, but very soon, the innovation got us into trouble.

The problem was that most clickthroughs were being scored by… clickbait. That is, the push notifications subverted the users’ expectations. Sure, it’d work once, as curiosity got the upper hand, and the user tapped on the notification. However, a steady flow of subverted expectations would repel the user; people began to turn off push notifications more often.

Team’s choice

By looking at the content in the push competitions, everyone on the team could tell whether it was clickbait. For a while, product managers would manually select non-clickbait push notifications among the winners, but two problems arose:

• Seven notifications per day, 49 per week: we had to check out at least 1,000 memes to send this many.
• Everyone had their own opinion of what was funny and what was not, and beliefs often didn’t match.

Even the idea of handing the push notification selection over to the editorial team didn’t solve the issue. Disputes on content selection were now raging inside it. The rest of the group kept coming at them with comments like “Hey, that’s not funny!” and hints at some “proper funny content.”

At one point, we decided to collect the best push notifications of the past years and send them out again. That worked!

Then, we tried this again. And again. Alas, our happy days didn’t last long. The best push notifications of the past years lost their effectiveness over time; users began to get annoyed with repetitive content, lashing out at us in the comments.

Well, of course, our choice was a matter of taste. To tackle the issue, we decided to trust the algorithms. They also helped us solve the clickbait problem to reduce the number of unsubscribes.

Algorithm’s choice

We revisited the idea of content competitions, this time without clickbait, which meant we needed a solution to recognize it. We used assessors to mark up several thousand examples and trained an ML model to pinpoint such content and exclude it from the competition.

Sometimes, memes completely incomprehensible to us would end up working; but CTR grew again.

After delegating content selection to algorithms, we focused on testing the push notification design.

Picture or text — which is more important?

Twitter sends the text together with the author’s userpic. Instagram sends plain text notifications. Why is that? We decided to test the effect of pictures and text in our push notifications via a degrading A/B experiment:

• Group A was sent text and a picture.
• Group B was sent only the text.
• Group C was sent only the picture.

Push notification without a picture (text only)

Android

iOS

This yielded ambiguous results. On iOS, a good text without a picture resulted in more taps than just a picture. For Android users, the text had a lower value (± 10%) than for iOS users (± 20%).

Push notification without text (picture only)

Android

iOS

On Android, push notifications without a picture always got the worst CTR. On iOS, however, a picture would even worsen the CTR in some cases, although it remained important for many users.

P.S. Evolution of success metrics in push notifications

Retention Rate is important for products that are used daily. However, minor changes in certain parts may not be noticeable on the general charts. Therefore, our key metric changed several times during the work with push notifications.

1. If you look at notifications as a separate product, interest in their content is what constitutes its value to users. Therefore, it was important for us to find a plateau on the CTR charts in the beginning. This meant we had found long-term value.
2. When optimizing CTR, we ran into the issue of clickbait, which means subverted user expectations. Therefore, the next metric we paid attention to was disabling notifications and deleting the app, caused by negative reaction towards push notifications. We started monitoring this indicator constantly, using it as a proxy metric for CTR. This helped us avoid userbase burnout and the critical outflow.
3. Then, we focused on working with outflow only. Keep in mind that a push notification is not a shortcut to your app. Frequent notifications cause users to disable them, and then, the users themselves are lost. Today, I recommend using push notifications when the users forget to come back on their own accord.