Expert User Acquisition Secrets from DraftKings, N3TWORK & Game of Whales

In collaboration with Silicon Valley Bank, we at Pollen VC put together an event - “Finance & Growth in the App Economy”. Here is a recording of the user acquisition panel with experts from N3TWORK, DraftKings and Game of Whales to share their top UA secrets.

You are about to discover the tactics that top mobile user acquisition experts are using for creative optimization, coming up with out of the box ideas for creatives, and what downstream metrics are the most important.

Thanks to Joseph Kim, Partner & Podcast Producer at Deconstructor of Fun, for moderating this panel.

Special thanks to the panelists for sharing their knowledge:

• Jonathan Lau, Growth at DraftKings
• Nebojsa Radovic, User Acquisition Platform at N3TWORK
• Doron Kagan is the Co-Founder & CEO at Game of Whales

Watch the discussion

Is creative optimization the biggest source of competitive advantage?

Joseph Kim, moderator: In the last few years there’s been this advent of machine learning-based bidding algorithms from Facebook and Google, and so there’s a fairly popular view that given all the automation and optimization that’s happening that the biggest source of competitive advantage now is creatives. I wanted to ask you guys do you agree or disagree with that proposition?

Nebojsa Radovic: Yes that’s exactly right. There are simply fewer levers to pull with UA requiring you to be more clever with the available levers. And creative is one of the most important ones.

However, creative has always been important, it’s just that now you have to over-index on that and try many different concepts. And the problem with creative being so important is that creative fatigue is becoming like a thing. We think more about and try to figure out how to extend the lifetime of a creative, but I think we might cover that in some of the next questions.

Jonathan Lau: With user acquisition, you really have 3 levers, one is the dollars (i.e. CPI and budget), second is the targeting and lastly is the creative. Realistically the creative is the single most important aspect of three. You can outspend the competition, everyone is targeting the same people, so what can you do to show the user that your value proposition is better than everybody else. I do believe creative is one of the key aspects of UA.

Doron Kagan: We all agree about the importance of what we show the users, however, I think that one of the biggest challenges for companies that are not in the size of these companies is the targeting because we all target the same users.

A small company of 30 people like mine who needs to compete with the likes of Disney, DraftKings and Machine Zone, can’t compete on budget no matter how good our art is.

We all have the same algorithms, so if we go to Facebook and we target in-app purchase whales they will eventually win given the size of their budget. We need to think creatively on the targeting and the ways to get to the users.

For smaller publishers, what aspect of the creative process is the most important?

Joseph Kim, moderator: Assuming that creative is the most important aspect of gaining a competitive advantage, then for a small company what aspect of the creative process should they be focusing on?

Nebojsa Radovic: That’s a great question and it’s one of the most common discussions I have with the marketing creative people.

You have to forget everything you know about creative because things you think are going to work are not going to work in most cases.

It’s all about testing different concepts even the really stupid ones because successful UA teams are testing all kinds of different concepts and trying to find which one is converting users the best.

You need to have a holistic view of how creatives perform especially for hard-core games and understand whether the users you are driving are actually good. Sometimes the click-through rate and click to install are not the best representatives of performance. There can be misleading creatives that lead to lower retention and lower ROI.

In short, install volume and cost are not the only things you should worry about.

Jonathan Lau: I think in marketing you should always build for your user and not for you.  

There have been plenty of times when the marketing team thinks a creative will perform well only to have the results be something completely different.

You have to forget everything that you want versus what your user actually wants.

Who is your user? What is enticing to them? What can create that linger effect, especially in a world where we are bombarded by ads? What will cause a user to see a banner and stop?

Taking that back to what your product is and understanding what can you do within your product to create that linger effect. You don’t want to go way off the wall and create this thing that’s super whacky that will create that linger effect but in no way ties back to your product. That’s where you would have a huge click-through rate, but really bad down funnel metrics.

If you’re a small company you don’t have the luxury of trying everything, so you have to be intelligent and really get in the shoes of the user.

Doron Kagan: Yes, I agree of course, but in order to win when you compete with unlimited budget companies, you must think out of the box.

For example, if you take the routine ad of showing the best out of your game it will never look as good at Disney or Marvel because they have better resources, better skills, and more money.

Thinking out of the box requires looking for the craziest things and trying something that hasn’t been seen before.

For example in our game, Trial Extreme, which is a motorcycle game we advertise the game in the obstacles, racing, and the PVP and performance was okay.

Then we tried one time to show fails, so people crashing into walls or into the ocean, and the CTRs and all other metrics went up the roof. While it's not the most presentative thing about the game, the ads made the users stop, click and more importantly convert.

In any campaign, you want a return on investment, and these users not only downloaded more, but they also played more in the game leading to a successful campaign.

How do you come up with out of the box ideas for creative optimization?

Joseph Kim, moderator: How do you guys come up with the ideas for creatives? Are there specific processes or tools that you guys are using to come up with the crazy out of the box type of ideas?

Nebojsa Radovic: Yes, we are using market intelligence tools like Sensor Tower and AppAnnie to see the ads that our competitors are serving, but we’re also trying to test memes, things that we see on TV and things that people talk about.

One thing that’s important to mention is that production value doesn’t really matter as much as the idea behind the ad.

It’s all about being able to create as many creatives as possible that are interesting, different, and that can catch people’s attention. It doesn’t matter how much you pay for the ads or how much time you spend building them.

Also, I take screenshots and record ads all the time to inspire people who work on building ads.

Jonathan Lau: I think one aspect of it is to avoid silos. I worked for companies where you have marketing teams that want to create something that’s very avant-garde for a slot app?

Ultimately, the creative person needs to understand where this ad is going to go, how is it different when you’re showing an ad on the newsfeed versus showing an ad within another game, or showing an ad on a banner placement?

Each one is going to give a very different type of user experience, so create an ad that is meant for that user experience. Often times companies with limited resources make the mistake of using a high-performing Facebook ad on other platforms and later finding out that the ad doesn’t perform as well on the other platforms.

Really have your creative team understand where this ad is going so that they can actually create something that is functional.

Doron Kagan: One of the advantages of a small company is that you don’t need to have processes, so we just encourage everybody to take part in creative ideas.

I think our last very successful video ad came from the most junior person in QA who just played with images from the game, put some very nice music that he found somewhere, added some effects and it turned out to be one of the most successful assets that we use for user acquisition.

This is very easy in a small company, but quite hard in a big company.

What is the top of the funnel metrics that UA managers should focus on?

Joseph Kim, moderator: Generally speaking, it seems like there is a focus on the top of the funnel metrics rather than tying to the downstream metrics. In terms of the top of the funnel metrics what are the types of metrics that people should be looking at (eg. CTR or IPM)? And in what cases should you be looking at those different metrics? Also, what are the types of companies or games should you be measuring downstream metrics for or should you always be doing that?

Nebojsa Radovic: During my career, I have mostly worked with mid to hardcore games, so it’s a little bit different for games that are more complex and that require users to play them for some time to understand the games.

Obviously, we care about IPM and CTR because they tell us the absolute reach that we can go after and the volume of installs to growing the game but at the same time, just installs without revenue don’t mean much to us.

That’s why when we have creatives that are a little bit different that perform significantly better on IPM or click-through rates, we have to look at metrics such as cost-per-retained user or cost-per-payer to understand whether all those installs are just noise that make the CPIs look great, but are not contributing to the top line - which is what we care about.

Front end metrics are important - click-through-rate, click to install, IPM - but unless those users are generating money for us, we don’t really care about those users or those installs.

What are the downstream metrics that you should look at when it can take a long time to figure out if a creative led to a higher LTV?

Joseph Kim, moderator: Maybe I could ask a follow on question, ideally you can tie a creative down to LTV but that’s going to take a long time for you to figure out. In terms of downstream metrics is there a Day X ROAS or ARPU or some other type of metric or is it just cost per payer that you guys look at?

Nebojsa Radovic: All of the above. Ultimately we care about payback windows, and all the other metrics are just proxies for that.

We look at Day X metrics to a certain extent, but we’re really trying to estimate what the payback window is going to be and we try to do that at the most granular level possible.

For example, if a certain creative is so good that it drove 1,000s of users then we can estimate the LTV for those users and make adjustments based on just that creative. The problem is if you spend $5,000 on a creative that doesn’t drive installs or doesn’t drive any payers then it’s much harder to kind of figure out how that creative impacts the business.

Joseph Kim, moderator: But for that feedback loop you’ve got a creative and then you understand the payback window. However, some companies have a 3-year payback window so then do you have a PLTV (predictive LTV)?

Nebojsa Radovic: Yes exactly. That’s what my team is working on. Based on the most recent data, we want to understand the performance of a certain creative and whether the creatives should remain alive or should we take them down.

That’s the point of the UA platform to automate all those things so you don’t have to manually change things and also minimize wastage by having computers do that and not people. 

Jonathan Lau: When you’re thinking about downstream metrics especially for core games that makes a lot of sense.

However, if you’re talking about a game that is hyper-casual and very ad monetization driven, it might not make sense for you to look at a Day X ROI. You might want to maximize click-through-rate and drive the lowest CPIs possible to get those users in, because if you have a very low ARPU but you make it up in volume then it’s a completely different type of buying strategy.

Regarding any product, whether it’s gaming or non-gaming, you really have to think about it from a full-funnel standpoint.

Traditionally the trap is marketing teams think that our job is to bring users in and once you users come in it’s no longer our problem and the burden falls on the product team to figure out a way to monetize them. I would argue that a professional UA person should think all the way through the funnel from when they bring that user in and what is the first time user experience. How likely is that user going to drop off if they see a really lousy tutorial or lousy first-time user experience?

Often times we don’t think about the value of a dollar saved versus a dollar spent.

If I can bring a user in and I can retain that user or reduce my Day 1 churn, what is the business dollar amount that drives to my portfolio versus spending another $10,000.

Doron Kagan: The front end is the first tier, and an ad that doesn’t have clicks or installs will not work. You need to measure everything up to the LTV.

Breaking it down not only to the art but also to the targeting, and then all the way to the LTV. Even if the game takes a few months or years to produce the LTV, there are tools in the market today that offer predictive LTV where you can quickly track if the campaign that you are running and the art that you’re using are driving good ROI or bad ROI.

There’s really no need to wait.

Jonathan Lau: One thing I’d like to add. Let’s say you’re a start-up and don’t have the luxury to build a predictive LTV engine that requires a lot of resources, time, and money that you may not have.

Then I would say go back to the fundamentals and examine the user flow. When I was at Play Studios, we were spending all this money creating lookalike lists of our payers and spending huge CPIs to acquire these users. However, when they play the game, the first thing they see is a tutorial on how to play slots.

These are people who probably spend thousands, tens of thousands of dollars on other slot games, and the first thing they see is a slot tutorial and our Day 1 retention was horrible. Once we changed the FTUE, we saw a huge increase in our Day 1 retention.

That’s why if you’re a start-up and don’t have the luxury to build these massive machine learning algorithms, focus on the flow, focus on what you can control, and focus on the things that you can actually work on.

For a smaller team with limited resources, what advice would you give them on how they spend their time and resources optimizing creatives?

Joseph Kim, moderator: Expanding on that advice, for the smaller game studios with limited resources, what advice would you give them to help get the biggest bang for their buck in terms of how they spend their time and resources optimizing creatives?

Nebojsa Radovic: Yes I work with a few smaller developers and it’s definitely challenging.

What’s happening in the industry is Google and Facebook are working against us and against smaller developers because if you don’t have a lot of money it’s hard to compete.

I would focus on smaller channels maybe work with your networks and try to nail the video creative because that’s one of the ways to lower cost and grow the game initially. Once you prove that the game is working, then it’s about retaining users and monetizing those users through Facebook, Google and other reward video networks. After that, you can expand and start buying more aggressively.

Unfortunately on Facebook, when you do value-based optimization or app event optimization, it becomes too expensive and if your LTVs don’t support it, then it’s hard to scale on a small budget.

It’s better to find a small niche, work with networks that are easier to optimize and cheaper to like acquire from, and then expand to more expensive and more complex networks.  

Jonathan Lau: Another thing to consider is that when you don’t have a lot of budget every dollar you spend, aside from acquiring a user what did you learn from that dollar spent?

You could spend a bunch of money with Google Ads but it’s a black box. You give Google a few creatives, tell it what the segments and then they’ll produce results but you don’t know why it worked or why it didn’t work.

When you’re spending money on acquisition, think about what channels you can spend on where even if it doesn’t work you can walk away with a wealth of learnings that you can apply to the next channel.

Doron Kagan: Well, I would take the money, and put it on AB testing before I do anything else and spend one dollar on user acquisition.

For game publishers and developers, the most important tool in the toolbox is AB testing. However, the ability to really test everything is tricky, because there are so many AB testing tools out there:

1. It’s hard to implement.
2. You need to find something that is reliable when you get the results.

A good tool allows you to set up an easy test and get a black and white answer so that you keep increasing your LTV and that’s what you really need for user acquisition.  

You need LTV that can compete with the CPIs that everybody is paying, so if you can generate a high LTV, then you can start competing and you can start spending money. The second goal after increasing the LTV is to spend money, as long as you can buy users cheaper than your LTV, then you can spend as many millions as you can in a month.

How to A/B test at the game development or post-release state

Audience Question: A question about the last comment that you made, I’m actually kind of curious, what about AB testing at the game development or post-release state? How do you train your engineers to think from that perspective and not just your marketing folks?

Doron Kagan: Warning, there would be an advertisement in my speech now. We released our last game 7 weeks ago, it’s called Duels, go ahead to the app store and download it (iOS | Google Play). That was not the advertisement, it will come.

This game was in soft launch for 13 and a half months and during this 13 and a half months, we AB tested more than 100 tests, and the tests were not on marketing, the marketing tests are happening now, but it was all about LTV.

We tested everything that you can imagine and can’t imagine. Some tests were ridiculous, and some we didn’t expect to get the results that we got, but and here’s the advertisement, what we built in the other company, Game of Whales, is basically the ability to execute tests fast and get a black and white answer. If we want to know if a tutorial is good, or do we need a longer tutorial or no tutorial at all, or tutorial with ads or tutorial with in-app purchase offer, so we just put 7 groups, and we get an answer no tutorial, next.

We don’t even stop to think about it, that it doesn’t make sense, not to have a tutorial in the game. It is what it is. These are the numbers, so next and then after you have LTV that you are happy with, this is the time that we are starting to test the marketing stuff.

Jonathan Lau: I will also add that when you guys are building out the game really think about how you are going to do the segmentation.

I’ve been with companies where they try to do feature rollouts to 10% and then realize internal game segmentation was broken right. They thought that they were only targeting 10% of users when they were actually exposing 50% of their users to that feature.

When you’re already designing the game, have segmentation in mind as you’re building the infrastructure of the game, so that it can give you easy access to how you are going to A/B test and control who gets to see what.

And have that mindset in place of am I going to do a 10% rollout and when you do a 20% rollout where am I going to do the test. If I can go back in time and talk to some of my old companies when they first started, I would tell them to think about these things as you’re creating a game and not doing it after the fact.

After the fact it’s going to be very hard for engineering to go back to change some of the things that you have to change and unmake those things.