AMA: Charlie Olson - A Developer Who Wrote Call of Duty's SBMM

[c0lson]

If SBMM is so good why has many people quit cod over it and why does it feel like its been ramped up to level 1000 over the last 6 years when cod titles of the past have done just fine without it, using team balancing when ping was actually KING and it took two seconds to find a lobby as opposed to two minutes because of the manipulation of SBMM, i have also been playing cod since cod4 i wasn't great but you know what i got better over time and it felt like an achievement whereas now its the same type of lobby over and over again and makes me and many others not want to play full stop.

If there were no SBMM, all players *would* actually play the same type of lobby over and over (on average). Even though there would be a wider variety of skill within the match, there would be less variety in personal outcomes. If you were a below-average player, you would be unlikely to even call in a UAV without SBMM.

For average players, the complaint is that the current SBMM system is too manipulative. If they do well for a couple of matches, then they're guaranteed to get stomped back down. What you're describing though is the experience of the top players, where they don't get enough variety, which is also a real problem.

While some players have the time and determination to get good at COD specifically, for most players it really is just a casual game and they have other options. If the only options are "get good or have a bad time", then that's not really a casual game or a recipe for keeping most players engaged for more than a few weeks. What worked for games 15 years ago -- where development costs were maybe 10% of what they are now and revenue was mostly from box sales -- is not a good live-service strategy in 2025. Anecdotally players have quit COD over SBMM, but the vast majority haven't (they're just not vocal about it). That's not to say your personal experience doesn't matter. The question is how to keep the game accessible to the widest player base (which is why SBMM exists) while also keeping it fun for the most skilled players, and I would agree that Activision could do a better job of balancing those objectives.

 

[c0lson]

• Does Warzone matchmaking tap into MP match making?
• If Xdefiant gained more popularity would Activision actually care? With the juggernaut that is CoD
• Let's say next year's game is complete a couple dumpster fire after 3 seasons updates. Does CoD have a emergency game ready to go? I always had the tinfoil hat idea that CoD has 1 blackops game 1 MW and a wild card game for the brake glass emergency.
• Do any high-ups in Activision or Microsoft care about the outcry for the CoD community? People who care to make a complaint and fallow up with it have to be a extremely small group out of the millions other players who don't care.
• GHOST 2 PLEASE the game had its issues but ghost would be such a good bridge for the MW and Blackops game. Due to CoD wanting a universe Cod-Iverse
• Why does SHG care more about player feedback compared to the modern blackops and MW devs. I think SHG makes a good risk to branch out of the MW & blackops cycle. Blackops is CoDs "fast and the furious" franchise
• If you could use any games match making system to help fix CoDs what would it be? What would you take and add from both. What is the close to perfect model and the worse model for match making? Would you make ping as king? Pnw has a much smaller pool compared to Eastern and Southern US. Would you take Internet speed into consideration? Sniper elite still asked what type of Internet you have and if it's good or bad ( I remember this in the past game I think the new sniper elite has it as well)

Would a model look something like this
Quick speed test on launch
Ping

Local pool of player

External area of player if internet speed and type allows if your speed hits a certain amount it keeps that logged so you can't abuse it. IP log and other ID

Skill from 1-6 1 being new 6 being the twitch tryhard g fuel huffing and Adderall fueled kids.

Account trust score. Past games to current. If you been investigated the anti cheat will place in a lobby with similar players but using ping. After 3 games you get put into the normal match system your account gets a in-game notification during the whole process allowing you to submit a system log for PC players. For console players you can contact the support team. This will allow a fast pass for the legit players who get Spam reported and can put a temporary halo so the game can develop a score of trust and skill.

Sorry for the long post. To much g fuel and Adderall. This is a joke please don't abuse anything.

1. WZ and MP use the same matchmaking tech, but I'm not sure if they're still sharing the same skill-rating number (MMR). There's a lot to be said about what "skill" means in a BR vs conventional MP, but maybe a topic for later.
2. I'm sure Activision leadership paid attention to XDefiant even if they didn't see it as a likely threat (there's always a possibility of success even if Ubisoft shot themselves in the feet repeatedly). For sure they care about what other successful games are doing; e.g. Battle Royale and free-to-play were not original ideas. So in the unlikely event XDefiant had managed to retain players, it would've certainly affected COD's strategy.
3. That's not a totally crazy theory, but it's more a matter of release timing -- like they wouldn't make a likely-profitable secret game and then just sit on it. Was it just a coincidence that Modern Warfare Remastered (2016) happened to ship with Infinite Warfare though -- the game whose trailer was (still is) one of the most disliked YouTube videos of all time?
4. Yeah, everyone cares about player sentiment, but I don't think they know how to address it without hurting revenue. It *is* possible to improve the experience for all players, but there's a lot of nuance to it, one of those scalpel-not-a-chainsaw problems (contrary to what many vocal players believe), and it comes down to math and design.
5. Ha, well I don't work there! (even if I did, it wouldn't matter)
6. Not sure. Just different personalities / studio leadership I suppose. Treyarch is OG though, while SHG is an underdog.
7. There are too many datacenters and game modes to just pick the best data center for each player. Also, as long as latency is decent, it's generally more important that all players in the match have similar latency than everyone having their lowest possible latency. That's probably going to trigger some people Short answer though, I would look to improve the existing matchmaker, not replace it. There are lots of small changes that would have big improvements, like better party-modeling so players of diverse skill can play together. Tuning of MMR behavior. Transparency (show MMR in combat record), etc.

 

[c0lson]

Hi Charlie, I'm one of those who stopped playing CoD a few years ago, mainly because of SBMM. I wasn’t super, super good (just a little above average I woudl say) but not the kind of player who dominates every game. When SBMM came in, I started feeling extremely frustrated. Trying to have some fun and play casual games while constantly feeling like I was competing in a world championship final was horrible and exhasting. It burned me out and it stopped being fun, so I quit.

I understand that, from a business perspective, SBMM is valuable, it keeps the general public engaged, and, at the end of the day, it helps the store generate more revenue. Since SBMM is here to stay, could something be done to make it more approachable for experienced players while still keeping casual players engaged? In that way, the general audience would remained engaged, and the small percentage of "good" players could also enjoy the game.

Here’s my idea, I’d love to know your thoughts on this and any potential concerns. I’m not sure about the technical limitations, but let’s imagine it could be implemented without any issues:

Basically, there would be two player pools, a "newbie" pool and a "general public" pool. The general pool would include most of the player base, and matchmaking would prioritize ping over individual skill. I understand that less experienced players might find it frustrating to be stomped by more skilled players, so my suggestion is that new players start in the "newbie" pool, and once the MMR determines that a player has a good understanding of the game, they would be moved to the general pool.

This way, all types of players could have fun, and the store would still be profitable. Less experienced players would remain engaged (as they are now), while more experienced players would enjoy the game, play longer, and ultimately spend money on in-game purchases. The main issue I see with this is longer queue times, but that's already a problem since matchmaking prioritizes finding players with similar MMR over ping, resulting in boring and low quality games.

Thanks for your time and sorry for the long post. Maybe what I’m proposing is completely crazy, but I’m just trying to find a solution. Since SBMM is here to stay, I’d love to return to CoD, but in its current state, I just don’t have fun anymore.

Systems like that have been tried before, but the transition from the n00b pool to gen pop loses a lot of players. Someone might do pretty well in the n00b pool, but then that gets taken away, they're forced to play in gen pop where they're no longer above average, so they stop having fun and quit.

It's analogous to your experience in going from loose SBMM to tight SBMM in 2019. I'm not saying I blame you for not wanting to play a game that suddenly expects you to play against tougher opponents, especially without any recognition or transparency. That's totally understandable, and Activision can (and should) address parts of that. But you went from being able to feast on low-skill players sometimes (which is fun) to now being matched against players of similar ability more often (less chill for you, but a better experience for the low-skill players), and that caused you to quit. So, if a long-term committed player will quit the franchise because matches have gotten relatively more difficult, a less-invested n00b is even more likely to churn when they make that leap out of the training pool.

A couple of things could help:

Most controversial probably is adding bots to low-skill lobbies (it also increases server costs). But that would allow low-skill players to find appropriate matches without needing to crank the SBMM tightness all the way up. That results in a wider band of skill across all lobbies, including the above-average lobbies.

Easier and less controversial would just be rewarding players for having high-skill, even in casual lobbies. It's not the same as Ranked; this would be pretty light. Like just showing the average MMR of the lobby, and showing personal MMR in combat records, not in-match or on the AA report. An XP multiplier for high-skill lobbies. Little acknowledgements and rewards like that. (Casual MMR is also fast and loose, while Ranked MMR is slow and precise)

Not everyone would be happy with any of that either of course, but I think those are at least realistic possibilities.

 

[c0lson]

First wanna say I appreciate you talking about this and taking your time to explain. I think alot of us are fired up because the way Activision uses this in a way that hurts the community. But i wanted to ask why or what are the stats to prove the matchmaking helps? And how do you feel about the way Activision is using it? Because I have played cod since mw3 (original), and i have had so much fun up until about mw19 where my matches just felt sweatier and the game is less fun overall. I understand there is a need for a mmr system, I have been playing the Hunt Showdown 1896 that shows you your mmr and the mmr of the team overall. And I can tell you I would get smoked by people that are higher than me. Also not really a question but you made the comment about Xdefiant matchmaking, I think you are wrong about saying that's what caused it's downfall when in reality what really caused it to fall off was the lack of content and Ubisofts confusing mismanagement of the project. I think if it had the content and support it could've went head to head. Especially when everyone is very tired of the manipulation.

Most of the data is not shared publicly, but Activision published some of what they've looked at (Figure 3 on page 9 for example):
Call of Duty - Skill Whitepaper

I've run similar experiments and seen similar results across everything from in-game quit rates, to return rates (how often players come back within N days), new player retention rates, etc.

The particular problem with SBMM in COD is that above-average-skill players were acclimated to being above-average in most of their lobbies pre 2019. If you were a 75th percentile player by skill overall, in an average match with weak SBMM, you were also pretty close to 75th percentile in that match, give or take. So that ingrained expectations in veteran players. With tight SBMM now, a 75th percentile player is now much closer to the 50th percentile in their average match, which is less fun intrinsically. However, SBMM also increases the variance of outcomes, which means a 75th percentile player will see more highs than before (which is fun), but also more lows (which will feel bad because they were used to being consistently in the upper-middle).

What really adds insult to injury is that SBMM takes away KD as the objective measure of skill; it diminishes bragging rights. And more generally it dulls the sense of skill improvement. IMHO Activision didn't think about any of that from a design perspective (and still doesn't). Studios don't control the matchmaker, they just focus on the gameplay and content side, but are generally oblivious to the player matchmaking experience. SBMM improves the bottom line, that's why it's there, but there are so many ways it could be handled better (starting with transparency).

Regarding XDefiant, I would agree the game failed for many reasons, but with high certainty I would also say their executive decisions around matchmaking did not give the game the best chance to succeed (I don't mean that as hindsight; I gave a talk the week they launched last May saying the same thing), and with reasonable certainty I would say the lack of SBMM significantly accelerated the demise.

Mark Rubin assumed that random matchmaking would lead to more variety of outcomes for players ("sometimes you stomp; sometimes you get stomped"), when the exact opposite is true in reality. Random matchmaking leads to *less* variety of outcomes -- mathematically, provably -- which means the bottom half of players mostly just get stomped. The gameplay itself isn't that special, so rather than get stomped in a game they didn't even pay for, casuals will go off and play a different game. For the game to succeed, something about it would have to be so novel, interesting, and fun that players would be willing to get constantly stomped just for the rush of playing it (e.g. the good old days of COD4). Unless they can sustain a F2P with a small group of hardcore players, there was basically no chance for XDefiant to survive.

Ubisoft hasn't shared their data directly, but the console DAU charts I've seen show the worst retention of any comparable AAA launch, which is consistent with that theory. Lack of content didn't even have time to be a major factor, but I can see an argument for it contributing. At the very least, we can be sure that the lack of SBMM didn't *help* them retain players, even though it was probably great for marketing and player acquisition. It would be great to see their internal data though, especially the change in return rates between onboarding and gen pop.

 

[c0lson]

Hi Charlie,

I started playing Call of Duty on World at War, in 2008. My favorite games are COD4, MW2 and BO2. Back in 2008, I wasn't very good at the game, however, I spent hours on the game, every day. Sometimes I won, sometimes I lost, but the experience felt very authentic and I really wanted to improve and get better.

It seems to me, those games didn't have any form of SBMM whatsoever, if it did, it was perfectly set up. Nevertheless, me and a lot of people in the community enjoyed those CoD games more than any CoD game that released after 2015. I also remember that the player count on each game was very high (more than a million on BO1 or MW3, just on the PS3 sometimes).

My question is, how can you explain that games from 2007-2012 were so succesful and beloved by a lot players if they didn't have the same algorithm as the newer entries? Do you think that those games would have been even better if they had BO6-like SBMM? Speaking of BO6, it's currently losing a lot of players, do you think that without SBMM it would have been even worse?

I personnally think that CoD would regain in popularity if they removed SBMM, or at least tone it down.

Your perception is correct The games prior to BO3 (2015) had ineffective SBMM, and it became significantly more strict starting with MW 2019. If you were an above-average player, your lobbies became more challenging.

The short answer to how games from 2007-2012 could be successful without SBMM though comes down to two things: SBMM has very little to do with sales, and the landscape has evolved since 2010 anyway.

Games from 2007 to 2012 got most of their revenue from box sales, so that was success back then. Now the game still has to sell well and also retain players for IAP / MTX (development costs are also orders of magnitude higher, so that's not just motivated by pure greed).

It's worth noting that post-2019 COD titles sell just as well as the old peaks in 2010 and 2011, before the low points in the mid 2010s, which makes it hard to argue from a business perspective that SBMM is significantly driving away players in the long-run. In the short-term, AB tests also show very conclusively that SBMM, despite its flaws, is a net win in terms of retention and revenue. So while intuitively it might feel like removing SBMM would make the game more popular (because it definitely would make it more fun for you), it would not; the majority of players would have increased churn, and that would pretty quickly erase the influx of players attracted by no-SBMM. But even though SBMM is a net win for most players, it's still undeniably a bad experience for some players, and that's worth trying to improve.

Regarding hypothetically adding SBMM to old COD games, I don't think it would matter as much as it does now, but it probably still would've improved retention (though that wouldn't have translated into revenue without monetization systems). Less impactful though partly because COD was novel, so people were willing to get stomped just because the game was new-ish and exciting. Also because the real skill distribution has probably shifted significantly over the last 10-15 years. The median player in the early days of console FPS was probably equivalent to a low-skill player now. SBMM probably would have changed the culture around chasing KD though. Either way, I wouldn't encourage any game to implement SBMM without either transparent skill-ratings and/or a true Ranked system.

 

[c0lson]

So if SBMM made its way into the franchise back in 2016 with infinite warfare. Why do you make the comment that live service games cant survive without the SBMM system that YOU created? Can you explain to me how this franchise survived as the #1 FPS shooter on the market from 2007 to 2013 without it? Call of Duty became what Call of Duty is today without this ridiculous system but you claim it wont survive without it. If you ask me the only reason you are making these comments is because A. You created the system and B. You now run a company that provides the services. If Call of Duty is losing more players than it has ever because of this system and this algorithm can you please explain why this is a necessity in live service games? SBMM may benefit these games in the short term but it is now showing that this system is never going to work long term and the rapid decline in the player base and the amount of players SAYING they cant stand it is showing. What else needs to be done to wake up and own up to the fact that its not working!

SBMM increases retention, but it has very little to do with retail sales. Retention was mostly irrelevant to COD until the introduction of loot boxes in BO3 (2015) -- which is also when Activision first started taking SBMM seriously. Prior to that, say 2007 to 2013, DAU would've spiked at launch, Christmas, and DLC releases, and then fallen off more quickly than it would have *with* SBMM, but the success of the game was based on retail box sales. Retention mattered less in the early days because it wasn't a live-service model; retention didn't translate into in-app purchases like it does in most AAA PvP games now. COD titles since 2019 sell just as well though as older titles AND they generate more IAP revenue, so it's hard to argue that it's not working. Here's a link to ATVI revenue by year.

Even though SBMM is a better experience for most players, and a net win for Activision/Microsoft based on all the data available to them, it's clearly not a good experience for some players, including yourself, so there's an opportunity for improvement. The question is how to improve your experience without asking Activision to make less money. The most viable options I can think of revolve around rating transparency (lots of good options there), better fine-tuning of the rating system, and possibly the addition of bots into low-skill matches. Removing SBMM might be great for you, but not for the majority of truly casual players.

As far as your allegation of SBMM being responsible for Call of Duty losing more players than it has ever... the system hasn't really changed all that much over the last 6 years. So it seems more plausible that the game design, the quality of execution, and competition in the market are bigger factors. Not to keep piling on XDefiant either, but if removing SBMM was actually good for games, how bad must XDefiant have been?? I didn't think it was absolutely awful, a lot of people even said it was more fun than COD, but somehow it had the worst retention rates of any recent AAA launch *despite* the hypothetical advantage of not having SBMM. Meanwhile Rivals uses blatant EOMM and is still going strong. So, maybe SBMM isn't to blame for all trends.

Side note for the record: I didn't create COD's SBMM, I just developed the skill-rating system that SBMM uses. If you think I somehow Jedi-mind-tricked Activision into using SBMM, you're definitely overestimating my influence; they're not even using my algorithm the way I would. I do have an ulterior motive for talking about it -- the more people understand the nuance of skill-ratings, the better it is for my business in particular -- but I didn't create the market. All AAA studios are capable of analyzing their own data and coming to their own conclusions about the need for SBMM; I don't have to convince them of that (except for XDefiant apparently).

 

[c0lson]

For the podcast with the SBMM creator:

CoD was doing well from 2007-2015 and specially before 2019 as we now know BO3 is the best selling CoD of all time, you could find lobbies way faster during that period (tested by several youtubers) and that is without crossplay, so what is the benefit? It used to work just fine and they fixed something that was not broken so what is the benefit for better than average players and up if their connection quality, matchmaking speed and overall enjoyability of the game decreased severely?

I also want to know, cause I tested this myself, in lower skill lobbies I had access to via a friend who is not good at all in MP, as much as I was stomping them, usually only one or two players quit the match, in my regular lobbies on my account, 4 to 6 players quit throughout the match ( including myself admittedly), so it is very evident to me, 1 good player can't ruin 11 bad players experience, but 12 of those good players can ruin each others experience, so, why do we need it? Clearly low skilled players don't even care if there is 1 guy doing really well

The questions just before yours were similar re: 2007-2015, so I'll also refer you to those responses so I don't repeat too much.

I think you meant BO1 (2010) was supposedly the best selling, but MW3 (2011) actually sold more... and AFAIK the rebooted MW2 (2022) actually sold the most, but the Microsoft acquisition obscured the numbers. Regardless, 2019 through 2024 are all on par with 2010 and 2011, with significantly higher total revenue than the older titles. That's just to the best of my knowledge; exact sales numbers are hard to come by.

But you're right, there is very little benefit for better-than-average players with the way SBMM is currently implemented, at least if your enjoyment primarily derives from having a high KD. You will get tougher matches than before, and many players understandably find that less fun. However, the return-rate data indicates it's only the top 10% by skill who are net-negatively affected by SBMM. So, presumably not every above-average player minds playing against opponents closer to their skill level (on average), and some must prefer it. The volatility of the MMR system also means you will get a mix of matches where you stomp and matches where you get stomped -- which is arguably the goal of "casual" matchmaking. The data also genuinely indicates SBMM leads most players to engage more; but it's of course possible that playing more doesn't necessarily mean it's more "fun".

So, I was actually surprised by the data there. I would've expected return rates to look worse for all above-average players. In theory, SBMM takes away bragging rights and dulls a player's sense of improvement. It forces players to work harder, adds more cognitive load, etc, and does not give acknowledgement or reward for it. Personally I think there's a lot to improve there from a design perspective, but that's up to the lead studios.

Regarding your experiment, I've looked at rage quit rates quite a bit, and it's consistently the slightly above-average players who do it the most (both in Western audiences and in China). Quit rates seem to peak around the 60th percentile of skill. One of my cynical theories is that it's like an adult softball league -- there are a lot of guys who take it more seriously than their skill level supports ("your ego is writing checks your body can't cash"). They're competitive, but not as good as they imagine, so getting stomped triggers cognitive dissonance. The n00bs have less illusions about their ability, while the highest-skill players just rarely get stomped. Another theory is that n00bs don't even realize they *can* quit the match.

That being said, low-skill players might not seem to mind the occasional party crasher, but everything I've seen suggests they're hanging on by a thread. They'll finish out the match, but without SBMM they're much more likely to unsintall than other cohorts. The data in Activision's Skill Whitepaper supports this pretty clearly, but I've seen it from my own analysis, and in other games as well (that's why bots are good).

 

[c0lson]

Do you think that with strict SBMM, people are actively discouraged from improving at the game? If you get better, you’re just punished by being matched against sweats. Thanks

Yeah, I mostly agree with that. They're not extrinsically rewarded for playing tougher matches, so implicitly discouraged at least.

 

[c0lson]

Q1)What new Sbmm would you put into CoD? Being unbias as possible what company currently makes the best Sbmm style?

Q2)What makes it different to your companies? Should CoD have two possibly 3 different styles of Sbmm? Rank MP and Warzone?

Q3)If you could go back to day 1 working for the CoD team what would you do differently? What would you do differently at the halfway point and end point?

Q4)Can anti-cheat be intertwine in match making systems? Would it be a good option to help?

Q4) favorite childhood sandwich that you still eat exactly the same? I like a ham sandwich with chips crushed into the bread. Some honey mustard light mayo.

1. Ha! Are you setting me up to promote my own company? SBMM has a lot of components to it, so I would mostly tune rather than replace, however I would swap out their current MMR system for IVK Skill (new algorithm I wrote), since it's built to solve exactly the kinds of problems that COD has. I would also update the party and team models so that it would be better playing with friends, and so high skill players would be expected to carry less. Swapping out the matchmaker itself is not practical, but regarding best SBMM, NetEase / Rivals has good matchmaker *tech*, but the way they use it kind of sucks IMHO. A big part of that is because they're using TrueSkill for MMR, which has foundational problems, e.g. overrating high-skill players, distribution drift, and stagnant ratings. The choice of TrueSkill means they have to use Hidden MMR for Ranked, and they also have to resort to heavy-handed mitigation in casual, like using deep-learning ML just to account for simple stuff like the effects of playing in a party. To keep it short, there are big flaws in most of the SBMM systems out there. No one has put all of the best pieces together just yet, but to sort-of answer your question, 1v1 games tend to have pretty good matchmaking in general (because it's a much easier problem then). My company is also developing a better matchmaker than NetEase, but we aren't talking much about that until it goes live

2. Yes, COD should have separate MMR systems and track separate numbers for Ranked, MP, and Warzone. They should probably also have different MMR configurations for each of the MP modes, because skill in a mode like Search & Destroy is fairly different than TDM. It's difficult to measure player performance in SnD; skill has to be measured primarily from the team outcome (and incorporating margin of victory). In TDM though, skill can be deduced almost entirely from individual player performance.

3. TBH I would probably just ask for more money at every point

4. Yes, and yes. Skill-ratings are a strong signal to feed into ML cheat detection. I would use a different skill-rating system than what the matchmaker uses though, because they have different objectives & properties.

5. Chicago-style Italian beef, mozzarella, hot peppers, dipped.

 

[c0lson]

What about using a Netduma Router? It allows you to restrict your connection to specific regions/Data centers, how effective is this against SBMM/how does it impact the Algorithm? You can also block data centers, does having a less amount of data centers improve your ability to manage the whiplash effect?

Yes, if you're willing to sacrifice ping and wait time, you theoretically can get yourself into lower population servers, like off-peak hours in another region.

If you're a high skill player, that means easier matches. It also means a wider variance of skills within your matches, which means you'll notice less whiplash overall. You'll still experience some whiplash just from the team-balancing though, like you may end up with more and more of the worst players on your team. Just in general the team balance will be less favorable. Your KD will probably improve either way though (unless your latency puts you at a major disadvantage).

 

[c0lson]

Thanks to everyone for the questions. I tried to put a fair amount of detail into answering them, but it goes without saying that it's all just to the best of my knowledge (and that's pretty limited).

I don't think anyone asked about disbanding lobbies, but I'm going to bring it up. My assumption is that it's to make sure the less popular game modes and data centers still get players and to reduce wait times. The old matchmaker partitioned everyone by playlist, so you could only search within one playlist at a time. The new matchmaker looks to build a lobby of players with at least one playlist in common, similar to how it searches for players with at least one viable datacenter in common. Previously those would have been distinct partitions, which doesn't leave any room for trade-offs. If you don't disband the lobbies, you don't get enough players in the search pool that meet all of the critieria -- or at least it's a lot more difficult to find combinations of players that work -- so only the most common criteria get matched. (That's what I think the reasoning is anyway)

Side note, the hardcore "Ping is King" players are basically asking to go back to partitioning by datacenter first, and then matching on the rest of the criteria within your chosen datacenter. That's fine if you only have one or two datacenters per region, and great for the players who live nextdoor to those DCs, but Activision has a lot of them now (I don't know exactly how many; might be mentioned in the matchmaking paper). Combine that with the large number of playlists, and wait times would be generally unbearable *even without* skill as a factor.

I think the matchmaker could use a different (higher complexity) algorithm to build closer-to-optimal matches, in which case ping could get closer to being partition-like, but that algorithm would make scaling more challenging, so that's probably why they haven't done it (at least according to last year's matchmaking paper). Again, that's just my opinion. I don't work there, just making a lot of assumptions about the internals.

 

[c0lson]

I should also explain the difference between CCU, DAU, and retention, since a lot of people were thrown off when I said on the IG podcast: "I don't think anybody has the true retention data. Like, you don't know what the active player counts were on the old CODs; I don't know that Activision knows that. So if somebody has that data, I would love to see it."

So that blew a lot of minds. The thing people need to understand is that in-game concurrent player counts (CCU) are not the same thing as the Daily Active Player counts (DAU), and DAU isn't even quite the raw data you need to measure that "true retention" (e.g. D7 retention: how many new players on a given day returned 7 days later). With the way older titles collected telemetry, I'm honestly not sure the necessary data for that analysis exists in a database somewhere. You'd think it would, but data has evolved a lot since then.

CCU *is* still a good metric for the total product of acquisition, retention, and engagement, but even if that's what I had meant (it's not), it needs to be a time series to make inferences about retention and/or SBMM -- not a screenshot, or a vague recollection that the number was pretty high -- because retention rates measure a change over time. The curve is what we care about. In either case, if someone has retention data, or DAU, or even CCU time series data from 10+ years ago, I would still love to see that.

The broader point that's getting missed by obsessing over semantics is that comparing retention across years is apples to oranges regardless. It's very possible that older CODs would've had even better retention with stronger SBMM (that should also trigger some people ), but we can't go back in time to test it under the same market conditions. We can test SBMM in modern live-service games though -- which need retention for revenue, not just box sales -- and every experiment I know of, from multiple studios and publishers, suggests pretty conclusively that SBMM improves overall retention and revenue. So that's why they use it, not because I told them to.

 

[c0lson]

Ope! One more post. Speaking of semantics, EOMM comes up quite a bit, so let me elaborate on that quick.

SBMM is simple. If your MMR (one-dimensional skill-rating) gets lower, you will get easier matches (including team balance); if MMR increases, then you get tougher matches.

The matchmaker described in last year's paper -- which sounds essentially the same as it was 5 years ago, though the ping/TTM/skill weights may have changed somewhat -- is pure SBMM and not EOMM. The match quality function has very simple inputs, and it DOES NOT take into account a player's inventory, play-style, recent match history, purchase history, loadouts, quit rate, etc like EOMM would. The only player profile data used by SBMM is the MMR. That's it. Even though Activision tracks a lot of player data for analytics, those are not factored into the match quality score used by the matchmaker.

EOMM by contrast requires building a player churn model (and possibly spend model), running a lot of player profile data through it, and using that output to calculate the expected match quality. That's a big change to the matchmaker, because you need to manage a lot of extra data per player (added overhead of querying and managing significantly more memory per player), on top of separately training and maintaining the churn and/or spend models. With SBMM it was just a matter of minimizing one number, but EOMM requires a very different architecture (e.g. maybe adding a microservice for match quality calculations with a cache for all that profile data). That's not an impossible change to make, but one does not simply drop in EOMM, which is why I think COD probably hasn't and won't. And if they did, there's no reason to go through such an insane level of detail in the matchmaking whitepaper just to construct an elaborate lie -- the people who would care the most already assume the worst.

BTW: Yes, there was a team at Activision who was interested in modeling churn so they could trigger messages and emails to keep high-risk players engaged, and they've given presentations on that, but they had nothing to do with matchmaking aside from also using MMR as an input to their churn model. People might be surprised at how little interaction or coordination there is between various departments in massive companies (especially departments located in different continents in this case).

In any case, "Volatile MMR" is still far from EOMM. Sure, the MMR is reacting to your recent performances, but that's it. The way that in-game performance is measured is also simple, though kept secret so players don't exploit it. The behavior of COD's MMR system can't even be tuned very much (which is one of the major problems with it IMHO). They're probably still running it with the default out-of-the-box settings, so it's a stretch to even call it "optimized." It's designed to adapt quickly, but in many cases it adapts too quickly or totally overcorrects, while for elite players it corrects too slowly. Some volatility is good though -- you don't want every match to be the same -- but it's a balance that COD hasn't found.

If the MMR system went to the opposite extreme, if it didn't have enough volatility -- like you see with Halo and TrueSkill MMR systems -- then you really would play the same type of opponent all the time (unless you deliberately inject randomness or streak-breaking). TrueSkill is easily smurf-able, since it overtrains on your first few matches, and it overestimates high-skill players, which is why you see some extremely good Halo players with just 49% lifetime win rates.

So, there's more nuance in MMR system behavior than most people realize, but the point is that COD's matchmaker is still just basic SBMM, not EOMM. To the extent that it feels "manipulative", that's just because of the MMR volatility, which Activision should tune, but evidently doesn't really know how to.