AMA: Charlie Olson - A Developer Who Wrote Call of Duty's SBMM

SBMM feels like it hits harder with solo queue players like myself. It becomes a common occurrence of “carrying potatoes”. It happens in COD, and is a common complaint in Apex Legends as well.

Is there anything that can be done for SBMM to better balance teams for solo queue players, instead of shifting the odds drastically in one direction or the other?
 
Activision has said Smbb started in CoD 4. Is that true? Why do the heads of CoD keep smbb in the game? Does it increase play time or profit? Why does Sbmm feel so different in Warzone compared to MP? In WZ it's like my group been CoD champs for the past month. MP the same night is average skill in my view. Also why wouldn't sbmm send cheaters and hackers to higher skill lobbies? In my view of smbb was a good system no avg joe would ever see a hacker at the skill level should pair them with pros not the common players?
 
If SBMM is so good why has many people quit cod over it and why does it feel like its been ramped up to level 1000 over the last 6 years when cod titles of the past have done just fine without it, using team balancing when ping was actually KING and it took two seconds to find a lobby as opposed to two minutes because of the manipulation of SBMM, i have also been playing cod since cod4 i wasn't great but you know what i got better over time and it felt like an achievement whereas now its the same type of lobby over and over again and makes me and many others not want to play full stop.
 
Last edited:
  • Does Warzone matchmaking tap into MP match making?
  • If Xdefiant gained more popularity would Activision actually care? With the juggernaut that is CoD
  • Let's say next year's game is complete a couple dumpster fire after 3 seasons updates. Does CoD have a emergency game ready to go? I always had the tinfoil hat idea that CoD has 1 blackops game 1 MW and a wild card game for the brake glass emergency.
  • Do any high-ups in Activision or Microsoft care about the outcry for the CoD community? People who care to make a complaint and fallow up with it have to be a extremely small group out of the millions other players who don't care.
  • GHOST 2 PLEASE the game had its issues but ghost would be such a good bridge for the MW and Blackops game. Due to CoD wanting a universe Cod-Iverse
  • Why does SHG care more about player feedback compared to the modern blackops and MW devs. I think SHG makes a good risk to branch out of the MW & blackops cycle. Blackops is CoDs "fast and the furious" franchise
  • If you could use any games match making system to help fix CoDs what would it be? What would you take and add from both. What is the close to perfect model and the worse model for match making? Would you make ping as king? Pnw has a much smaller pool compared to Eastern and Southern US. Would you take Internet speed into consideration? Sniper elite still asked what type of Internet you have and if it's good or bad ( I remember this in the past game I think the new sniper elite has it as well)
Would a model look something like this
Quick speed test on launch
Ping

Local pool of player

External area of player if internet speed and type allows if your speed hits a certain amount it keeps that logged so you can't abuse it. IP log and other ID

Skill from 1-6 1 being new 6 being the twitch tryhard g fuel huffing and Adderall fueled kids.

Account trust score. Past games to current. If you been investigated the anti cheat will place in a lobby with similar players but using ping. After 3 games you get put into the normal match system your account gets a in-game notification during the whole process allowing you to submit a system log for PC players. For console players you can contact the support team. This will allow a fast pass for the legit players who get Spam reported and can put a temporary halo so the game can develop a score of trust and skill.

Sorry for the long post. To much g fuel and Adderall. This is a joke please don't abuse anything.
 
Hi Charlie, I'm one of those who stopped playing CoD a few years ago, mainly because of SBMM. I wasn’t super, super good (just a little above average I woudl say) but not the kind of player who dominates every game. When SBMM came in, I started feeling extremely frustrated. Trying to have some fun and play casual games while constantly feeling like I was competing in a world championship final was horrible and exhasting. It burned me out and it stopped being fun, so I quit.

I understand that, from a business perspective, SBMM is valuable, it keeps the general public engaged, and, at the end of the day, it helps the store generate more revenue. Since SBMM is here to stay, could something be done to make it more approachable for experienced players while still keeping casual players engaged? In that way, the general audience would remained engaged, and the small percentage of "good" players could also enjoy the game.

Here’s my idea, I’d love to know your thoughts on this and any potential concerns. I’m not sure about the technical limitations, but let’s imagine it could be implemented without any issues:

Basically, there would be two player pools, a "newbie" pool and a "general public" pool. The general pool would include most of the player base, and matchmaking would prioritize ping over individual skill. I understand that less experienced players might find it frustrating to be stomped by more skilled players, so my suggestion is that new players start in the "newbie" pool, and once the MMR determines that a player has a good understanding of the game, they would be moved to the general pool.

This way, all types of players could have fun, and the store would still be profitable. Less experienced players would remain engaged (as they are now), while more experienced players would enjoy the game, play longer, and ultimately spend money on in-game purchases. The main issue I see with this is longer queue times, but that's already a problem since matchmaking prioritizes finding players with similar MMR over ping, resulting in boring and low quality games.

Thanks for your time and sorry for the long post. Maybe what I’m proposing is completely crazy, but I’m just trying to find a solution. Since SBMM is here to stay, I’d love to return to CoD, but in its current state, I just don’t have fun anymore.
 
First wanna say I appreciate you talking about this and taking your time to explain. I think alot of us are fired up because the way Activision uses this in a way that hurts the community. But i wanted to ask why or what are the stats to prove the matchmaking helps? And how do you feel about the way Activision is using it? Because I have played cod since mw3 (original), and i have had so much fun up until about mw19 where my matches just felt sweatier and the game is less fun overall. I understand there is a need for a mmr system, I have been playing the Hunt Showdown 1896 that shows you your mmr and the mmr of the team overall. And I can tell you I would get smoked by people that are higher than me. Also not really a question but you made the comment about Xdefiant matchmaking, I think you are wrong about saying that's what caused it's downfall when in reality what really caused it to fall off was the lack of content and Ubisofts confusing mismanagement of the project. I think if it had the content and support it could've went head to head. Especially when everyone is very tired of the manipulation.
 
Hi Charlie,

I started playing Call of Duty on World at War, in 2008. My favorite games are COD4, MW2 and BO2. Back in 2008, I wasn't very good at the game, however, I spent hours on the game, every day. Sometimes I won, sometimes I lost, but the experience felt very authentic and I really wanted to improve and get better.

It seems to me, those games didn't have any form of SBMM whatsoever, if it did, it was perfectly set up. Nevertheless, me and a lot of people in the community enjoyed those CoD games more than any CoD game that released after 2015. I also remember that the player count on each game was very high (more than a million on BO1 or MW3, just on the PS3 sometimes).

My question is, how can you explain that games from 2007-2012 were so succesful and beloved by a lot players if they didn't have the same algorithm as the newer entries? Do you think that those games would have been even better if they had BO6-like SBMM? Speaking of BO6, it's currently losing a lot of players, do you think that without SBMM it would have been even worse?

I personnally think that CoD would regain in popularity if they removed SBMM, or at least tone it down.
 
So if SBMM made its way into the franchise back in 2016 with infinite warfare. Why do you make the comment that live service games cant survive without the SBMM system that YOU created? Can you explain to me how this franchise survived as the #1 FPS shooter on the market from 2007 to 2013 without it? Call of Duty became what Call of Duty is today without this ridiculous system but you claim it wont survive without it. If you ask me the only reason you are making these comments is because A. You created the system and B. You now run a company that provides the services. If Call of Duty is losing more players than it has ever because of this system and this algorithm can you please explain why this is a necessity in live service games? SBMM may benefit these games in the short term but it is now showing that this system is never going to work long term and the rapid decline in the player base and the amount of players SAYING they cant stand it is showing. What else needs to be done to wake up and own up to the fact that its not working!
 
For the podcast with the SBMM creator:

CoD was doing well from 2007-2015 and specially before 2019 as we now know BO3 is the best selling CoD of all time, you could find lobbies way faster during that period (tested by several youtubers) and that is without crossplay, so what is the benefit? It used to work just fine and they fixed something that was not broken so what is the benefit for better than average players and up if their connection quality, matchmaking speed and overall enjoyability of the game decreased severely?

I also want to know, cause I tested this myself, in lower skill lobbies I had access to via a friend who is not good at all in MP, as much as I was stomping them, usually only one or two players quit the match, in my regular lobbies on my account, 4 to 6 players quit throughout the match ( including myself admittedly), so it is very evident to me, 1 good player can't ruin 11 bad players experience, but 12 of those good players can ruin each others experience, so, why do we need it? Clearly low skilled players don't even care if there is 1 guy doing really well
 
Last edited:
Q1)What new Sbmm would you put into CoD? Being unbias as possible what company currently makes the best Sbmm style?

Q2)What makes it different to your companies? Should CoD have two possibly 3 different styles of Sbmm? Rank MP and Warzone?

Q3)If you could go back to day 1 working for the CoD team what would you do differently? What would you do differently at the halfway point and end point?

Q4)Can anti-cheat be intertwine in match making systems? Would it be a good option to help?

Q4) favorite childhood sandwich that you still eat exactly the same? I like a ham sandwich with chips crushed into the bread. Some honey mustard light mayo.
 
What about using a Netduma Router? It allows you to restrict your connection to specific regions/Data centers, how effective is this against SBMM/how does it impact the Algorithm? You can also block data centers, does having a less amount of data centers improve your ability to manage the whiplash effect?
 
Hi everyone, thanks for the questions and sorry for the delay. I'm going to try to respond to all of these posts in order over the next week or so. I'm traveling a lot this month, so my replies will be staggered, but I genuinely appreciate everyone taking the time to post.
 
LoLzWorld said:
Is it true that SBMM increases profits? What benefit outside of player counts does SBMM have on games?
Click to expand...
Yes. To the best of my knowledge, every AAA studio/publisher that has run experiments on SBMM (at the very least Activision/Blizzard, Microsoft, EA, Nexon, Riot, NetEase, Tencent, etc) have all seen similar results clearly indicating increased retention and lifetime-value per customer *on average* with SBMM. Not a lot of them share their data publicly, but both Activision and EA have published some of their results.

That's counterintuitive to a lot of high-skill & highly-engaged players who feel like they are negatively affected by SBMM, but for the vast majority of casual players, even flawed SBMM clearly boosts their engagement, retention (how likely new players are to come back on the Nth day after first playing), and return rates (how likely any player is to come back within N days of their previous session). Those increases then lead to measurable increases in lifetime-spend per-player. If any analytics team saw that SBMM was increasing churn and losing them money, they simply wouldn't use it.

That being said, nearly all SBMM systems are significantly flawed and bad experiences for certain segments of players, so the question is how to fix those flaws while maintaining the benefits (from both player and dev perspectives).
 
insp3ctre said:
Knowing the impact your SBMM algorithm has had on the COD community, if you could go back and change anything about it, would you?
Click to expand...
Before I knock it, first I want to say that COD's MMR algorithm actually has a lot of good properties that other systems do not (e.g. TrueSkill). Those details are maybe a topic for another post though. Yeah yeah, of course I should say that because I wrote it, but believe it or not, COD's system could be a lot worse :P

The main thing I would change on the MMR system (which measures player skill; not the matchmaker itself) is the volatility control. I mentioned on the podcast that mid-range players experience too much whiplash (too much volatility), while extremely high-skill players experience too little volatility. That's one of the reasons why an average player will do really well in one match and then get destroyed in the next.* On the flipside, an elite player will find it hard to get a break. So, I know now how to even out the volatility and make it easy to tune, but I didn't know that 10 years ago, and it's pretty unlikely anyone at Activision knows how to do that. It would lead to a smoother player experience though IMHO.

The other key thing I would do is make MMR increase only when you win and decrease only when you lose. I really like some of the analysis players have been doing on their skill data, and one of the confusing things they've noticed is that sometimes MMR decreases when you do well just because you're playing in a lobby with low-skill players. That obviously seems pretty messed up from a player perspective. There's a lot more nuance to why that is, but I don't think I can go into the details without divulging proprietary algorithm details, so I'll leave it at that.

On the design side, I would also petition harder for player-facing skill-feedback. Like, visible lobby-difficulties, and showing MMR in the combat record. (Realistically not much I could do there as a non-designer at a support studio though).

As far as the rest of the matchmaker goes, I have a few ideas how I would improve it, but I didn't write it, so won't comment on that.

*Volatility is not the only reason why sometimes you stomp and then in the next match you get stomped. There's also just a random component to it, especially if you jump to less-popular game modes, which means weaker SBMM.

Ironically, for most players, loose matchmaking will feel sweatier than tight SBMM -- because there will be a greater proportion of relatively higher-skilled players in the lobby. Loose SBMM only really benefits you if you're above-average, and even then, unless you're elite, you will dominate a lobby *less often* than you would with tight SBMM (which you may perceive as it being more sweaty). So, for a slight majority of players, the perception of matchmaking being cranked up is actually inverted.
 
Last edited:
MrRain said:
SBMM feels like it hits harder with solo queue players like myself. It becomes a common occurrence of “carrying potatoes”. It happens in COD, and is a common complaint in Apex Legends as well.

Is there anything that can be done for SBMM to better balance teams for solo queue players, instead of shifting the odds drastically in one direction or the other?
Click to expand...
Yes. COD has a naive team-balancer that could easily be improved. I can't speak for Apex, but I would bet their approach is similar... and probably even worse than COD if they're using a "Bayesian" MMR algorithm like TrueSkill. Worse because TrueSkill has a classic problem of overestimating skill at the extremes, meaning elite players get extra unfavorable team-balancing.

The problem almost all games have with team-balancing though is that they model team skill as the simple sum of player skills (a uniformly-weighted sum). In most game modes, that model overestimates a high-skill player's ability to carry, and the more the game mode requires teamwork, the worse the team-balance becomes. A good solution though is to model team skill using exponentially weighted MMR with a negative exponent -- which is not going to make sense in plain English like that, but for real is a good solution. Some slides that might help communicate that better:
Team Modeling PDF

In any case, COD doesn't do that, but they could. There are some other hacks they could try too (like matching intra-team skill variance), but that's probably getting pretty far off in the weeds.
 
lackluster said:
Activision has said Smbb started in CoD 4. Is that true? Why do the heads of CoD keep smbb in the game? Does it increase play time or profit? Why does Sbmm feel so different in Warzone compared to MP? In WZ it's like my group been CoD champs for the past month. MP the same night is average skill in my view. Also why wouldn't sbmm send cheaters and hackers to higher skill lobbies? In my view of smbb was a good system no avg joe would ever see a hacker at the skill level should pair them with pros not the common players?
Click to expand...
It's true that COD4 had some sort of SBMM, but it wasn't the same system they have now. It wasn't possible to match players tightly on skill in COD4 because the MMR system would suffer from model collapse, plus the matchmaker itself was pretty basic (MMR is the skill number the matchmaker uses).

The highest skill players are negatively affected by SBMM, but it measurably increases engagement and revenue on average across all players (for 90% of players), so that's why it's there. I would say it's possible Activision overlooks long-term damage to the brand and gradual burnout of high-skill players in favor of the more easily measured short-term effects. The fact that their own data indicates 10% of players are negatively affected by SBMM at least shows there's room for improvement. I'd go a step farther and say that increased transparency and recognition of skill are the least they could do.

I couldn't say for sure without seeing your data, but it's possible you're around average skill, and that's why you see a difference between WZ and MP. WZ has *looser* SBMM than MP, especially if player numbers are declining. For the majority of players, loose SBMM will actually feel sweatier than tight SBMM, because loose SBMM means more high skill players in the match. On top of that, there's limited team balancing in WZ. In MP, even if matchmaking is loose, the teams can still be balanced (unless there's a full stack). WZ can't do that as much, or at all if you're playing in a full squad.

If you're seeing hackers, it's also probably because SBMM is relatively loose, so you'll probably also see more hackers in WZ than MP.
 
You must log in or register to reply here.
Back
Top