Balance Testing

Click here to visit the Reddit thread.

Click here to visit the GD thread.

Click here to visit the CB thread.

Introduction

Curious to know how the various ranks stack up when playing against each other?

How likely is a GSSSB team to beat a SSSSS team? In other words, do you gain more from bumping a Silver player up to Gold than you lose from dropping a Silver player down to Bronze? How much does it matter?

To find out, we need some data. This is a balance-testing project that anyone can contribute to, just by running a quick match. It is also especially helpful to Factions, in that we want to know (e.g.) how many points an all-Gold Noxian team should get for beating an all-Bronze Demacian team, and vice versa.

All data is publicly available, so that anyone who wants to run analyses can do so. I will also be doing some stats myself, and my findings will be shared.

Simplest way to help: be a guinea pig

Here’s how to join the in-client chat for this round of balance testing. People setting up test matches can invite from here.

If you autojoin the chat as indicated above, you’ll probably get some invites every now and then. This is the easiest way to help: just be around so that people can pull you in when they want to set up a match.

Note: these test matches are not Factions matches. Play with the full roster. We’re trying to gather some data that isn’t influenced by differences in faction strength.

Rules for running matches

Beggars can’t be choosers, but for the sake of having nice, clean data, there are a few guidelines I’d like to ask you to follow.

1. Play seriously. One concern is that, where people realize their team has a rank advantage, or that they’re personally higher-ranked than the others, they have a tendency to derp around. “I’m a Diamond playing with a bunch of Silvers; time to play AP Garen!” The problem with this is that it distorts the data. That said, you’re of course encouraged to be nice to one another, to offer tips to lower-ranked players post-game, and so on.
2. Tournament draft. Ideally, you should set matches up as tournament draft. Ban normally. (For Factions players: use the full, 100+ Champion roster, not faction rosters. Part of the point of this is to measure the impact of rank itself; it’s hard to do that when there’s also faction vs. faction balance issues involved.) Each side gets up to 5 minutes of pause time in the case of a d/c or afk; after that, play on.
3. Make sure you identify who’s submitting the results before you start. It’d be a shame to lose good data because people forget to send in the match results.
4. Don’t play these as Factions matches. We’re trying to gather some data that isn’t influenced by strength differences among factions. Ban normally and don’t limit yourself to the Factions rosters.

How to contribute data

It’s really quite simple! Organize a custom Summoner’s Rift match, grab a LoL Nexus screenshot during loadscreen or during the game, and submit the results using the form below.

LoLNexus.com screenshots conveniently summarize the Season 3 and Season 4 ranks of players in the game. Remember to grab the screenshot during loadscreen or during the match: you can’t get LoL Nexus data for a match that’s already finished.

You can make the games as balanced or imbalanced as you like: both types of data are useful.

Although not required, I will say that the fewer ranks are involved in a particular match, the easier it is to use the data. It’s easier to draw conclusions from SSBBB vs. GGBBB than it is to draw conclusions from PGSBB vs. DPPSS, for instance. Still, all data is useful and welcome.

Submit results here

Use this form to submit results.

Finding players for matches

To help facilitate these matches, I’ve designated a chatroom as a meeting place. Within the LoL client, join the CCTBalance chat if you’re up for one of these matches, or if you’re looking for some guinea pigs for your mad experimentation.

You’re encouraged to set the chat room to “auto-join”, so you don’t have to manually log into the chat every time you connect.

Have fun!

Results and analysis

I’m committed to keeping this data fully public.

• Raw data — The raw data from the GDoc. It’s empty right now, because I just started this project.
• Survey results — Since I don’t have any match data to share just yet, here are the results of a related survey I conducted a while ago, concerning perceptions of the relative magnitude of gaps between ranks.
• Factions-inclusive data — This is a working dataset which includes Factions matches. Obviously, faction is a confounding variable, and this dataset is consequently “lower-quality” than pure balance testing data. Still, you might be able to do something with dummy variables.

Factions match analysis

Here is some preliminary analysis using a dataset made up almost entirely with Factions data.

In short: it appears that there is a small gap between Bronze and Silver, and between Platinum and Diamond. There appears to be a big jump from Bronze/Silver to Gold, and a big jump from Gold to Platinum/Diamond.

Win/loss computations from Factions data as of 2014-07-02. This is the most naive descriptive analysis possible: each win gives a single point to each rank on the winning team (multiple points if multiple players of that rank were on the team) and each loss similarly deducts a point. Thus, if 5 Silvers (SSSSS) beats 3 Silvers and 2 Bronzes (SSSBB), Silver ends up losing two points and Bronze ends up gaining two points.

Pairings analysis. This is a more specific analysis than the winrate analysis. Rather than simply awarding generic “win” and “loss” points, this method breaks each match down into evidence for or against specific claims, such as “Silver is stronger than Bronze” (S > B). For example, if PSSSB beats SSSSS, that would be counted as increased support for the “P > S” claim and decreased support for the “S > B” claim.

Message CupcakeTrap

You can send me comments and questions about the project here.