TB, I've always thought very highly for you. That's the single reason why I'm writing this post despite promising myself to give up on trying to explain the issue. Hope you actually bother to read it and think about what I'm trying to say.
_SRM_TheBored wrote:QUOTE (_SRM_TheBored @ Jul 15 2006, 03:18 AM) Yes, balancing on win percentage counts.
Yes, it does. Noir doesn't interpret the data right though. Why?
Because people aren't stupid.
Let me explain:
Performance of a faction isn't binary {win, lose}, it's n-ary {against fac 2, against fac 3, ... , against fac n-1} (yes, I'm on a layer of abstraction where settings are not considered, for simplicity). What Noir's method is doing is merging all performances of a given faction into one number. What this leads to is that imbalances tend to be obfuscated. Why? Here we're coming to the "people aren't stupid" part. Let me give you an example: Let's consider a situation where we have 4 factions (let's call them IC, Dreg, Bios and Belters) Two of them are "strong factions" (IC, Dreg), two are "weak factions" (Bios, Belters) (just labels of a set for ease of reference) Let's take it for a fact that a "strong faction" always beats a "weak faction" (remember, we're working on an abstract model here - in reality, extreme skill imbalance actually has a shot at turning this around). Two factions from the same category are "perfectly balanced" (they have good games against each other, they're different but equally matched in strength, you know the drill).You won't detect this with Noir's method. Why? Simply because the "cross-cathegory" matchups won't show up as much as the "same cathegory" games will (aka the "blegh... dreg vs ic again?" syndrome)... because most players aren't masochists. Every now and then, a newbish comm, a masochist, or someone severely stacked will take a "weak faction" against a "strong faction", but the games will be few and far between (people don't like to be humiliated when they're playing). Therefore, IC's win percentage (under Noir's system) will consist of more ic-dreg games than ic-bios or ic-belters. Because those are actually balanced, they will push the overall win ratio of the faction into the 40-60% range. The occasional "strong vs weak" games will be too few to push it over.
It should be obvious that a system like this won't have "unbalanced win percentages" under Noir's method of interpretation, despite the fact that there's a problem (66% of the possible matchups are actually pre-determined). When faction A is really strong, people will find the one thing that has a reasonable shot at beating it, and then play that matchup whenever someone goes faction A. There will be a slight fluctuation in the overall win ratios as the imbalance is introduced/discovered, but as a counter is found and people will play B whenever the enemy goes A, the overall percentage will slowly go back to normal.
-------------------------------------------------
Ok, so now that I've written 4 paragraphs about how much the current interpretation method sucks, I should probably propose something better, right?
Well, it's simple, really.
Code: Select all
Faction A
-----
Games vs Faction B | Total number | Won | Lost | Drawn | Percentage
Games vs Faction C | Total number | Won | Lost | Drawn | Percentage
Games vs Faction D | Total number | Won | Lost | Drawn | Percentage
Faction B
-----
Games vs Faction A | Total number | Won | Lost | Drawn | Percentage
Games vs Faction C | Total number | Won | Lost | Drawn | Percentage
Games vs Faction D | Total number | Won | Lost | Drawn | Percentage
Faction C
-----
Games vs Faction A | Total number | Won | Lost | Drawn | Percentage
Games vs Faction B | Total number | Won | Lost | Drawn | Percentage
Games vs Faction D | Total number | Won | Lost | Drawn | Percentage
Faction D
-----
Games vs Faction A | Total number | Won | Lost | Drawn | Percentage
Games vs Faction B | Total number | Won | Lost | Drawn | Percentage
Games vs Faction C | Total number | Won | Lost | Drawn | Percentage
A table like this is feasibly reachable from the data the system is gathering. Once you have it, what you're aiming for is a rating within the 40-60% interval in
every line (i.e. every matchup). Once you achieve that, only then will "the win percentages say it's balanced".
(the Total Number column is there because obviously, the accuracy/relevance of a given percentage increases significantly with more plays... so you could set an arbitrary threshold, say, 20 games, before you act on a percentage).
Creating an interpretation system that gives you tables like this shouldn't be overly hard.
If there's interest, I can create one over the summer vacation. I just didn't see the point until now, as everyone refuses to see why Noir's current data interpretation is flawed.
p.s. once the basic framework (the above) is done, I'd probably expand it further, providing a breakdown by settings for each matchup. That would make the system considerably resistant to results skewed due to stacked settings.
p.p.s. the second upgrade I'd implement would probably be a breakdown by team ELOs and Commander win rates.
