Off Topic: Know thy Analysis

Meyou · 11-03-2009, 02:05 AM

This thread is for the discussion of my latest article, Off Topic: Know thy Analysis. We would be grateful if you would let us know what you think, but please keep your comments on topic.

Alash · 11-03-2009, 04:33 AM

Yet again a very interesting article. I too have been a victim of Canasta (although my parents didn't go as far as your grandma), so I hear where you're coming from.

One point I think deserves mention in there is that of statistical significance. Your data might be good (it looks good), but due to how playtesting works you don't have a lot of it. And when you start getting more data, you decide to replace a card with another, and have to start all over. Odds are you're rarely testing the exact same deck for more than 15 games, and as such your numbers have a high risk of statistical interference. For instance, drawing five excellent starting hands, something we've all tried, would severely skew the numbers. I think the risk in this case - the case of approaching Magic mathematically - is that of insufficient data, more than anything.

This leads me to my next point. As you said yourself, there's a huge bunch of randomness involved in this game. Number of lands you draw, creatures, die roll, whether or not your opponent topdecks a response etc. etc. Moving from a card game with 52 cards to one of 992 (as of current Standard) makes it virtually impossible to predict what card is in the other guy's hand or library, and even if you manage to memorize the full 150 cards in a matchup, the benefits are questionable. One aspect where the math can provide tangible answers for us, though, is probability. By simulating and analysing data from (say) 2000 starting hands, your picture of mana fixing, mana flood, and mulligan rates reaches a whole new level, and the the only challenge is in setting up the criteria. Once you've done that, not only is it more convenient to analyze, but also incomparably more reliable. While an in-depth analysis of games won with Lightning Bolt certainly has its uses, I don't think it can stand alone - for an efficient analysis, you also need all the major probability rates. And those you just can't get by looking at 15 games.

Combine the two, and you have a very powerful tool.

Triggers · 11-03-2009, 09:12 AM

MWS actually has some pretty decent tools for calculating starting hands.

Pretty interesting article, and I love the deck. I've been playing with Iona, and she is a house.

Starkiller2 · 11-03-2009, 10:03 AM

Iona is definitely better than the Archangel, IMO. Could you PM me the spreadsheet?

Meyou · 11-03-2009, 10:33 AM

Quote:

Originally Posted by Starkiller2

Iona is definitely better than the Archangel, IMO. Could you PM me the spreadsheet?

Will do, but the file is to big to PM. Just send me a message under my profile via email or PM your email.

Since I am at it. FYI: for life gain just use a minus sign.

purduerocks · 11-03-2009, 12:32 PM

I definitely enjoyed the article, pm is sent for the spreadsheet. I have been toying with your decklist, and I have been trying this deck list

Deck
4 Rampant Growth 4 Khalni Heart Expedition 3 Trapmaker's Snare 4 Harrow 4 Polymorph 3 Garruk Wildspeaker 4 Ant Queen / Martial Coup 4 Summoning Trap 4 Sphinx of the Steel Wind 2 Iona, Shield of Emeria 4 Misty Rainforest 2 Scalding Tarn 4 Plains 2 Island 2 Swamp 10 Forest

The polymorph changes could even be sideboard options, but with this set up you can use your summoning traps to get the large creatures, or destroy a token to get to polymorph a creature. It leaves multiple options so that you can get the big creatures potentially quicker. It is doing well against the Bloodquake/Unearthquake deck that is destroying the metagame around my area. I would love to see this deck become competitive.

SeanNZ · 11-03-2009, 01:55 PM

I've found this article, and the previous one, very interesting as I am just starting to get into more serious playtesting. It's great to get more ideas on how to approach testing new deck ideas. It's a pity this topic isn't discussed more often since it's so important, particularly for those of us that enjoy creating our own decks.

As mentioned above, MWS has some reasonable tools for analysing starting hands, giving info on lands drawn and how many 1 drops etc. which may be of some use.

The discussion of intuition carrying over from other games into magic reminded me of an article called "The Expert Mind" from Scientific American, which I believe can be found online. The article actually suggests learned intuition or expertise is very specific (i.e. not necessarily transferable). I'm sure someone will find that an interesting read

Thanks again for an interesting article.

ColonelCoo · 11-03-2009, 03:32 PM

Quote:

Originally Posted by Starkiller2

Iona is definitely better than the Archangel, IMO. Could you PM me the spreadsheet?

againts Red, the Archangel just wins. Oddly, Iona still falls to Valakut, an existing Chandra or splashed White (PTE).
the archangel also wins against Vampire.

What is the email address of the writer? I didn't see it anywhere. I've been toying with Big Green Trap and was wondering if that is his deck. (I played on it on gameday to a #2 position: lost to 5cc)

I am also curious if the deck is hot-swappable with a different deck approach to the game instead of sideboarding against threats. I did a hot-swap Bant sideboard that took Game 2 every time. Actually, the only games I lost were to 5cc control (typically losing game 1, winning game 2 and losing game 3)

creatures
bird of paradise x4
noble heirarch x4
lotus cobra x4
emperial archangel x2
rafiq of the many x1
iona shield of emeria x2
platinum angel x2
rampaging baloth x1
baneslayer angel x1 (I only have 1)
acidic slime x3

2 trap finders, 4 traps.

Hot swap all 6 traps, and 9 creatures (leave Rafiq up in the deck, all others out) for 15 bant(ish) creatures. Generally it worked like a charm. Game one won with the big bad mothers. Game two with the new Bant horder (including dauntless escort and knight of new alara)

chizarlicious · 11-03-2009, 04:10 PM

You can actually do a lot when you apply computational data analysis to your playtesting. A while ago I was looking at the viability of adding Manamorphose, Street Wraith, and Edge of Autumn into my extended All-In Red deck to increase the odds of getting a first turn Deus of Calamity. It's obvious that those cards all increase the odds, but figuring out exactly how much help they are is really difficult.

I ended up writing a program in C++ to fishbowl the deck (which doesn't really interact with opponents anyway) hundreds of millions of times.

Then Conflux came out, and I gave the deck up because it scooped to Path to Exile.

Meyou · 11-03-2009, 04:46 PM

Quote:

Originally Posted by ColonelCoo

What is the email address of the writer? I didn't see it anywhere. I've been toying with Big Green Trap and was wondering if that is his deck. (I played on it on gameday to a #2 position: lost to 5cc)

You can send me an email or pm under my profile.

It is my deck. Or should I say, I am the first one to publish it online in a mass media outlet. I'm sure there has been others trying to break Summoning Trap. Just as I am sure plenty of people, including myself, saw the Vampire Hexmage and quickly thought of Dark Depths.

Quote:

Originally Posted by ColonelCoo

I am also curious if the deck is hot-swappable with a different deck approach to the game instead of sideboarding against threats. I did a hot-swap Bant sideboard that took Game 2 every time. Actually, the only games I lost were to 5cc control (typically losing game 1, winning game 2 and losing game 3)

Interesting idea of being able to hot swap the deck.

HeroTheDragon · 11-03-2009, 11:07 PM

I like your column, but I have to ask--did this article undergo any editing? It felt like grammarless rambling at times, and there were a lot of confusing errors.

ColonelCoo · 11-03-2009, 11:34 PM

Quote:

Originally Posted by Meyou

You can send me an email or pm under my profile.

It is my deck. Or should I say, I am the first one to publish it online in a mass media outlet. I'm sure there has been others trying to break Summoning Trap. Just as I am sure plenty of people, including myself, saw the Vampire Hexmage and quickly thought of Dark Depths.

Interesting idea of being able to hot swap the deck.

I'll shoot you a PM in the AM.

My main deck was trying to get the 6 mana every game on turn 3. Inthe end I liked gettign my brids and nobles killed so that the removal didn't target my big stuff.

But having two Emperial Angels, Rafiq and 4 nobles along with the 4 birds main-deck made me really think hard on the sideboard. A squished version of Bant emerged (3 pridemage, 2 knights, 3 Monks, etc). In the end the change up of the deck dsign made them hold back and make mistakes.

BTW, if you've not looked at Lorthos, he is game finisher when trapped in at EOT. But you need to be playing the full suit eof mana placement to be sure to have 8 mana on turn 4 to have him pay off. He's his own oneway stasis.

lilnev · 11-03-2009, 11:43 PM

Mulligans: I generally play six card hands, but if I would mull to five I'll go back to seven again instead. Yes, five card hands (or worse!) sometimes happen in the real world, and if the goal were to obtain the most accurate estimate of the match percentage, then I'd play them out. But the point of playtesting is actually to get the most improvement per hour spent. That means (1) identifying changes to the decklist that would make it better, and (2) learning how to play the matchup -- which cards are important, what are the key decisions and how should I make them, what is my route to victory, etc. I don't feel that playing out five card hands will do as much to improve those things as some other use of my time. I do play six card hands, because they come up a lot, they help me to understand how well the deck can perform down a card, and they help me develop a better sense of the mulligan decision itself. Closer to game day, with a fully finalized list, I might play out some five card hands just to get a feel for them.

Overall, I'm less concerned with stats than I am with "getting a feel" for what the key cards are in each matchup. After a 10 game set, my notes might read: "7-3, but I was getting lucky. Maybe even. Key cards are X and Y, which should be held for kicker if possible. I want better answers to Z." I'm probably going to be tweaking the list later anyway. And when it comes time to build a sideboard, this kind of information is more useful than knowing the precise game 1 win percentage.

Big Jim · 11-03-2009, 11:55 PM

I kind of agree with Alash, but I think the problem of reporting play testing results is much deeper. As Alash says, the sample size, or the number of individual play testing games is usually quite low (<30). Why is this a problem? Variance. Think about it this way, in a given game I might win or lose, but I also might win by a lot or lose by a lot. If I'm winning by a lot each time, I can be more confident in my wins, but if I'm only winning by a little, I am less confident. In other words, while we measure games by who wins and loses, winning and losing is actually a flawed measure of outcome.

This, of course, doesn't make intuitive sense at first. But remember those games where you were clearly in control until one card ruins your plans? In that game, the result may not have been the best measure of how well each player was doing. "But Big Jim," You say, "I don't care who was doing better, I only care about winning." That is true, and it points to the difference between "playing" and "play testing". When we play games, of course we care about winning, but when we play test, we are trying to determine who is likely to win.

In the end, the result of a game is a pretty good, but ultimately imperfect measure of who is likely to win a given game. Yes, if I know I will win a game, it is a perfect measure, but the goal of play testing is to infer a likelihood of winning from a pattern of previous results, and in that case, I can't predict the future. In other words, I have to make an inference, and because typically people refer purely to win percentages, their estimates don't account for error in measurement (i.e., they don't account for the chance that you could have lost games you won and vice-versa).

So back to the original point of small sample sizes...in small sample sizes, the variance can be quite large, meaning you have to be less confident in your results.

This brings up a point probably many of you are thinking, "How can we determine if I am winning a game at a particular point?" This is a complex issue. Many have theories; most of us understand the life totals don't mean that much, so we use things like tempo, card advantage, and other concepts to explain game advantage at a given time. These are both flawed measures as well as difficult to measure.

The last thing I'll mention in this rant is that the typical way we compare decks, by comparing percentages, is a poor way of comparing relative performance because it doesn't account for variance. Basically, just comparing percentages is a bad way to determine if one deck is better than another at beating say RDW because we don't know by how much each deck wins. This could be why people often argue about win percentages; one guy says deck X beats RDW 65% of the time in his results, another guy says it only wins 45% of the time...guess what? They could both be right. It just might be that the variance in Deck X's success accounts for the divide. So where is the true win percentage? Good question.

Don't even get me started on the variance introduced by different players playing the same deck...I mean, all competent players play RDW the exact same way, right? Right?...

ColonelCoo · 11-04-2009, 09:13 AM

Quote:

Originally Posted by Big Jim

. . .

Don't even get me started on the variance introduced by different players playing the same deck...I mean, all competent players play RDW the exact same way, right? Right?...

You're dead on there. I had been playtesting a deck (Octo-mom) that heavily investedin Chandra's (8 of them, hence the name). In the end I ran 5 and just called it a chadra burn deck.

Anyhow, to your point: I played this deck extensively against a pool of decks in playtest. It just ate vampires, goes nuts on Planeswalkers, and generally causes headaches for Blightning players. It also had a pretty easy time against stall-control decks (who had typically ran a lot of Sorcery speed removal).

So I give the deck to a friend for Game Day. He runs it and finally gets a match win (1-4) at the end: losing to vampires and Super-Friends.

All in the play.

11-03-2009, 02:05 AM	#1
Meyou Experienced Mage Join Date: Feb 2009 Location: Cannon Falls, MN Posts: 74	Off Topic: Know thy Analysis This thread is for the discussion of my latest article, Off Topic: Know thy Analysis. We would be grateful if you would let us know what you think, but please keep your comments on topic.

11-03-2009, 10:03 AM	#4
Starkiller2 HATEFLAYER Join Date: Dec 2008 Location: My Computer. Duh. Posts: 1,760	Iona is definitely better than the Archangel, IMO. Could you PM me the spreadsheet? __________________ Thanks to SB4 at High-Light Studios for the sig and avvie! Currently Running: Standard JAWs Casual: Rogues Currently Building: Martyr Foundry Uril EDH Luminarch Enchantress Last edited by Starkiller2 : 11-03-2009 at 10:03 AM. Reason: .<?

Thread Tools	Search this Thread
Show Printable Version Email this Page	User Name: Search this Thread: Advanced Search


Home Articles Zendikar Spoiler (249/249) Radar Forums Blogs Wiki Writing/Contests Chat About

11-03-2009, 04:33 AM	#2
Alash Ascended Mage Join Date: Oct 2009 Posts: 192	Yet again a very interesting article. I too have been a victim of Canasta (although my parents didn't go as far as your grandma), so I hear where you're coming from. One point I think deserves mention in there is that of statistical significance. Your data might be good (it looks good), but due to how playtesting works you don't have a lot of it. And when you start getting more data, you decide to replace a card with another, and have to start all over. Odds are you're rarely testing the exact same deck for more than 15 games, and as such your numbers have a high risk of statistical interference. For instance, drawing five excellent starting hands, something we've all tried, would severely skew the numbers. I think the risk in this case - the case of approaching Magic mathematically - is that of insufficient data, more than anything. This leads me to my next point. As you said yourself, there's a huge bunch of randomness involved in this game. Number of lands you draw, creatures, die roll, whether or not your opponent topdecks a response etc. etc. Moving from a card game with 52 cards to one of 992 (as of current Standard) makes it virtually impossible to predict what card is in the other guy's hand or library, and even if you manage to memorize the full 150 cards in a matchup, the benefits are questionable. One aspect where the math can provide tangible answers for us, though, is probability. By simulating and analysing data from (say) 2000 starting hands, your picture of mana fixing, mana flood, and mulligan rates reaches a whole new level, and the the only challenge is in setting up the criteria. Once you've done that, not only is it more convenient to analyze, but also incomparably more reliable. While an in-depth analysis of games won with Lightning Bolt certainly has its uses, I don't think it can stand alone - for an efficient analysis, you also need all the major probability rates. And those you just can't get by looking at 15 games. Combine the two, and you have a very powerful tool. Last edited by Alash : 11-03-2009 at 04:36 AM.

11-03-2009, 09:12 AM	#3
Triggers Experienced Mage Join Date: Jan 2009 Posts: 91	MWS actually has some pretty decent tools for calculating starting hands. Pretty interesting article, and I love the deck. I've been playing with Iona, and she is a house.

11-03-2009, 01:55 PM	#7
SeanNZ Just Getting Started Join Date: Nov 2009 Posts: 1	I've found this article, and the previous one, very interesting as I am just starting to get into more serious playtesting. It's great to get more ideas on how to approach testing new deck ideas. It's a pity this topic isn't discussed more often since it's so important, particularly for those of us that enjoy creating our own decks. As mentioned above, MWS has some reasonable tools for analysing starting hands, giving info on lands drawn and how many 1 drops etc. which may be of some use. The discussion of intuition carrying over from other games into magic reminded me of an article called "The Expert Mind" from Scientific American, which I believe can be found online. The article actually suggests learned intuition or expertise is very specific (i.e. not necessarily transferable). I'm sure someone will find that an interesting read Thanks again for an interesting article.

11-03-2009, 04:10 PM	#9
chizarlicious Just Getting Started Join Date: Jul 2006 Location: Minnesota Posts: 22	You can actually do a lot when you apply computational data analysis to your playtesting. A while ago I was looking at the viability of adding Manamorphose, Street Wraith, and Edge of Autumn into my extended All-In Red deck to increase the odds of getting a first turn Deus of Calamity. It's obvious that those cards all increase the odds, but figuring out exactly how much help they are is really difficult. I ended up writing a program in C++ to fishbowl the deck (which doesn't really interact with opponents anyway) hundreds of millions of times. Then Conflux came out, and I gave the deck up because it scooped to Path to Exile.

11-03-2009, 11:43 PM	#13
lilnev Just Getting Started Join Date: May 2009 Posts: 19	Mulligans: I generally play six card hands, but if I would mull to five I'll go back to seven again instead. Yes, five card hands (or worse!) sometimes happen in the real world, and if the goal were to obtain the most accurate estimate of the match percentage, then I'd play them out. But the point of playtesting is actually to get the most improvement per hour spent. That means (1) identifying changes to the decklist that would make it better, and (2) learning how to play the matchup -- which cards are important, what are the key decisions and how should I make them, what is my route to victory, etc. I don't feel that playing out five card hands will do as much to improve those things as some other use of my time. I do play six card hands, because they come up a lot, they help me to understand how well the deck can perform down a card, and they help me develop a better sense of the mulligan decision itself. Closer to game day, with a fully finalized list, I might play out some five card hands just to get a feel for them. Overall, I'm less concerned with stats than I am with "getting a feel" for what the key cards are in each matchup. After a 10 game set, my notes might read: "7-3, but I was getting lucky. Maybe even. Key cards are X and Y, which should be held for kicker if possible. I want better answers to Z." I'm probably going to be tweaking the list later anyway. And when it comes time to build a sideboard, this kind of information is more useful than knowing the precise game 1 win percentage.

11-03-2009, 11:55 PM	#14
Big Jim Ascended Mage Join Date: Jan 2008 Location: Portland, OR Posts: 325	I kind of agree with Alash, but I think the problem of reporting play testing results is much deeper. As Alash says, the sample size, or the number of individual play testing games is usually quite low (<30). Why is this a problem? Variance. Think about it this way, in a given game I might win or lose, but I also might win by a lot or lose by a lot. If I'm winning by a lot each time, I can be more confident in my wins, but if I'm only winning by a little, I am less confident. In other words, while we measure games by who wins and loses, winning and losing is actually a flawed measure of outcome. This, of course, doesn't make intuitive sense at first. But remember those games where you were clearly in control until one card ruins your plans? In that game, the result may not have been the best measure of how well each player was doing. "But Big Jim," You say, "I don't care who was doing better, I only care about winning." That is true, and it points to the difference between "playing" and "play testing". When we play games, of course we care about winning, but when we play test, we are trying to determine who is likely to win. In the end, the result of a game is a pretty good, but ultimately imperfect measure of who is likely to win a given game. Yes, if I know I will win a game, it is a perfect measure, but the goal of play testing is to infer a likelihood of winning from a pattern of previous results, and in that case, I can't predict the future. In other words, I have to make an inference, and because typically people refer purely to win percentages, their estimates don't account for error in measurement (i.e., they don't account for the chance that you could have lost games you won and vice-versa). So back to the original point of small sample sizes...in small sample sizes, the variance can be quite large, meaning you have to be less confident in your results. This brings up a point probably many of you are thinking, "How can we determine if I am winning a game at a particular point?" This is a complex issue. Many have theories; most of us understand the life totals don't mean that much, so we use things like tempo, card advantage, and other concepts to explain game advantage at a given time. These are both flawed measures as well as difficult to measure. The last thing I'll mention in this rant is that the typical way we compare decks, by comparing percentages, is a poor way of comparing relative performance because it doesn't account for variance. Basically, just comparing percentages is a bad way to determine if one deck is better than another at beating say RDW because we don't know by how much each deck wins. This could be why people often argue about win percentages; one guy says deck X beats RDW 65% of the time in his results, another guy says it only wins 45% of the time...guess what? They could both be right. It just might be that the variance in Deck X's success accounts for the divide. So where is the true win percentage? Good question. Don't even get me started on the variance introduced by different players playing the same deck...I mean, all competent players play RDW the exact same way, right? Right?...