Original content created by community member: Trilwych

Important lessons:

1. Any position on probability that is based on a small sample size is automatically invalid. :)
2. Improbable doesn't mean impossible. Oh, and improbable doesn't mean impossible.
3. The lack of the improbable is a better indication of insufficient randomness than the presence of the improbable.
4. WAI.
   

The Three Thousand Club

Cumulative Results for Cooking at a 17% Critical Probability: Sample size 3497

By Trilwych

Original thread: "Sample size: THREE THOUSAND"

Special comment: "Crit RATE" vs. "Crit CHANCE"--It's been noted elsewhere that rate and chance are actually two very different things, and this is true. The problem is that the majority of players without any knowledge of probability conflate the two, which is how I treated the terms in my experiment. Technically, it's supposed to be crit chance due to how the RNG works for every roll.

Hypothesis

Across a time-cumulative, large sample (numbering in the thousands at minimum), the actual accumulated crit-vs-normal crafting results will align with the crafter’s advertised crit rate.

Methods & Materials

A combination of Farmer/Cook professions was determined to be the most efficient method to test a large number of crafting rolls. The cumulative sample types for the level 43 supreme master cook would consist of- 

• Cups of Winter Barley Flour
• Balls of Dough
• Honey-Cakes

The intended sample size was 3000, divided across the above sample types and crafted across three days in 9 total crafting sessions. The actual total sample size was 3497 across 11 sessions (yeah, I lied; explanation at the bottom). The cook had 17% native crit chance between mastery (5%) and the superior cooking supplies (12%). The cook toon was level 43 at the time of this writing and thus could only equip that level of tool. No optional critical ingredients nor Scholar-created crafting scrolls were used. All results were logged by placing the /Standard output channel into its own tab. Results were parsed in Excel 2010 and summarily graphed in pretty colors.(Further details after Results.)

Results

I'll let the results speak for themselves and not bother with analysis and conclusion.

Nitty Gritty

My farmer toon crafted 3000 winter barley across three days (1000 per day); costs were not counted. My cook bought with kinship discount the following materials for a 3-day sample spread: 

• 750 bottles of water
• 750 drops of fine clover honey
• 750 pinches of mixed spice

After the initial 3000 cumulative rolls were completed, additional materials were purchased to convert the remainder ingredients produced by critical success from the previous crafting sessions.

• Cooking costs for three days’ worth of materials covering 3000 items, not including repairs: 4g 814s 50c
• Cooking costs for the bonus round, not including repairs: 1g 605s 88c 
• Total costs for experiment, not including repairs or porting: 6g 420s 38c

Definitions

• RNG: The computerized random number generator behaves, theoretically, like a die. With a 17% crit rate, this is like taking a 100-sided die and rolling it. If the die lands on 1-17, you get a critical success. If the die lands on 18-100, then you get a normal result. Thus, even with a known percentage “crit rate,” the actual results are still “random.”
• Channel (channeling): A timed act with the blue timer bar (such as certain class skills, or mapping, or in this case creating a single craft item)
• Session: A crafting session is defined here as an uninterrupted series of “make all” channeling.
• Roll: A “roll” here is defined as a single recipe channeling act that creates a final product(s), with the RNG applying at the end of the channel.
• Crit rate: The crit rate indicates the probability for the crafter to produce a critical success result for every individual crafting roll. The crit rate does not apply to a total cumulative sample per se. With an 83% crit rate, it is possible to fail a dozen times in a row; the probability of that happening is low, but not impossible.
• Ingredient: In the Cooking profession, an “ingredient” is a crafted component necessary to complete other cook recipes to produce edible food items. In this case, both “Cup of Winter Barley Flour” and “Ball of Dough” are ingredients falling under my cook’s 17% crit rate.
• Trail Food: In the Cooking profession, Trail Food is produced from recipes that require various ingredients and a campfire. Trail Food typically gives the consumer a specific stat buff for a 20- or 30-minute length of time. In this case, Honey-cakes were chosen as the testing product.

Both crafting window and vendor windows were open during every session; the crafting window’s selection was set to whatever item I was making at the time. Superior cooking supplies were repaired whenever they fell to around 40-45/60 without interrupting a crafting session.(*) The purpose for repairing in this manner was twofold: first to ensure that the cook would not go “afk” and possibly log off after a long period of alt-tabbed crafting, and second to ensure the tools wouldn’t break and inadvertently stop a session.

(*) This could be a bug; vendor interaction is possible after a crit during a single crafting session.
 

Additional areas to be addressed: Streaks  

Here are some basic streak results; I don't know of any way to graph this sort of thing, especially with a data set around 3500. Be aware that I'm counting a "streak" as any identical, adjacent cell, so even a 2-string is a streak here. Keep in mind that my crafter's crit chance was 17%, so getting something like 4 crits in a row on that low probability is quite unlikely. And yet it happened thrice. Therefore, the inverse is true too--with a low failure probability (say 17% which is a 83% crit rate), a crafter can still end up failing a bunch of times in a row.

Mediafire link for ZIP of raw logs.

Long-term Cross-Crafting Analysis of Critical Success Rate: Sample size 3650

By demosthenes

3650 crafting attempts across a wider range of professions (Mostly Prospector, Forrester, Jeweler, Tailor, plus a few Weaponsmith and Cook results) over 90 days. I obtained a similar result to yours, where the actual critical rate was .08% higher than the theoretical rate - ie pretty much an exact match.

Here are my original results plotted over time. Note how its only after 500 attempts that there is any convergence, and that good long term convergence occurs after 2500 tests. I checked my data and I actually ran the test in 2008 but presented the results in Jan 2009.

Results

The One Thousand Club

Cumulative Results for Jeweller at a 62% Probability: Sample size 1000

by Trilwych

This is basically an extension of the original experiment (3000 sample above). I decided to post this in similar style but separately because the sample size is the minimum I'd personally accept for casual testing purposes like this. 

Original thread: "Sample size: ONE THOUSAND"

Hypothesis

Across a time-cumulative large sample (four-digit minimum), application of the optional crit ingredient will result in an expected percentage that aligns with the crafter’s advertised crit probability.

Methods & Materials 

My supreme master Tinker (started at level 43, ended at level 46) tested 1000 jeweller crafting rolls across T1 apprentice through T7 Supreme, during the period of 30JUL2011 through 17SEPT2011. In every single roll, the superior level 35 jeweller tools were used along with a salt of the appropriate tier, giving a final probability of 62% for a crit. The sample was spread across 60 sittings (60 discrete logs), and each individual recorded sitting ranged from a sample size of 2 up to 44; the average individual sample was 17 per sitting.

The intended sample size was an ideal 2000, but due to a bottleneck of resources (gems and salts specifically cannot be farmed easily), I stopped at 1000. All results were logged by placing the /Standard output channel into its own tab. Results were parsed in Excel 2010 and summarily graphed in pretty colors.

Results

Due to the higher number of sittings compared to the cooking experiment, this was difficult to graph in the usual "100 ratio" style I like, so I had to be a little creative. The row on the very bottom of the graph is the expected 62%; the final actual percent of crit-vs-normal came out to be 62.4%. From the very jagged appearance of the green/blue areas, you can see that streaks and random chance are all over the place, yet the total percent came very close to the expected value.

Because each sitting on the graph represents different actual sample sizes (the very top row is actually only two rolls, making a sample of 2), keep in mind that the "weight" for every sitting will be different and is not captured effectively in the graph. For example, a sitting of 40 rolls that gives 72.5% is going to have much more weight than a sitting of 2 rolls giving 50%.

Nitty Gritty

Craft sittings were not consistent--they occurred at various hours of the day and night for over 1.5 months, with each recorded sitting varying from 2 rolls to over 40. I did not interact much with the vendors in this experiment compared to the cooking experiment and did not consistently repair tools (i.e., I forgot). The crafting window was open during every sitting, though I may or may not have had a vendor window open as well. There was no horseshoe involved. 

Crafting rolls included tokens and gems, which both stacked, and all other manner of usable final products. The crit versions were sold to fund further accumulation of necessary materials (ores, ingots, gems, salts). I didn't keep track of costs compared to the cooking experiment, but I think I came out ahead if not even. The number of rolls per jeweller tier was extremely dependent on the number of salts I had at the time per that tier, and I was unwilling to pay beyond a certain amount per salts at the auction house.

Definitions

• RNG: The computerized random number generator behaves, theoretically, like a die. Even with a known percentage "crit rate," the actual results are still "random."
• Crit rate: The crit rate indicates the probability for the crafter to produce a critical success result for every individual crafting roll. The crit rate does not apply to a total cumulative sample per se. With an 83% crit rate, it is possible to fail a dozen times in a row; the probability of that happening is low, but not impossible. Please note that I conflate the terms "crit rate" and "crit probability" (the latter is accurate) because a lot of players do that, and I have to be consistent with my previous experiment.
• Channel (channeling): A timed act with the blue timer bar (such as certain class skills, or mapping, or in this case creating a single craft item); only gems and tokens can be channeled in the jeweller profession.
• Session: For consistency with my previous experiment, a crafting session is defined here as an uninterrupted series of "make all" channeling. For this jeweller experiment, sessions were mixed into sittings whenever possible to increase the sample size. For practical reasons (limited resources, limited space, etc.), I never did a complete "make all" session that would convert large amounts of material into a single product type.
• Sitting: A sitting is essentially a single recorded log. Each sitting varied in sample size, though I started out aiming for 10 per sitting. In order to prevent too much log contamination, if I started a recorded craft sitting and realized I needed more of something, I would stop recording (which creates a discrete log) and begin logging again only after I got what I needed. This means that for every login, I could have multiple recorded sittings.
• Roll: A "roll" here is defined as a single product channeling act that creates a final product(s), with the RNG applying at the end of the channel.

Additional Discussion and Basic Streak Data  

I do not expect Rise of Isengard to change the actual underlying critical mechanism of crafting. The only changes that I know of apply to the forthcoming Tier 7, and then mainly apply only to the gathering and refining aspects.  

I also assume that this experiment applies to all other crafting professions that take crit ingredients. My reasoning is that, from a programming perspective, using different code for different crit objects in different professions would be inefficient; thus, it would make sense that the underlying code behaves the same way across professions, also taking into consideration that the game is built on top of an existing, generic RNG mechanism as well.

Bottlenecks--salts and gems, but salts were the biggest bottleneck, and I don't have the patience to farm the AH for another 1000 sample. My tinker already had quite a few banked salts and gems because I was not terribly active in her jeweller profession; I might remember to make a few things on a random weekend, basically. But after a certain point, trying to find salts for the tiers where I had an excess of precious metals became a little frustrating.

Basic Streak Data 

Crit streaks

Median: 3
Mean: 3.266
Max: 2 streaks of 10 crits

Noncrit streaks
Median: 2
Mean: 2.561
Max: 2 streaks of 5 noncrits (Huh.)

Mediafire link for logs.
 

Other Reading 

Provided by demosthenes
http://www-psych.stanford.edu/~bigopp/randomness.html
http://web.mit.edu/cocosci/Papers/random.pdf
http://www.usc.edu/CSSF/History/2006/Projects/J0305.pdf
http://etd.lsu.edu/docs/available/etd-07092004-113059/unrestricted/Armstrong_thesis.pdf
http://psycnet.apa.org/index.cfm?fa=buy.optionToBuy&id=2003-00781-006
Google Books: (keywords) memory about negative events versus positive
http://csjarchive.cogsci.rpi.edu/proceedings/2010/papers/0382/paper0382.pdf
http://www.hbs.edu/research/pdf/09-029.pdf