Simulation of sampling variability continued...

• Convert random sample values to observer rating on an R-point rating scale

• Repeat for N samples per SN and N event

• Calculate the area under the ROC curve

• Repeat for M replications

• Calculate mean and standard deviation of the area under the ROC curve

• Repeat for different size N

Notes:

Next, I transformed these samples into ratings on a 100 point rating scale so that ROC analysis could be done. I repeated the sampling of size N 100 times so that the variability of the area could be calculated. And finally, I systematically varied the number of samples. Starting with 50 samples per event, I doubled the sample size until there were 819200 samples per event.