ROC plots receiver operator curve and computes derived statistics.
   computes the ROC curve, and a number of derived paramaters include
   AUC, optimal threshold values, corresponding confusion matrices, etc.

 Remark: if the sample values in d are not unique, there is a certain
   ambiguity in the results; the results may vary depending on
   on the ordering of the samples. Usually, this is only an issue,
   if the number of unique data value is much smaller than the total
   number of samples.

 Tratitionally, ROC was defined in the "Biosig for Octave and matlab"
   toolbox, later an ROC function became available in Matlab's NNET
   (Deep Learning) toolbox with a different usage interface.
   Therfore, there are different usage-styles.

 Usage (traditional/biosig style):
   RES = roc(d, c);
   RES = roc(d1, d0);
   RES = roc(...);

   RES = roc(...,'flag_plot');
   RES = roc(..., s);
	plot ROC curve, including suggested thresholds
       In order to speed up the plotting, no more than 10000 data
       points are displayed. If you need more, you need to change
       the source code).

   The ROC curve can be plotted with
       plot(RES.FPR*100, RES.TPR*100);

 Usage style compatible with matlab's roc implementation:
   [TPR, FPR, THRESHOLDS] = ROC(targets, outputs)
        matlab-style interface for compatibiliy with Matlab's ROC implementation;
        Note that the input arguments are reversed;
        targets correspond to c, and outputs correspond to d.

 INPUT:
 d	DATA,
 c	CLASS, vector with 0 and 1
 d1	DATA of class 1
 d2	DATA of class 0
 s	line style (as used in plot)
 targets  DATA, when using matlab-style ROC
 outputs  CLASS when using matlab-style ROC

 OUTPUT:
   TPR   true positive rate
   FPR   false positive rate
   THRESHOLDS  corresponding Threshold values
   ACC     accuracy
   AUC     area under ROC curve
   Yi	  max(SEN+SPEC-1), Youden index
   c	  TH(c) is the threshold that maximizes Yi

   RES is a structure and provides many more results
       including optimum threshold values, correpinding confusion matrices, etc.
   RES.THRESHOLD.FPR returns the threshold value to obtain
	the given FPR rate.
   RES.THRESHOLD.{maxYI,maxACC,maxKAPPA,maxMCC,maxMI,maxF1,maxPHI} return the
	threshold obtained from maximum Youden Index (YI), Accuracy, Cohen's Kappa [3],
       Matthews correlation coefficient [2] (also known as Phi coefficient [1]),
       Mutual information, and F1 score [4], resp.
   RES.TH([RES.THRESHOLD.maxYIix, RES.THRESHOLD.maxACCix, RES.THRESHOLD.maxKAPPAix,
             RES.THRESHOLD.maxMCCix, RES.THRESHOLD.maxMIix, RES.THRESHOLD.maxF1ix])
       return the optimal threshold for the respective measure.
   RES.H_kappa: confusion matrix when Threshold of maximum Kappa is applied.
   RES.H_{yi,acc,kappa,mcc,mi,f1,phi}: confusion matrix when threshold of
       optimum {...} is applied. Its structure is [TN, FN; FP; TP].

 see also: AUC, PLOT, ROC

 References:
 [0] https://en.wikipedia.org/wiki/ROC_curve
 [1] https://en.wikipedia.org/wiki/Phi_coefficient
 [2] https://en.wikipedia.org/wiki/Matthews_correlation_coefficient
 [3] https://en.wikipedia.org/wiki/Cohen%27s_kappa
 [4] https://en.wikipedia.org/wiki/F1_score
 [5] A. Schlögl, J. Kronegg, J.E. Huggins, S. G. Mason;
     Evaluation criteria in BCI research.
     (Eds.) G. Dornhege, J.R. Millan, T. Hinterberger, D.J. McFarland, K.-R.Müller;
     Towards Brain-Computer Interfacing, MIT Press, 2007, p.327-342

Package: nan