Please cite us if you use the software
Example-5 (Sample weights)¶
Environment check¶
Checking that the notebook is running on Google Colab or not.
import sys
try:
import google.colab
!{sys.executable} -m pip -q -q install pycm
except:
pass
Without weights¶
from pycm import ConfusionMatrix
y_test = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]
y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2]
cm1=ConfusionMatrix(y_test, y_pred)
cm1
pycm.ConfusionMatrix(classes: [0, 1, 2])
print(cm1)
Predict 0 1 2 Actual 0 3 0 0 1 0 1 2 2 2 1 3 Overall Statistics : 95% CI (0.30439,0.86228) ACC Macro 0.72222 ARI 0.09206 AUNP 0.66667 AUNU 0.69444 Bangdiwala B 0.37255 Bennett S 0.375 CBA 0.47778 CSI 0.17778 Chi-Squared 6.6 Chi-Squared DF 4 Conditional Entropy 0.95915 Cramer V 0.5244 Cross Entropy 1.59352 F1 Macro 0.56515 F1 Micro 0.58333 FNR Macro 0.38889 FNR Micro 0.41667 FPR Macro 0.22222 FPR Micro 0.20833 Gwet AC1 0.38931 Hamming Loss 0.41667 Joint Entropy 2.45915 KL Divergence 0.09352 Kappa 0.35484 Kappa 95% CI (-0.07708,0.78675) Kappa No Prevalence 0.16667 Kappa Standard Error 0.22036 Kappa Unbiased 0.34426 Krippendorff Alpha 0.37158 Lambda A 0.16667 Lambda B 0.42857 Mutual Information 0.52421 NIR 0.5 NPV Macro 0.79048 NPV Micro 0.79167 Overall ACC 0.58333 Overall CEN 0.46381 Overall J (1.225,0.40833) Overall MCC 0.36667 Overall MCEN 0.51894 Overall RACC 0.35417 Overall RACCU 0.36458 P-Value 0.38721 PPV Macro 0.56667 PPV Micro 0.58333 Pearson C 0.59568 Phi-Squared 0.55 RCI 0.34947 RR 4.0 Reference Entropy 1.5 Response Entropy 1.48336 SOA1(Landis & Koch) Fair SOA2(Fleiss) Poor SOA3(Altman) Fair SOA4(Cicchetti) Poor SOA5(Cramer) Relatively Strong SOA6(Matthews) Weak SOA7(Lambda A) Very Weak SOA8(Lambda B) Moderate SOA9(Krippendorff Alpha) Low SOA10(Pearson C) Strong Scott PI 0.34426 Standard Error 0.14232 TNR Macro 0.77778 TNR Micro 0.79167 TPR Macro 0.61111 TPR Micro 0.58333 Zero-one Loss 5 Class Statistics : Classes 0 1 2 ACC(Accuracy) 0.83333 0.75 0.58333 AGF(Adjusted F-score) 0.9136 0.53995 0.5516 AGM(Adjusted geometric mean) 0.83729 0.692 0.60712 AM(Difference between automatic and manual classification) 2 -1 -1 AUC(Area under the ROC curve) 0.88889 0.61111 0.58333 AUCI(AUC value interpretation) Very Good Fair Poor AUPR(Area under the PR curve) 0.8 0.41667 0.55 BB(Braun-Blanquet similarity) 0.6 0.33333 0.5 BCD(Bray-Curtis dissimilarity) 0.08333 0.04167 0.04167 BM(Informedness or bookmaker informedness) 0.77778 0.22222 0.16667 CEN(Confusion entropy) 0.25 0.49658 0.60442 DOR(Diagnostic odds ratio) None 4.0 2.0 DP(Discriminant power) None 0.33193 0.16597 DPI(Discriminant power interpretation) None Poor Poor ERR(Error rate) 0.16667 0.25 0.41667 F0.5(F0.5 score) 0.65217 0.45455 0.57692 F1(F1 score - harmonic mean of precision and sensitivity) 0.75 0.4 0.54545 F2(F2 score) 0.88235 0.35714 0.51724 FDR(False discovery rate) 0.4 0.5 0.4 FN(False negative/miss/type 2 error) 0 2 3 FNR(Miss rate or false negative rate) 0.0 0.66667 0.5 FOR(False omission rate) 0.0 0.2 0.42857 FP(False positive/type 1 error/false alarm) 2 1 2 FPR(Fall-out or false positive rate) 0.22222 0.11111 0.33333 G(G-measure geometric mean of precision and sensitivity) 0.7746 0.40825 0.54772 GI(Gini index) 0.77778 0.22222 0.16667 GM(G-mean geometric mean of specificity and sensitivity) 0.88192 0.54433 0.57735 HD(Hamming distance) 2 3 5 IBA(Index of balanced accuracy) 0.95062 0.13169 0.27778 ICSI(Individual classification success index) 0.6 -0.16667 0.1 IS(Information score) 1.26303 1.0 0.26303 J(Jaccard index) 0.6 0.25 0.375 LS(Lift score) 2.4 2.0 1.2 MCC(Matthews correlation coefficient) 0.68313 0.2582 0.16903 MCCI(Matthews correlation coefficient interpretation) Moderate Negligible Negligible MCEN(Modified confusion entropy) 0.26439 0.5 0.6875 MK(Markedness) 0.6 0.3 0.17143 N(Condition negative) 9 9 6 NLR(Negative likelihood ratio) 0.0 0.75 0.75 NLRI(Negative likelihood ratio interpretation) Good Negligible Negligible NPV(Negative predictive value) 1.0 0.8 0.57143 OC(Overlap coefficient) 1.0 0.5 0.6 OOC(Otsuka-Ochiai coefficient) 0.7746 0.40825 0.54772 OP(Optimized precision) 0.70833 0.29545 0.44048 P(Condition positive or support) 3 3 6 PLR(Positive likelihood ratio) 4.5 3.0 1.5 PLRI(Positive likelihood ratio interpretation) Poor Poor Poor POP(Population) 12 12 12 PPV(Precision or positive predictive value) 0.6 0.5 0.6 PRE(Prevalence) 0.25 0.25 0.5 Q(Yule Q - coefficient of colligation) None 0.6 0.33333 QI(Yule Q interpretation) None Moderate Weak RACC(Random accuracy) 0.10417 0.04167 0.20833 RACCU(Random accuracy unbiased) 0.11111 0.0434 0.21007 TN(True negative/correct rejection) 7 8 4 TNR(Specificity or true negative rate) 0.77778 0.88889 0.66667 TON(Test outcome negative) 7 10 7 TOP(Test outcome positive) 5 2 5 TP(True positive/hit) 3 1 3 TPR(Sensitivity, recall, hit rate, or true positive rate) 1.0 0.33333 0.5 Y(Youden index) 0.77778 0.22222 0.16667 dInd(Distance index) 0.22222 0.67586 0.60093 sInd(Similarity index) 0.84287 0.52209 0.57508
With random weights¶
from random import randint,seed
seed(100)
weights = [randint(1, 10) for i in range(len(y_test))]
weights[2]*=9
cm2=ConfusionMatrix(y_test, y_pred, sample_weight=weights)
cm2
pycm.ConfusionMatrix(classes: [0, 1, 2])
print(cm2)
Predict 0 1 2 Actual 0 24 0 0 1 0 7 8 2 12 3 76 Overall Statistics : 95% CI (0.75748,0.88868) ACC Macro 0.88205 ARI 0.48768 AUNP 0.82779 AUNU 0.82623 Bangdiwala B 0.73932 Bennett S 0.73462 CBA 0.65617 CSI 0.52442 Chi-Squared 110.9678 Chi-Squared DF 4 Conditional Entropy 0.65034 Cramer V 0.6533 Cross Entropy 1.21 F1 Macro 0.74286 F1 Micro 0.82308 FNR Macro 0.23272 FNR Micro 0.17692 FPR Macro 0.11481 FPR Micro 0.08846 Gwet AC1 0.76652 Hamming Loss 0.17692 Joint Entropy 1.82 KL Divergence 0.04034 Kappa 0.63722 Kappa 95% CI (0.50272,0.77173) Kappa No Prevalence 0.64615 Kappa Standard Error 0.06863 Kappa Unbiased 0.63481 Krippendorff Alpha 0.63621 Lambda A 0.41026 Lambda B 0.52174 Mutual Information 0.5544 NIR 0.7 NPV Macro 0.86908 NPV Micro 0.91154 Overall ACC 0.82308 Overall CEN 0.28807 Overall J (1.82323,0.60774) Overall MCC 0.64625 Overall MCEN 0.38181 Overall RACC 0.51231 Overall RACCU 0.51553 P-Value 0.00097 PPV Macro 0.75714 PPV Micro 0.82308 Pearson C 0.67861 Phi-Squared 0.8536 RCI 0.47399 RR 43.33333 Reference Entropy 1.16966 Response Entropy 1.20474 SOA1(Landis & Koch) Substantial SOA2(Fleiss) Intermediate to Good SOA3(Altman) Good SOA4(Cicchetti) Good SOA5(Cramer) Strong SOA6(Matthews) Moderate SOA7(Lambda A) Moderate SOA8(Lambda B) Moderate SOA9(Krippendorff Alpha) Low SOA10(Pearson C) Strong Scott PI 0.63481 Standard Error 0.03347 TNR Macro 0.88519 TNR Micro 0.91154 TPR Macro 0.76728 TPR Micro 0.82308 Zero-one Loss 23 Class Statistics : Classes 0 1 2 ACC(Accuracy) 0.90769 0.91538 0.82308 AGF(Adjusted F-score) 0.94152 0.68599 0.76783 AGM(Adjusted geometric mean) 0.91704 0.81486 0.81018 AM(Difference between automatic and manual classification) 12 -5 -7 AUC(Area under the ROC curve) 0.9434 0.72029 0.81502 AUCI(AUC value interpretation) Excellent Good Very Good AUPR(Area under the PR curve) 0.83333 0.58333 0.86996 BB(Braun-Blanquet similarity) 0.66667 0.46667 0.83516 BCD(Bray-Curtis dissimilarity) 0.04615 0.01923 0.02692 BM(Informedness or bookmaker informedness) 0.88679 0.44058 0.63004 CEN(Confusion entropy) 0.23219 0.44655 0.28458 DOR(Diagnostic odds ratio) None 32.66667 19.63333 DP(Discriminant power) None 0.83477 0.71286 DPI(Discriminant power interpretation) None Poor Poor ERR(Error rate) 0.09231 0.08462 0.17692 F0.5(F0.5 score) 0.71429 0.63636 0.88993 F1(F1 score - harmonic mean of precision and sensitivity) 0.8 0.56 0.86857 F2(F2 score) 0.90909 0.5 0.84821 FDR(False discovery rate) 0.33333 0.3 0.09524 FN(False negative/miss/type 2 error) 0 8 15 FNR(Miss rate or false negative rate) 0.0 0.53333 0.16484 FOR(False omission rate) 0.0 0.06667 0.32609 FP(False positive/type 1 error/false alarm) 12 3 8 FPR(Fall-out or false positive rate) 0.11321 0.02609 0.20513 G(G-measure geometric mean of precision and sensitivity) 0.8165 0.57155 0.86927 GI(Gini index) 0.88679 0.44058 0.63004 GM(G-mean geometric mean of specificity and sensitivity) 0.9417 0.67416 0.81477 HD(Hamming distance) 12 11 23 IBA(Index of balanced accuracy) 0.98718 0.22395 0.6906 ICSI(Individual classification success index) 0.66667 0.16667 0.73993 IS(Information score) 1.85244 2.6009 0.37018 J(Jaccard index) 0.66667 0.38889 0.76768 LS(Lift score) 3.61111 6.06667 1.29252 MCC(Matthews correlation coefficient) 0.76889 0.52824 0.60381 MCCI(Matthews correlation coefficient interpretation) Strong Moderate Moderate MCEN(Modified confusion entropy) 0.26416 0.4754 0.40758 MK(Markedness) 0.66667 0.63333 0.57867 N(Condition negative) 106 115 39 NLR(Negative likelihood ratio) 0.0 0.54762 0.20737 NLRI(Negative likelihood ratio interpretation) Good Negligible Poor NPV(Negative predictive value) 1.0 0.93333 0.67391 OC(Overlap coefficient) 1.0 0.7 0.90476 OOC(Otsuka-Ochiai coefficient) 0.8165 0.57155 0.86927 OP(Optimized precision) 0.84769 0.56327 0.79836 P(Condition positive or support) 24 15 91 PLR(Positive likelihood ratio) 8.83333 17.88889 4.07143 PLRI(Positive likelihood ratio interpretation) Fair Good Poor POP(Population) 130 130 130 PPV(Precision or positive predictive value) 0.66667 0.7 0.90476 PRE(Prevalence) 0.18462 0.11538 0.7 Q(Yule Q - coefficient of colligation) None 0.94059 0.90307 QI(Yule Q interpretation) None Strong Strong RACC(Random accuracy) 0.05112 0.00888 0.45231 RACCU(Random accuracy unbiased) 0.05325 0.00925 0.45303 TN(True negative/correct rejection) 94 112 31 TNR(Specificity or true negative rate) 0.88679 0.97391 0.79487 TON(Test outcome negative) 94 120 46 TOP(Test outcome positive) 36 10 84 TP(True positive/hit) 24 7 76 TPR(Sensitivity, recall, hit rate, or true positive rate) 1.0 0.46667 0.83516 Y(Youden index) 0.88679 0.44058 0.63004 dInd(Distance index) 0.11321 0.53397 0.26315 sInd(Similarity index) 0.91995 0.62243 0.81392