NLnet foundation has supported the PyCM project from version 3.6 to 4.0 through the NGI Assure Fund. This fund is set up by NLnet foundation with funding from the European Commission's Next Generation Internet program, administered by DG Communications Networks, Content, and Technology under grant agreement No 957073.
Python Software Foundation (PSF) grants PyCM library partially for version 3.7. PSF is the organization behind Python. Their mission is to promote, protect, and advance the Python programming language and to support and facilitate the growth of a diverse and international community of Python programmers.
Papers and Reports
- PyCM: Multiclass confusion matrix library in Python - May 2018
- A Literature Review on Rater Agreement Metrics - Sep 2022
Version 4.2
- 5 new distance/similarity added
- KuhnsIII
- KuhnsIV
- KuhnsV
- KuhnsVI
- KuhnsVII
- Test system modified
- PyPI badge in
README.md
- GitHub actions are limited to the
dev
andmaster
branches AUTHORS.md
updatedREADME.md
modified- Document modified
Source Code (.Zip) (1.37 MB)
Source Code (.tar.gz) (1.33 MB)
Version 4.1
- 5 new distance/similarity added
- KoppenI
- KoppenII
- KuderRichardson
- KuhnsI
- KuhnsII
feature_request.yml
template addedconfig.yml
for issue template addedSECURITY.md
added- Bug report template modified
thresholds_calc
function updated__midpoint_numeric_integral__
function updated__trapezoidal_numeric_integral__
function updated- Diagrams updated
- Document modified
- Document build system updated
AUTHORS.md
updatedREADME.md
modified- Test system modified
Python 3.12
added totest.yml
Python 3.13
added totest.yml
- Warning and error messages updated
pycm_util.py
renamed toutils.py
pycm_test.py
renamed tobasic_test.py
pycm_profile.py
renamed toprofile.py
pycm_param.py
renamed toparams.py
pycm_overall_func.py
renamed tooverall_funcs.py
pycm_output.py
renamed tooutput.py
pycm_obj.py
renamed tocm.py
pycm_multilabel_cm.py
renamed tomultilabel_cm.py
pycm_interpret.py
renamed tointerpret.py
pycm_handler.py
renamed tohandlers.py
pycm_error.py
renamed toerrors.py
pycm_distance.py
renamed todistance.py
pycm_curve.py
renamed tocurve.py
pycm_compare.py
renamed tocompare.py
pycm_class_func.py
renamed toclass_funcs.py
pycm_ci.py
renamed toci.py
Source Code (.Zip) (1.34 MB)
Source Code (.tar.gz) (1.31 MB)
Version 4.0
pycmMultiLabelError
class addedMultiLabelCM
class addedget_cm_by_class
method addedget_cm_by_sample
method added__mlcm_vector_handler__
function added__mlcm_assign_classes__
function added__mlcm_vectors_filter__
function added__set_to_multihot__
function addeddeprecated
function added- Document modified
README.md
modified- Example-4 modified
- Test system modified
- Python 3.5 support dropped
Source Code (.Zip) (943 KB)
Source Code (.tar.gz) (908 KB)
Version 3.9
OVERALL_PARAMS
dictionary added__imbalancement_handler__
function addedvector_serializer
function added- NPV micro/macro added
log_loss
method added- 23 new distance/similarity added
- Dennis
- Digby
- Dispersion
- Doolittle
- Eyraud
- Fager & McGowan
- Faith
- Fleiss-Levin-Paik
- Forbes I
- Forbes II
- Fossum
- Gilbert & Wells
- Goodall
- Goodman & Kruskal's Lambda
- Goodman & Kruskal Lambda-r
- Guttman's Lambda A
- Guttman's Lambda B
- Hamann
- Harris & Lahey
- Hawkins & Dotson
- Kendall's Tau
- Kent & Foster I
- Kent & Foster II
metrics_off
parameter added to ConfusionMatrix__init__
methodCLASS_PARAMS
changed to a dictionary- Code style modified
sort
parameter added torelabel
method- Document modified
CONTRIBUTING.md
updatedcodecov
removed fromdev-requirements.txt
- Test system modified
Source Code (.Zip) (940 KB)
Source Code (.tar.gz) (906 KB)
Version 3.8
distance
method added__contains__
method added__getitem__
method added- Goodman-Kruskal's Lambda A benchmark added
- Goodman-Kruskal's Lambda B benchmark added
- Krippendorff's Alpha benchmark added
- Pearson's C benchmark added
- 30 new distance/similarity added
- AMPLE
- Anderberg's D
- Andres & Marzo's Delta
- Baroni-Urbani & Buser I
- Baroni-Urbani & Buser II
- Batagelj & Bren
- Baulieu I
- Baulieu II
- Baulieu III
- Baulieu IV
- Baulieu V
- Baulieu VI
- Baulieu VII
- Baulieu VIII
- Baulieu IX
- Baulieu X
- Baulieu XI
- Baulieu XII
- Baulieu XIII
- Baulieu XIV
- Baulieu XV
- Benini I
- Benini II
- Canberra
- Clement
- Consonni & Todeschini I
- Consonni & Todeschini II
- Consonni & Todeschini III
- Consonni & Todeschini IV
- Consonni & Todeschini V
relabel
method sort bug fixedREADME.md
modifiedCompare
overall benchmarks default weights updated- Document modified
- Test system modified
Source Code (.Zip) (927 KB)
Source Code (.tar.gz) (892 KB)
Version 3.7
Curve
class addedROCCurve
class addedPRCurve
class addedpycmCurveError
class addedCONTRIBUTING.md
updatedmatrix_params_calc
function optimizedREADME.md
modified- Document modified
- Test system modified
Python 3.11
added totest.yml
Source Code (.Zip) (916 KB)
Source Code (.tar.gz) (881 KB)
Version 3.6
- Hamming distance added
- Braun-Blanquet similarity added
classes
parameter added tomatrix_params_from_table
function- Matrices with
numpy.integer
elements are now accepted - Arrays added to
matrix
parameter accepting formats - Website changed to http://www.pycm.io
- Document modified
README.md
modified
Source Code (.Zip) (883 KB)
Source Code (.tar.gz) (850 KB)
Version 3.5
- Anaconda workflow added
- Custom iterating setting added
- Custom casting setting added
plot
method updatedclass_statistics
function modifiedoverall_statistics
function modifiedBCD_calc
function modifiedCONTRIBUTING.md
updatedCODE_OF_CONDUCT.md
updated- Document modified
Source Code (.Zip) (871 KB)
Source Code (.tar.gz) (836 KB)
Version 3.4
- Colab badge added
- Discord badge added
brier_score
method addedJ (Jaccard index)
section inDocument.ipynb
updatedsave_obj
method updatedPython 3.10
added totest.yml
- Example-3 updated
- Docstrings of the functions updated
CONTRIBUTING.md
updated
Source Code (.Zip) (870 KB)
Source Code (.tar.gz) (834 KB)
Version 3.3
__compare_weight_handler__
function addedis_imbalanced
parameter added to ConfusionMatrix__init__
methodclass_benchmark_weight
andoverall_benchmark_weight
parameters added to Compare__init__
methodstatistic_recommend
function modified- Compare
weight
parameter renamed toclass_weight
- Document modified
- License updated
AUTHORS.md
updatedREADME.md
modified- Block diagrams updated
Source Code (.Zip) (865 KB)
Source Code (.tar.gz) (830 KB)
Version 3.2
classes_filter
function addedclasses
parameter added tomatrix_params_calc
functionclasses
parameter added to__obj_vector_handler__
functionclasses
parameter added to ConfusionMatrix__init__
methodname
parameter removed fromhtml_init
functionshortener
parameter added tohtml_table
functionshortener
parameter added tosave_html
method- Document modified
- HTML report modified
Source Code (.Zip) (1.1 MB)
Source Code (.tar.gz) (1.1 MB)
Version 3.1
requirements-splitter.py
addedsensitivity_index
method added- Test system modified
overall_statistics
function modified- HTML report modified
- Document modified
- References format updated
CONTRIBUTING.md
updated
Source Code (.Zip) (1.1 MB)
Source Code (.tar.gz) (1.1 MB)
Version 3.0
plot_test.py
addedaxes_gen
function addedadd_number_label
function addedplot
method addedcombine
method addedmatrix_combine
function added- Document modified
README.md
modified- Example-2 deprecated
- Example-7 deprecated
- Error messages modified
Source Code (.Zip) (1.1 MB)
Source Code (.tar.gz) (1.1 MB)
Version 2.9
notebook_check.py
addedto_array
method added__copy__
method addedcopy
method addedaverage
method refactored
Source Code (.Zip) (1 MB)
Source Code (.tar.gz) (1 MB)
Version 2.8
label_map
attribute addedpositions
attribute addedposition
method added- Krippendorff's Alpha added
- Aickin's Alpha added
weighted_alpha
method added- Single class bug fixed
CLASS_NUMBER_ERROR
error type changed topycmMatrixError
relabel
method bug fixed- Document modified
README.md
modified
Source Code (.Zip) (1 MB)
Source Code (.tar.gz) (1 MB)
Version 2.7
average
method addedweighted_average
method addedweighted_kappa
method addedpycmAverageError
class added- Bangdiwala's B added
- MATLAB examples added
- Github action added
- Document modified
README.md
modifiedrelabel
method bug fixedsparse_table_print
function bug fixedmatrix_check
function bug fixed- Minor bug in
Compare
class fixed - Class names mismatch bug fixed
Source Code (.Zip) (1 MB)
Source Code (.tar.gz) (1 MB)
Version 2.6
custom_rounder
function addedcomplement
function addedsparse_matrix
attribute addedsparse_normalized_matrix
attribute added- Net benefit (NB) added
- Yule's Q interpretation (QI) added
- Adjusted Rand index (ARI) added
- TNR micro/macro added
- FPR micro/macro added
- FNR micro/macro added
sparse
parameter addedheader
parameter added tosave_csv
method- Handler functions moved to
pycm_handler.py
- Error objects moved to
pycm_error.py
- Verified tests references updated
- Verified tests moved to
verified_test.py
- Test system modified
CONTRIBUTING.md
updated- Namespace optimized
README.md
modified- Document modified
print_normalized_matrix
method modifiednormalized_table_calc
function modifiedsetup.py
modified- summary mode updated
- Dockerfile updated
Python 3.8
added to.travis.yaml
andappveyor.yml
Source Code (.Zip) (1 MB)
Source Code (.tar.gz) (1 MB)
Version 2.5
__version__
variable added- Individual classification success index (ICSI) added
- Classification success index (CSI) added
- Example-8 (Confidence interval) added
install.sh
addedautopep8.sh
added- Dockerfile added
-
CI
method added to- ACC
- AUC
- Overall ACC
- Kappa
- TPR
- TNR
- PPV
- NPV
- PLR
- NLR
- PRE
test.sh
moved to.travis
folder- Python 3.4 support dropped
- Python 2.7 support dropped
AUTHORS.md
updatedsave_stat
,save_csv
andsave_html
methods Non-ASCII character bug fixed- Mixed type input vectors bug fixed
CONTRIBUTING.md
updated- Example-3 updated
README.md
modified- Document modified
CI
attribute renamed toCI95
kappa_se_calc
function renamed tokappa_SE_calc
se_calc
function modified and renamed toSE_calc
- CI/SE functions moved to
pycm_ci.py
- Minor bug in save_html method fixed
Source Code (.Zip) (1 MB)
Source Code (.tar.gz) (1 MB)
Version 2.4
- Tversky index (TI) added
- Area under the PR curve (AUPR) added
FUNDING.yml
addedAUC_calc
function modified- Document modified
summary
parameter added to save and stat methodssample_weight
bug innumpy
array format fixed- Inputs manipulation bug fixed
- Test system modified
- Warning system modified
alt_link
parameter added tosave_html
method andonline_help
functionCompare
class tests moved tocompare_test.py
- Warning tests moved to
warning_test.py
Source Code (.Zip) (894 KB)
Source Code (.tar.gz) (869 KB)
Version 2.3
- Adjusted F-score (AGF) added
- Overlap coefficient (OC) added
- Otsuka-Ochiai coefficient (OOC) added
save_stat
andsave_vector
parameters added tosave_obj
method- Document modified
README.md
modified- Parameters recommendation for imbalance dataset modified
- Minor bug in
Compare
class fixed pycm_help
function modified- Benchmarks color modified
Source Code (.Zip) (882 KB)
Source Code (.tar.gz) (861 KB)
Version 2.2
- Negative likelihood ratio interpretation (NLRI) added
- Cramer's benchmark (SOA5) added
- Matthews correlation coefficient interpretation (MCCI) added
- Matthews's benchmark (SOA6) added
- F1 macro added
- F1 micro added
- Accuracy macro added
Compare
class score calculation modified- Parameters recommendation for multi-class dataset modified
- Parameters recommendation for imbalance dataset modified
README.md
modified- Document modified
- Logo updated
Source Code (.Zip) (871 KB)
Source Code (.tar.gz) (849 KB)
Version 2.1
- Adjusted geometric mean (AGM) added
- Yule's Q (Q) added
Compare
class and parameters recommendation system block diagrams added- Document links bug fixed
- Document modified
Source Code (.Zip) (790 KB)
Source Code (.tar.gz) (768 KB)
Version 2.0
- G-Mean (GM) added
- Index of balanced accuracy (IBA) added
- Optimized precision (OP) added
- Pearson's C (C) added
Compare
class added- Parameters recommendation warning added
ConfusionMatrix
equal method added- Document modified
stat_print
function bug fixedtable_print
function bug fixedBeta
parameter renamed tobeta
- Parameters recommendation for imbalance dataset modified
normalize
parameter added tosave_html
methodpycm_func.py
splitted- filter and check functions moved to
pycm_util.py
RACC_calc
andRACCU_calc
functions exception handler modified- Docstrings modified
Source Code (.Zip) (731 KB)
Source Code (.tar.gz) (709 KB)
Version 1.9
- Automatic/Manual (AM) added
- Bray-Curtis dissimilarity (BCD) added
CODE_OF_CONDUCT.md
addedISSUE_TEMPLATE.md
addedPULL_REQUEST_TEMPLATE.md
addedCONTRIBUTING.md
added- X11 color names support for
save_html
method added - Parameters recommendation system added
- Warning message for high dimension matrix print added
- Interactive notebooks section (binder) added
save_matrix
andnormalize
arguments added tosave_csv
methodREADME.md
modified- Document modified
__init__
optimized- Document and examples output files moved to different folders
- Test system modified
relabel
method bug fixed
Source Code (.Zip) (710 KB)
Source Code (.tar.gz) (689 KB)
Version 1.8
- Lift Score (LS) added
color
argument added tosave_html
methodversion_check.py
added- Error messages modified
- Document modified
- Website changed to http://www.pycm.ir
- Interpretation functions moved to
pycm_interpret.py
- Utility functions moved to
pycm_util.py
- Unnecessary else and elif removed
==
changed tois
Source Code (.Zip) (675 KB)
Source Code (.tar.gz) (663 KB)
Version 1.7
- Gini Index (GI) added
- Example-7 added
pycm_profile.py
addedclass_name
argument added tostat
,save_stat
,save_csv
andsave_html
methodsoverall_param
andclass_param
arguments empty list bug fixedmatrix_params_calc
,matrix_params_from_table
andvector_filter
functions optimizedoverall_MCC_calc
,CEN_misclassification_calc
andconvex_combination
functions optimized- Document modified
Source Code (.Zip) (671 KB)
Source Code (.tar.gz) (660 KB)
Version 1.6
- AUC Value Interpretation (AUCI) added
- Example-6 added
- Anaconda cloud package added
overall_param
andclass_param
arguments added tostat
,save_stat
andsave_html
methodsclass_param
argument added tosave_csv
method_
removed from overall statistics namesREADME.md
modified- Document modified
Source Code (.Zip) (608 KB)
Source Code (.tar.gz) (597 KB)
Version 1.5
- Relative Classifier Information (RCI) added
- Discriminator Power (DP) added
- Youden's Index (Y) added
- Discriminant Power Interpretation (DPI) added
- Positive Likelihood Ratio Interpretation (PLRI) added
__len__
method addedrelabel
method added__class_stat_init__
function added__overall_stat_init__
function addedmatrix
attribute addednormalized_matrix
attribute addednormalized_table
attribute addedPython 3.7
added to.travis.yml
&appveyor.yml
README.md
modified- Document modified
LR+
renamed toPLR
LR-
renamed toNLR
normalized_matrix
method renamed to print_normalized_matrixmatrix method
renamed to print_matrixentropy_calc
fixedcross_entropy_calc
fixedconditional_entropy_calc
fixedprint_table
bug for large numbers fixed- JSON key bug in
save_obj
fixed transpose
bug insave_obj
Source Code (.Zip) (604 KB)
Source Code (.tar.gz) (593 KB)
Version 1.4
- Area Under Curve added
- AUNU added
- AUNP added
- Class Balance Accuracy added
- Global Performance Index (RR) added
- Overall MCC added
- Distance index (dInd) added
- Similarity index (sInd) added
one_vs_all
addeddev-requirements.txt
addedREADME.md
modified- Document modified
save_stat
modifiedrequirements.txt
modified
Source Code (.Zip) (597 KB)
Source Code (.tar.gz) (586 KB)
Version 1.3
- Confusion Entropy added
- Overall Confusion Entropy added
- Modified Confusion Entropy added
- Overall Modified Confusion Entropy added
- Information Score added
README.md
modified
Source Code (.Zip) (588 KB)
Source Code (.tar.gz) (577 KB)
Version 1.2
- NIR (No Information Rate) added
- P-Value added
sample_weight
addedtranspose
addedREADME.md
modified- Key error in some parameters fixed
OSX
env added to.travis.yml
Source Code (.Zip) (584 KB)
Source Code (.tar.gz) (574 KB)
Version 1.1
- Zero-one loss added
- Support added
online_help
function addedREADME.md
modifiedhtml_table
function modifiedtable_print
function modifiednormalized_table_print
function modified
Source Code (.Zip) (503 KB)
Source Code (.tar.gz) (495 KB)
Version 1.0
- Hamming loss added
README.md
modified
Source Code (.Zip) (500 KB)
Source Code (.tar.gz) (493 KB)
Version 0.9.5
- Obj load added
- Obj save added
- Example-4 added
README.md
modified- Block diagram updated
Source Code (.Zip) (500 KB)
Source Code (.tar.gz) (493 KB)
Version 0.9
- Activation Threshold added
- Example-3 added
- Jaccard index added
- Overall Jaccard index added
README.md
modifiedsetup.py
modified
Source Code (.Zip) (361 KB)
Source Code (.tar.gz) (353 KB)
Version 0.8.6
- Example section in document added
- Python 2.7 CI added
- JOSS paper pdf added
- Cite section modified
- ConfusionMatrix docstring modified
- round function changed to numpy.around
README.md
modified
Source Code (.Zip) (358 KB)
Source Code (.tar.gz) (351 KB)
Version 0.8.5
- Confusion matrix size in HTML report modified
- Example-1 (Comparison of three different classifiers) added
- Example-2 (How to plot via matplotlib) added
- ConfusionMatrix docstring added
- JOSS paper added
- Test system modified
README.md
modified
Source Code (.Zip) (237 KB)
Source Code (.tar.gz) (227 KB)
Version 0.8.1
-
New parameters added
- Goodman and Kruskal's lambda B
- Goodman and Kruskal's lambda A
- Cross Entropy
- Conditional Entropy
- Joint Entropy
- Reference Entropy
- Response Entropy
- Kullback-Liebler divergence
- Mutual Information
- Kappa Unbiased
- Kappa No Prevalence
- Random Accuracy Unbiased
- Direct ConfusionMatrix added
pycmVectorError
class addedpycmMatrixError
class added- Support
numpy
arrays pycmError
class removed
Source Code (.Zip) (111 KB)
Source Code (.tar.gz) (105 KB)
Version 0.7
-
New parameters added
- Cramer's V
- 95% Confidence interval
- Chi-Squared
- Phi-Squared
- Chi-Squared DF
- Standard error
- Kappa standard error
- Kappa 95% confidence interval
- Cicchetti benchmark
- Parameters description links in HTML report modified
- Confusion matrix color in HTML report modified
Source Code (.Zip) (60 KB)
Source Code (.tar.gz) (56 KB)
Version 0.6
- CSV Report added (
save_csv
) - Changelog added
- Confusion matrix color in HTML report changed
- Parameters description links in HTML report added
Source Code (.Zip) (56 KB)
Source Code (.tar.gz) (52 KB)
Version 0.5
-
New parameters added
- Scott's pi
- Gwet's AC1
- Bennett S score
- HTML Report Added (
save_html
)
Source Code (.Zip) (50 KB)
Source Code (.tar.gz) (48 KB)
Version 0.4
-
New parameters added
- TPR Micro/Macro
- PPV Micro/Macro
- RACC Overall
- Normalized matrix bugs fixed
- Zero class bugs fixed
- Fleiss & Altman benchmarks added
- Output file(.pycm) method added
Source Code (.Zip) (47 KB)
Source Code (.tar.gz) (45 KB)
Version 0.3
-
New parameters added
- Kappa
- Random Accuracy
- Strength of Agreement
- Overall statistics added
- Round bug fixed
Source Code (.Zip) (45 KB)
Source Code (.tar.gz) (43 KB)
Version 0.2
-
New parameters added
- Population
- Condition positive
- Test outcome positive
- Test outcome negative
- Prevalence
- G-measure
- matrix & normalized_matrix method added
- params method added
Source Code (.Zip) (42 KB)
Source Code (.tar.gz) (40 KB)
Version 0.1
- First Release
-
Supported Class Statistics
- ACC
- BM
- DOR
- F1-Score
- FDR
- FNR
- FOR
- FPR
- LR+
- LR-
- MCC
- MK
- NPV
- PPV
- TNR
- TPR
Source Code (.Zip) (40 KB)
Source Code (.tar.gz) (38 KB)