This book provided a concise but comprehensive introduction to ensemble learning. Author's contributions can be found in many important topics of this field, including theoretical, algorithmic as well as applicational aspects.

This work argued that "deep learning" is not limited to be "deep neural networks", and not limited to be built on differentiable modules. The proposed deep forest is the first deep model built on non-differentiable modules, relying on neither BP nor gradient. It has exhibited advantages in tabular data and can be enhanced when hardware for tree learning, like GPU for deep neural networks, become available in future.

This work addressed the long-standing fundamental problem: Why AdaBoost seems resistant to overfitting? The answer lies in the theoretical finding that AdaBoost maximizes margin mean and minimizes margin variance simultaneously, and can continue to do that even after training error reaches zero (this also implies that AdaBoost will overfit finally, though very late). This understanding inspired the ODM (Optimal margin Distribution Machines) [Zhou, ANNPR'14; Zhang & Zhou, TKDE'20]. [CVPR'21 keynote video]

This work claimed that isolation from the majority is the fundamental property of anomalies, possibly more crucial than previously believed (large) distance or (low) density. The proposed isolation forest (iForest) becomes a popularly used anomaly detection algorithm that can be found in various toolboxes such as scikit-learn.

Ensemble learning and semi-supervised learning are two different machine learning branches that were almost separately developed. This work advocated the leverage of ensemble and unlabeled data simultaneously, which has been well adopted by the ensemble community. It also gave born to a new strategy for ensemble diversity enhancement [Zhang & Zhou, DMKD'13].

An early work leveraging ensemble learning and unlabeled data.

A similar idea ten years later becomes popular under the name Knowledge Distillation.

This work showed that, an ensemble can be pruned to a smaller-sized one with improved generalization; this altered the common sense that ensemble pruning could reduce storage and computational cost in prediction but had to sacrifice generalization. It initiated optimization-based pruning, a mainstream of ensemble pruning, or called selective ensemble.

An early application of neural network ensemble to lung cancer diagnosis. A simple yet effective two-level ensemble architecture was designed to decrease missing alarms and increase overall reliability.

This book summarized authors' efforts during the past two decades on establishing theoretical foundation for exploiting evolutionary mechanisms in machine learning, trying to overcome the "heuristics" nature of evolutionary algorithms (EAs). General theoretical techniques for runtime and approximation analysis of EAs were developed. Theoretical results about important factors of evolutionary process were obtained. New algorithms guided by theoretical understandings were developed, including EAs that achieve better approximation guarantees than conventional algorithms.

This work offered a concise but comprehensive introduction to weakly supervised learning, and defined the scope of this field according to insufficient, inexact and inaccurate supervisions.

A fundamental weakness of semi-supervised learning algorithms lies in the fact that in many cases they can perform worse than supervised counterparts that use only labeled data. This issue was tackled in this work by the proposal of the S4VM (Safe Semi-Supervised SVM) algorithm with theoretical guarantee.

This work proposed to leverage informativeness and representativeness for active learning in a principled way, and presented the QUIRE algorithm.

Active learning was believed to be helpful in reducing sample complexity, but theoretical results were mostly built on the realizability assumption which rarely holds in practice, while those considered non-realizability were generally pessimistic. This work presented the first encouraging result on active learning in non-realizable case by exploiting multi-view data characteristics.

Co-training is a very successful semi-supervised learning algorithm but was believed to rely on the existence of two independent and sufficient feature sets. This work proved the necessary and sufficient condition for co-training, confirming our ECML'07 result (sufficient condition) that co-training only requires enough diversity between the classifiers; this provided theoretical foundation for co-training variants designed for common data having only one single feature set.

This work claimed that instances within a multi-instance bag should not be regarded as i.i.d. samples, and proposed the miGraph algorithm.

This work disclosed the connection between multi-instance learning and semi-supervised learning, showing that it is possible to address multi-instance learning from the view of semi-supervised learning.

The first work about semi-supervised dimensionality reduction.

The first work about semi-supervised regression.

An early work about combining semi-supervised learning and active learning, in the context of image retrieval.

This work offered a theoretical understanding of the relation of various multi-label learning performance measures, showing that they can be optimized by maximizing the instance-wise margin and/or label-wise margin defined in the work.

This work proposed to categorize multi-label learning algorithms into two groups, i.e., problem transformation and algorithm adaptation algorithms, and offered a review according to this categorization.

This work presented the first theoretical analysis on Bayes consistency of multi-label learning, disclosing the amazing fact that several commonly used multi-label performance measures are inconsistent.

The work proposed the MIML (Multi-Instance Multi-Label learning) framework widely used for learning with complicated data objects.

This work proposed the well-known multi-label learning algorithm MLkNN that can be found in various toolboxes such as scikit-multilearn.

This work proposed the first multi-label neural network algorithm BP-MLL that can be found in various toolboxes such as MULAN.

For multi-class classification with unequal misclassification costs, this work provided the condition under which there exist close-form solutions. It has become common-sense and been exploited in various toolboxes such as MATLAB.

This work proposed EasyEnsemble, an efficient class-imbalance learning algorithm that benefits efficiency from undersampling but avoids discarding important samples.

This work disclosed that rescaling is needed for severe class imbalance whereas no specific processing needed for mild class imbalance, which has become common-sense.

This work described an attempt of addressing unequal misclassification cost and class-imbalance in a unified framework, leading to the argument that the nature of cost-sensitive and class-imbalance learning are different, which has become common-sense.

This work disclosed that for shallow neural networks (with one or no hidden layer) dropout is able to reduce Rademacher complexity in polynomial, whereas for deep neural networks it can lead to an exponential reduction.

The Nyström method and random Fourier features are both effective for large-scale kernel learning. This work disclosed that the Nyström method can be significantly better in generalization when there exists large gap in the eigen-spectrum of the kernel matrix.

The bag-of-words model has been popularly used in various applications such as object recognition, but difficult to be analyzed theoretically. This work presented a statistical framework enabling its analysis.

This work proposed the Label distribution learning framework which has been widely used, such as by NASA scientists for the analysis of Mars minerals found by NASA MSL rover Curiosity.

This work argued that face recognition has to consider unequal misclassification costs and developed multi-class solutions.

An early study about facial age estimation. The proposed AGES (Aging Pattern Subspace) algorithm has become a baseline in this area.

A survey including many of authors' work about face recognition with a single training image per person. Many of the methods were later exploited under the names zero-shot learning, few-shot learning, etc.

In contrast to most manifold learning algorithms aiming to recover intrinsic low-dimensional embeddings faithfully, this work claimed that recovering a skewed embedding distorted through incorporating label information can be more convenient for classification.

A simple yet efficient dimensionality reduction algorithm considering both row and column directions.

In contrast to other pose invariant face recognition solutions (at that time) that required error-prone pose estimation before recognition, the proposed neural network ensemble approach did not require pose estimation as input, but offered recognition along with pose estimation results as output.