The use of machine learning algorithms in finance, medicine, and many other domains can profoundly impact human lives. Consequently, extensive efforts have been made to improve machine learning pipelines, making them more accurate, robust, and interpretable. In this presentation, we explore the synergy between combinatorial optimization algorithms and the machine learning domain. We focus on tree ensembles (including random forests and gradient boosting), a popular family of models with good empirical performance which is often used as a more transparent replacement to neural networks. We revisit important tasks related to model training, compression and explanation from combinatorial optimization lenses, harnessing solution techniques such as dynamic programming and mixed integer programming. Finally, we conclude the talk with other research perspectives connected to the application of combinational optimization techniques in interpretable machine learning.