Letter to the editor

Performance metrics of in vitro tests

Florian Seidel¹[*]

EXCLI J 2019;18:Doc604



Dear Editor,

A frequent scenario concerning predictive in vitro tests in toxicology is that a compound is either tested as toxic or non-toxic in vitro and this prediction is then compared to the human in vivo situation for validation. The performance of such binary classification tests is assessed by established metrics, for example, sensitivity as the proportion of actual toxic compounds that were predicted as such; or sensitivity that measures the proportion of compounds that are non-toxic and were correctly predicted as non-toxic by the in vitro test.

However, measures of the performance of binary classifications become suboptimal, when concentration or dose-dependent analyses are performed and the tests aim at predicting doses that cause an increased risk of toxicity in vivo. Recently, Albrecht et al. (2019[1]) addressed this challenge and established the Toxicity Separation Index (TSI) and Toxicity Estimation Index (TEI) as new performance metrics. Both, TSI and TEI, are calculated based on the projection of positive and negative test compounds onto a two-dimensional coordinate system. Here, the y-axis indicates the in vivo blood concentration - for example C_max - that results from a dosing schedule of a test compound, usually from therapeutic doses or from accidental overdoses. The x-axis represents the lowest concentration that causes a positive in vitro test result, also called in vitro alert. If the test differentiates well between toxic and non-toxic compounds, the toxic compounds will appear on top of the non-toxic substances in this presentation. The TSI is a continuous number that informs how well the test system differentiates between toxic and non-toxic compounds; a TSI of 1.0 indicates perfect separation, while a TSI of 0.5 represents a random result. The second recently introduced performance measure, the Toxicity Estimation Index (TEI), informs how well toxic blood concentrations in vivo can be estimated by the in vitro test system. The advantage of these new performance measures is that they can be used to optimize test systems. For example, the authors showed that the use of an EC₁₀ instead of EC₅₀ for cytotoxicity analysis in hepatocytes leads to a higher TSI. Moreover, TEI was improved, when gene expression was included into the test battery, meaning that the lower alert concentration of both, cytotoxicity and gene expression resulted in a better TEI than using the alert concentration of each test individually. Therefore, the TSI and TEI concept allows to modify a test and learn whether the modified version performs better than the original one. Of course, conclusions drawn from a training set of compounds need to be validated in an independent compound set to avoid overfitting.

Currently, numerous activities are ongoing to predict in vivo toxicity by in vitro tests (Leist et al., 2017[10]; Vinken and Hengstler, 2018[18]), particularly in the fields of hepatotoxicity (Godoy et al., 2013[5], 2016[6]; Hammad, 2013[7]; Frey et al., 2014[4]; Jansen et al., 2017[8]), cardiotoxicity (Sampaio et al., 2016[14]; Chaudhari et al., 2016[2][3]), developmental toxicity (Rempel et al., 2015[13]; Krug et al., 2013[9]) and neurotoxicity (Sisnaiske et al., 2014[17]; Micheli et al., 2018[12]; Meléndez et al., 2019[11]; Shinde et al., 2015[15], 2016[16]). The novel performance metrics introduced by Albrecht et al. will help to objectify how well in vitro tests predict specific forms of toxicity in vivo.

Conflict of interest

The author declares no conflict of interest.

References