Letter to the editor
Performance metrics of in vitro tests
Florian Seidel11Leibniz Research Centre for Working Environment and Human Factors, Ardeystr. 67, 44139-Dortmund, Germany
EXCLI J 2019;18:Doc604
A frequent scenario concerning predictive in vitro tests in toxicology is that a compound is either tested as toxic or non-toxic in vitro and this prediction is then compared to the human in vivo situation for validation. The performance of such binary classification tests is assessed by established metrics, for example, sensitivity as the proportion of actual toxic compounds that were predicted as such; or sensitivity that measures the proportion of compounds that are non-toxic and were correctly predicted as non-toxic by the in vitro test.
However, measures of the performance of binary classifications become suboptimal, when concentration or dose-dependent analyses are performed and the tests aim at predicting doses that cause an increased risk of toxicity in vivo. Recently, Albrecht et al. (2019) addressed this challenge and established the Toxicity Separation Index (TSI) and Toxicity Estimation Index (TEI) as new performance metrics. Both, TSI and TEI, are calculated based on the projection of positive and negative test compounds onto a two-dimensional coordinate system. Here, the y-axis indicates the in vivo blood concentration - for example Cmax - that results from a dosing schedule of a test compound, usually from therapeutic doses or from accidental overdoses. The x-axis represents the lowest concentration that causes a positive in vitro test result, also called in vitro alert. If the test differentiates well between toxic and non-toxic compounds, the toxic compounds will appear on top of the non-toxic substances in this presentation. The TSI is a continuous number that informs how well the test system differentiates between toxic and non-toxic compounds; a TSI of 1.0 indicates perfect separation, while a TSI of 0.5 represents a random result. The second recently introduced performance measure, the Toxicity Estimation Index (TEI), informs how well toxic blood concentrations in vivo can be estimated by the in vitro test system. The advantage of these new performance measures is that they can be used to optimize test systems. For example, the authors showed that the use of an EC10 instead of EC50 for cytotoxicity analysis in hepatocytes leads to a higher TSI. Moreover, TEI was improved, when gene expression was included into the test battery, meaning that the lower alert concentration of both, cytotoxicity and gene expression resulted in a better TEI than using the alert concentration of each test individually. Therefore, the TSI and TEI concept allows to modify a test and learn whether the modified version performs better than the original one. Of course, conclusions drawn from a training set of compounds need to be validated in an independent compound set to avoid overfitting.
Currently, numerous activities are ongoing to predict in vivo toxicity by in vitro tests (Leist et al., 2017; Vinken and Hengstler, 2018), particularly in the fields of hepatotoxicity (Godoy et al., 2013, 2016; Hammad, 2013; Frey et al., 2014; Jansen et al., 2017), cardiotoxicity (Sampaio et al., 2016; Chaudhari et al., 2016), developmental toxicity (Rempel et al., 2015; Krug et al., 2013) and neurotoxicity (Sisnaiske et al., 2014; Micheli et al., 2018; Meléndez et al., 2019; Shinde et al., 2015, 2016). The novel performance metrics introduced by Albrecht et al. will help to objectify how well in vitro tests predict specific forms of toxicity in vivo.
Conflict of interest
The author declares no conflict of interest.
1. Albrecht W, Kappenberg F, Brecklinghaus T, Stoeber R, Marchan R, Zhang M, et al. Prediction of human drug-induced liver injury (DILI) in relation to oral doses and blood concentrations. Arch Toxicol. 2019;93:1609-37.
2. Chaudhari U, Nemade H, Gaspar JA, Hescheler J, Hengstler JG, Sachinidis A. MicroRNAs as early toxicity signatures of doxorubicin in human-induced pluripotent stem cell-derived cardiomyocytes. Arch Toxicol. 2016;90:3087-98.
3. Chaudhari U, Nemade H, Wagh V, Gaspar JA, Ellis JK, Srinivasan SP, et al. Identification of genomic biomarkers for anthracycline-induced cardiotoxicity in human iPSC-derived cardiomyocytes: an in vitro repeated exposure toxicity approach for safety assessment. Arch Toxicol. 2016;90:2763-77.
4. Frey O, Misun PM, Fluri DA, Hengstler JG, Hierlemann A. Reconfigurable microfluidic hanging drop network for multi-tissue interaction and analysis. Nat Commun. 2014;5:4250.
5. Godoy P, Hewitt NJ, Albrecht U, Andersen ME, Ansari N, Bhattacharya S, Bode JG, et al. Recent advances in 2D and 3D in vitro systems using primary hepatocytes, alternative hepatocyte sources and non-parenchymal liver cells and their use in investigating mechanisms of hepatotoxicity, cell signaling and ADME. Arch Toxicol. 2013;87:1315-530.
6. Godoy P, Schmidt-Heck W, Natarajan K, Lucendo-Villarin B, Szkolnicka D, Asplund A, et al. Gene networks and transcription factor motifs defining the differentiation of stem cells into hepatocyte-like cells. J Hepatol. 2015;63:934-42;Erratum in: J Hepatol. 2016;64:525-6.
7. Hammad S. Advances in 2D and 3D in vitro systems for hepatotoxicity testing. EXCLI J. 2013;12:993-6.
8. Jansen PL, Ghallab A, Vartak N, Reif R, Schaap FG, Hampe J, et al. The ascending pathophysiology of cholestatic liver disease. Hepatology. 2017;65:722-38.
9. Krug AK, Kolde R, Gaspar JA, Rempel E, Balmer NV, Meganathan K, et al. Human embryonic stem cell-derived test systems for developmental neurotoxicity: a transcriptomics approach. Arch Toxicol. 2013;87:123-43.
10. Leist M, Ghallab A, Graepel R, Marchan R, Hassan R, Bennekou SH, et al. Adverse outcome pathways: opportunities, limitations and open questions. Arch Toxicol. 2017;91:3477-505.
11. Meléndez C, Muñoz P, Segura-Aguilar J. DT-diaphorase prevents aminochrome-induced lysosome dysfunction in SH-SY5Y cells. Neurotox Res. 2019;35:255-9.
12. Micheli L, Mattoli L, Maidecchi A, Pacini A, Ghelardini C, Di Cesare Mannelli L. Effect of Vitis vinifera hydroalcoholic extract against oxaliplatin neurotoxicity: in vitro and in vivo evidence. Sci Rep. 2018;8(1):14364.
13. Rempel E, Hoelting L, Waldmann T, Balmer NV, Schildknecht S, Grinberg M, et al. A transcriptome-based classifier to identify developmental toxicants by stem cell testing: design, validation and optimization for histone deacetylase inhibitors. Arch Toxicol. 2015;89:1599-618.
14. Sampaio SF, Branco AF, Wojtala A, Vega-Naredo I, Wieckowski MR, Oliveira PJ. p66Shc signaling is involved in stress responses elicited by anthracycline treatment of rat cardiomyoblasts. Arch Toxicol. 2016;90:1669-84.
15. Shinde V, Klima S, Sureshkumar PS, Meganathan K, Jagtap S, Rempel E, et al. Human pluripotent stem cell based developmental toxicity assays for chemical safety screening and systems biology data generation. J Vis Exp. 2015;100:e52333.
16. Shinde V, Sureshkumar P, Sotiriadou I, Hescheler J, Sachinidis A. Human embryonic and induced pluripotent stem cell based toxicity testing models: future applications in new drug discovery. Curr Med Chem. 2016;23:3495-509.
17. Sisnaiske J, Hausherr V, Krug AK, Zimmer B, Hengstler JG, Leist M, et al. Acrylamide alters neurotransmitter induced calcium responses in murine ESC-derived and primary neurons. Neurotoxicology. 2014;43:117-26.
18. Vinken M, Hengstler JG. Characterization of hepatocyte-based in vitro systems for reliable toxicity testing. Arch Toxicol. 2018;92:2981-6.