Accuracy of Prediction

Kubinyi has another simple explanation of the prediction paradox [1]. Even in the absence of real outliers, external prediction will be worse than fit because the model tries to 'fit the errors' and attempts to explain them. Accordingly, external predictions contain the model error and the experimental error. When variable selection is carried out, no independent variable selection is performed in the cross-validation runs; correspondingly, variables that were included to 'explain the error' remain in the model and cause wrong predictions [1]. The higher the number of descriptors relative to the number of compounds, the higher is the chance to select those of them that give high q² values [8]. Other reasons for overestimating q² are redundancy in the training set, or, in the case of non-linear methods, the existence of multiple minima [8].

Arthur Doweyko has also published on the elusive nature of 3D QSAR predictions [10], but concludes: "Predictions can be enhanced when the test set is bounded by the descriptor space represented in the training set. Interpretation of significant interaction regions becomes more meaningful when alignment is constrained by a binding site."

At a workshop held in Setubal, Portugal in 2002, a set of principles was proposed to define the validity and applicability domain of QSAR models. These then evolved into the OECD principles in 2004 [11]. Paola Gramatica discusses three of these principles in a recent publication [12], and in particular, emphasizes the need for external validation using at least 20% of the data. Gramatica, Tropsha and others believe that validation is the absolute essential for successful application and interpretation of QSAR models [3, 13].

The necessity for validation has been accepted by leading journals. The policy of J. Chem. Inf. Model. on QSAR manuscripts [14] has been adopted by other journals such as J. Med. Chem. [15] and ChemMedChem. In part, it states: "If a new method/theory is being reported in the paper, it should be compared and 'validated' against at least one other common data set for which a published study exists, using at least one other method/approach and preferably a method/approach that has been widely used in the field. The data set should not be small... Evidence that any reported QSAR/QSPR model has been properly validated using data not in the training set must be provided."