Model validation: Difference between revisions
No edit summary |
|||
| Line 1: | Line 1: | ||
In fusion plasma physics, rather complex models are used to [[Plasma simulation|simulate]], e.g., transport. | In fusion plasma physics, rather complex models are used to [[Plasma simulation|simulate]], e.g., transport. | ||
In view of the large number of parameters of such models, the question arises as to whether these models are 'really true' (accurately describe the physical reality). | In view of the large number of parameters of such (numerical) models, the question arises as to whether these models are 'really true' (accurately describe the physical reality). | ||
In other words, the models should be ''validated''. | In other words, the models should be ''validated''. | ||
<ref>[[doi:10.1063/1.2928909|P.W. Terry, M. Greenwald, J.-N. Leboeuf, et al., ''Validation in fusion research: Towards guidelines and best practices'', Phys. Plasmas '''15''' (2008) 062503]]</ref> | <ref>[[doi:10.1063/1.2928909|P.W. Terry, M. Greenwald, J.-N. Leboeuf, et al., ''Validation in fusion research: Towards guidelines and best practices'', Phys. Plasmas '''15''' (2008) 062503]]</ref> | ||
<ref>[[doi:10.1063/1.3298884|M. Greenwald, ''Verification and validation for magnetic fusion'', Phys. Plasmas '''17''' (2010) 058101]]</ref> | |||
== Model verification and validation == | |||
The term 'verification' is understood to refer to an internal consistency check of the model in itself, i.e., to verify that it actually computes what it is meant to compute without (numerical or conceptual) error. | |||
This contrasts with 'validation', which refers to checking that the model actually describes the physical reality to which it applies within a given error margin. Evidently, validation is the more difficult of the two checks. | |||
== The logical trap == | == The logical trap == | ||
Anyone would agree that the logical inference 'if A is true, then B must be true' combined with the observation that 'B is true' does not imply that 'A is true'. | Anyone would agree that the logical inference 'if A is true, then B must be true' combined with the observation that 'B is true' does not imply that 'A is true'. | ||
And yet this mistake appears to be rather common: if a given plasma model (A) describes a given experiment (B), it is inferred that the model must be 'OK' - erroneously, because the agreement may be fortuitous or due to constraints that are hard to identify. | And yet this mistake appears to be rather common: if a given plasma model (A) describes a given experiment (B), it is inferred that the model must be 'OK' - erroneously, because the agreement may be fortuitous or due to constraints that are hard to identify, or the data interpretation problem may be ''badly posed'' (see below). | ||
There are several ways of avoiding this trap: | There are several ways of avoiding this trap: | ||
| Line 16: | Line 22: | ||
Second, the number of experiments for which 'A implies B' holds can be increased. E.g., if a model (A) is found to describe the obervations (B) for a large number of significantly different cases (experimental situations), without 'tweaking' parameters, then the validity of the model is enhanced; although it can never be ''proven'' that the model will always work in this way. Its validity will always remain subject to further testing. | Second, the number of experiments for which 'A implies B' holds can be increased. E.g., if a model (A) is found to describe the obervations (B) for a large number of significantly different cases (experimental situations), without 'tweaking' parameters, then the validity of the model is enhanced; although it can never be ''proven'' that the model will always work in this way. Its validity will always remain subject to further testing. | ||
== Badly posed problems == | |||
A so-called 'badly posed problem' is a problem such that many model parameter choices map to the same measurement outcome (within measurement error), i.e., the model is a [[:Wikipedia:Projection_(mathematics)|projection]]. Thus, no analysis of the available measurements can reveal the 'true' value of the model parameters, even if the model is in itself correct. See | |||
* [[Function parametrization]] | |||
* [[Bayesian data analysis]] | |||
== References == | == References == | ||
<references /> | <references /> | ||
Revision as of 09:39, 27 August 2012
In fusion plasma physics, rather complex models are used to simulate, e.g., transport. In view of the large number of parameters of such (numerical) models, the question arises as to whether these models are 'really true' (accurately describe the physical reality). In other words, the models should be validated. [1] [2]
Model verification and validation
The term 'verification' is understood to refer to an internal consistency check of the model in itself, i.e., to verify that it actually computes what it is meant to compute without (numerical or conceptual) error. This contrasts with 'validation', which refers to checking that the model actually describes the physical reality to which it applies within a given error margin. Evidently, validation is the more difficult of the two checks.
The logical trap
Anyone would agree that the logical inference 'if A is true, then B must be true' combined with the observation that 'B is true' does not imply that 'A is true'. And yet this mistake appears to be rather common: if a given plasma model (A) describes a given experiment (B), it is inferred that the model must be 'OK' - erroneously, because the agreement may be fortuitous or due to constraints that are hard to identify, or the data interpretation problem may be badly posed (see below).
There are several ways of avoiding this trap:
First, the logical inference can be completed with a clause like 'A is the only circumstance for which B can be true'. Then, if B is true, it is obvious that A must be true. An example of this is the bootstrap current, which has been observed and for which Neoclassical theory provides the only available reasonable explanation. It is therefore generally considered that NC theory has been validated via the correct prediction of the bootstrap current.
Second, the number of experiments for which 'A implies B' holds can be increased. E.g., if a model (A) is found to describe the obervations (B) for a large number of significantly different cases (experimental situations), without 'tweaking' parameters, then the validity of the model is enhanced; although it can never be proven that the model will always work in this way. Its validity will always remain subject to further testing.
Badly posed problems
A so-called 'badly posed problem' is a problem such that many model parameter choices map to the same measurement outcome (within measurement error), i.e., the model is a projection. Thus, no analysis of the available measurements can reveal the 'true' value of the model parameters, even if the model is in itself correct. See