Model validation - Revision history

Admin: /* The logical trap */

2012-08-29T13:20:20Z

The logical trap

← Older revision		Revision as of 15:20, 29 August 2012
Line 13:		Line 13:

	Anyone would agree that the logical inference 'if A is true, then B must be true' combined with the observation that 'B is true' does not imply that 'A is true'.		Anyone would agree that the logical inference 'if A is true, then B must be true' combined with the observation that 'B is true' does not imply that 'A is true'.
	And yet this mistake appears to be rather common: if a given plasma model (A) describes a given ~~experiment~~ (B), it is inferred that the model must be 'OK' - erroneously, because the agreement may be fortuitous or due to constraints that are hard to identify, or the data interpretation problem may be ''badly posed'' (see below).		And yet this mistake appears to be rather common: if a given plasma model (A) describes a given experimental result (B), it is inferred that the model must be 'OK' - erroneously, because the agreement may be fortuitous or due to constraints that are hard to identify, or the data interpretation problem may be ''badly posed'' (see below).

	There are several ways of avoiding this trap:		There are several ways of avoiding this trap:

Admin: /* Hidden assumptions */

2012-08-29T13:19:11Z

Hidden assumptions

← Older revision		Revision as of 15:19, 29 August 2012
Line 25:		Line 25:
	== Hidden assumptions ==		== Hidden assumptions ==

	The equations describing the behaviour of plasmas are mostly known (Maxwell's equations, etc.) but are untractable due to the large number fo particles involved.		The equations describing the behaviour of plasmas are mostly known (Maxwell's equations, etc.) but are untractable due to the large number of particles involved.
	Hence, simplifying assumptions are always made, usually of the type 'assume ''X >> Y'' '. It is quite common that these assumptions are not made fully explicit, which entrains the risk that the assumptions are violated in some specific case without this circumstance being detected. Therefore, it is important to clarify as precisely as possible under what conditions the model is valid, and check that these conditions are met for all relevant applications of the model.		Hence, simplifying assumptions are always made, usually of the type 'assume ''X >> Y'' '. It is quite common that these assumptions are not made fully explicit, which entrains the risk that the assumptions are violated in some specific case without this circumstance being detected. Therefore, it is important to clarify as precisely as possible under what conditions the model is valid, and check that these conditions are met for all relevant applications of the model.

Admin at 10:45, 28 August 2012

2012-08-28T10:45:15Z

← Older revision		Revision as of 12:45, 28 August 2012
Line 22:		Line 22:

	Second, the number of experiments for which 'A implies B' holds can be increased. E.g., if a model (A) is found to describe the obervations (B) for a large number of significantly different cases (experimental situations), without 'tweaking' parameters, then the validity of the model is enhanced; although it can never be ''proven'' that the model will always work in this way. Its validity will always remain subject to further testing.		Second, the number of experiments for which 'A implies B' holds can be increased. E.g., if a model (A) is found to describe the obervations (B) for a large number of significantly different cases (experimental situations), without 'tweaking' parameters, then the validity of the model is enhanced; although it can never be ''proven'' that the model will always work in this way. Its validity will always remain subject to further testing.

			== Hidden assumptions ==

			The equations describing the behaviour of plasmas are mostly known (Maxwell's equations, etc.) but are untractable due to the large number fo particles involved.
			Hence, simplifying assumptions are always made, usually of the type 'assume ''X >> Y'' '. It is quite common that these assumptions are not made fully explicit, which entrains the risk that the assumptions are violated in some specific case without this circumstance being detected. Therefore, it is important to clarify as precisely as possible under what conditions the model is valid, and check that these conditions are met for all relevant applications of the model.

			Of course, initial and boundary conditions are just as important to specify.

			== Circular reasoning ==

			Ideally, one would like the model only to take input from boundary and/or initial conditions, and predict the experimental outcome.
			However, often experimental measurements are taken as input (e.g., a density profile) to predict (using a model, e.g., NC theory) another experimental profile (e.g., the radial electric field). In this case, coincidence with radial electric field measurements only show ''consistency'' with the theory used, but does not prove the model is correct in and of itself, as the coincidence may be due to constraints also obeyed by other (transport) theories, while the input profile may be (partly) a result of turbulent transport (not contained in NC theory).

	== Badly posed problems ==		== Badly posed problems ==
Line 31:		Line 43:
	* [[Bayesian data analysis]]		* [[Bayesian data analysis]]
	* [[Error propagation]]		* [[Error propagation]]
			Models can of course not be validated with respect to any parameters that are badly determined in an experimental situation.

	== References ==		== References ==
	<references />		<references />

Admin: /* Badly posed problems */

2012-08-27T09:38:22Z

Badly posed problems

← Older revision		Revision as of 11:38, 27 August 2012
Line 26:		Line 26:

	A so-called 'badly posed problem' is a problem such that many model parameter choices map to the same measurement outcome (within measurement error), i.e., the model is a [[:Wikipedia:Projection_(mathematics)\|projection]]. Thus, no analysis of the available measurements can reveal the 'true' value of all model parameters, even if the model is in itself correct.		A so-called 'badly posed problem' is a problem such that many model parameter choices map to the same measurement outcome (within measurement error), i.e., the model is a [[:Wikipedia:Projection_(mathematics)\|projection]]. Thus, no analysis of the available measurements can reveal the 'true' value of all model parameters, even if the model is in itself correct.
	It may only be possible to determine the value of some parameters (with their corresponding error), while some others are largely or completely undetermined.		It may only be possible to determine the value of some parameters (with their corresponding error), while some others are largely or completely undetermined for lack of information from the measurements.
	See		See
	* [[Function parametrization]]		* [[Function parametrization]]

Admin: /* Badly posed problems */

2012-08-27T08:58:20Z

Badly posed problems

← Older revision		Revision as of 10:58, 27 August 2012
Line 30:		Line 30:
	* [[Function parametrization]]		* [[Function parametrization]]
	* [[Bayesian data analysis]]		* [[Bayesian data analysis]]
			* [[Error propagation]]

	== References ==		== References ==
	<references />		<references />

Admin: /* Badly posed problems */

2012-08-27T08:49:17Z

Badly posed problems

← Older revision		Revision as of 10:49, 27 August 2012
Line 25:		Line 25:
	== Badly posed problems ==		== Badly posed problems ==

	A so-called 'badly posed problem' is a problem such that many model parameter choices map to the same measurement outcome (within measurement error), i.e., the model is a [[:Wikipedia:Projection_(mathematics)\|projection]]. Thus, no analysis of the available measurements can reveal the 'true' value of ~~the~~ model parameters, even if the model is in itself correct. See		A so-called 'badly posed problem' is a problem such that many model parameter choices map to the same measurement outcome (within measurement error), i.e., the model is a [[:Wikipedia:Projection_(mathematics)\|projection]]. Thus, no analysis of the available measurements can reveal the 'true' value of all model parameters, even if the model is in itself correct.
			It may only be possible to determine the value of some parameters (with their corresponding error), while some others are largely or completely undetermined.
			See
	* [[Function parametrization]]		* [[Function parametrization]]
	* [[Bayesian data analysis]]		* [[Bayesian data analysis]]

Admin at 08:39, 27 August 2012

2012-08-27T08:39:26Z

← Older revision		Revision as of 10:39, 27 August 2012
Line 1:		Line 1:
	In fusion plasma physics, rather complex models are used to [[Plasma simulation\|simulate]], e.g., transport.		In fusion plasma physics, rather complex models are used to [[Plasma simulation\|simulate]], e.g., transport.
	In view of the large number of parameters of such models, the question arises as to whether these models are 'really true' (accurately describe the physical reality).		In view of the large number of parameters of such (numerical) models, the question arises as to whether these models are 'really true' (accurately describe the physical reality).
	In other words, the models should be ''validated''.		In other words, the models should be ''validated''.
	<ref>[[doi:10.1063/1.2928909\|P.W. Terry, M. Greenwald, J.-N. Leboeuf, et al., ''Validation in fusion research: Towards guidelines and best practices'', Phys. Plasmas '''15''' (2008) 062503]]</ref>		<ref>[[doi:10.1063/1.2928909\|P.W. Terry, M. Greenwald, J.-N. Leboeuf, et al., ''Validation in fusion research: Towards guidelines and best practices'', Phys. Plasmas '''15''' (2008) 062503]]</ref>
			<ref>[[doi:10.1063/1.3298884\|M. Greenwald, ''Verification and validation for magnetic fusion'', Phys. Plasmas '''17''' (2010) 058101]]</ref>

			== Model verification and validation ==

			The term 'verification' is understood to refer to an internal consistency check of the model in itself, i.e., to verify that it actually computes what it is meant to compute without (numerical or conceptual) error.
			This contrasts with 'validation', which refers to checking that the model actually describes the physical reality to which it applies within a given error margin. Evidently, validation is the more difficult of the two checks.

	== The logical trap ==		== The logical trap ==

	Anyone would agree that the logical inference 'if A is true, then B must be true' combined with the observation that 'B is true' does not imply that 'A is true'.		Anyone would agree that the logical inference 'if A is true, then B must be true' combined with the observation that 'B is true' does not imply that 'A is true'.
	And yet this mistake appears to be rather common: if a given plasma model (A) describes a given experiment (B), it is inferred that the model must be 'OK' - erroneously, because the agreement may be fortuitous or due to constraints that are hard to identify.		And yet this mistake appears to be rather common: if a given plasma model (A) describes a given experiment (B), it is inferred that the model must be 'OK' - erroneously, because the agreement may be fortuitous or due to constraints that are hard to identify, or the data interpretation problem may be ''badly posed'' (see below).

	There are several ways of avoiding this trap:		There are several ways of avoiding this trap:
Line 16:		Line 22:

	Second, the number of experiments for which 'A implies B' holds can be increased. E.g., if a model (A) is found to describe the obervations (B) for a large number of significantly different cases (experimental situations), without 'tweaking' parameters, then the validity of the model is enhanced; although it can never be ''proven'' that the model will always work in this way. Its validity will always remain subject to further testing.		Second, the number of experiments for which 'A implies B' holds can be increased. E.g., if a model (A) is found to describe the obervations (B) for a large number of significantly different cases (experimental situations), without 'tweaking' parameters, then the validity of the model is enhanced; although it can never be ''proven'' that the model will always work in this way. Its validity will always remain subject to further testing.

			== Badly posed problems ==

			A so-called 'badly posed problem' is a problem such that many model parameter choices map to the same measurement outcome (within measurement error), i.e., the model is a [[:Wikipedia:Projection_(mathematics)\|projection]]. Thus, no analysis of the available measurements can reveal the 'true' value of the model parameters, even if the model is in itself correct. See
			* [[Function parametrization]]
			* [[Bayesian data analysis]]

	== References ==		== References ==
	<references />		<references />

Admin: /* The logical trap */

2012-08-24T09:16:41Z

The logical trap

← Older revision		Revision as of 11:16, 24 August 2012
Line 15:		Line 15:
	It is therefore generally considered that NC theory has been validated via the correct prediction of the bootstrap current.		It is therefore generally considered that NC theory has been validated via the correct prediction of the bootstrap current.

	Second, the number of experiments for which 'A implies B' holds can be increased. E.g., if a model (A) is found to describe the obervations (B) for a large number of significantly different cases (experimental situations), then the validity of the model is enhanced; although it can never be ''proven'' that the model will always work in this way. Its validity will always remain subject to further testing.		Second, the number of experiments for which 'A implies B' holds can be increased. E.g., if a model (A) is found to describe the obervations (B) for a large number of significantly different cases (experimental situations), without 'tweaking' parameters, then the validity of the model is enhanced; although it can never be ''proven'' that the model will always work in this way. Its validity will always remain subject to further testing.

	== References ==		== References ==
	<references />		<references />

Admin: Created page with 'In fusion plasma physics, rather complex models are used to simulate, e.g., transport. In view of the large number of parameters of such models, the questio…'

2012-08-24T08:18:10Z

Created page with 'In fusion plasma physics, rather complex models are used to simulate, e.g., transport. In view of the large number of parameters of such models, the questio…'

New page

In fusion plasma physics, rather complex models are used to [[Plasma simulation|simulate]], e.g., transport.
In view of the large number of parameters of such models, the question arises as to whether these models are 'really true' (accurately describe the physical reality).
In other words, the models should be ''validated''.
<ref>[[doi:10.1063/1.2928909|P.W. Terry, M. Greenwald, J.-N. Leboeuf, et al., ''Validation in fusion research: Towards guidelines and best practices'', Phys. Plasmas '''15''' (2008) 062503]]</ref>

== The logical trap ==

Anyone would agree that the logical inference 'if A is true, then B must be true' combined with the observation that 'B is true' does not imply that 'A is true'.
And yet this mistake appears to be rather common: if a given plasma model (A) describes a given experiment (B), it is inferred that the model must be 'OK' - erroneously, because the agreement may be fortuitous or due to constraints that are hard to identify.

There are several ways of avoiding this trap:

First, the logical inference can be completed with a clause like 'A is the ''only'' circumstance for which B can be true'. Then, if B is true, it is obvious that A must be true.
An example of this is the [[bootstrap current]], which has been observed and for which [[Neoclassical transport|Neoclassical theory]] provides the only available reasonable explanation.
It is therefore generally considered that NC theory has been validated via the correct prediction of the bootstrap current.

Second, the number of experiments for which 'A implies B' holds can be increased. E.g., if a model (A) is found to describe the obervations (B) for a large number of significantly different cases (experimental situations), then the validity of the model is enhanced; although it can never be ''proven'' that the model will always work in this way. Its validity will always remain subject to further testing.

== References ==
<references />