#1 Experimental Design: Generalization and Theoretical Explanation
The Generalization Approach
when: there is a particular situation of marketing interest
equate the variable relationships in the test experiment to the marketing situation via an analogy
how: experimental test of alternatives constituting different variable and levels
determination of generalization: satisfaction of internal validity -> compared to the marketing situation of interest -> if analogous, generalization is possible
analogous: match on what the researcher takes to be the critical points of comparison
sample selection: representative samples
selecting IV: IV is of interest per se
selecting DV: research specific DV
definition: validity of generalized (causal) inferences in scientific research the extent to which the results of a study can be generalized to other situations and to other people
definition: how well an experiment is done, especially whether it avoids confounding
requirements: causal agent precedes the outcomes observed, covariance between the causal agent and outcome variables, plausible alternative causal agents do not exist
when: researcher is interested in independent variables that cannot be randomly assigned (threat to internal validity)
when: use experiments to test a theoretical explanation
what: questions posed for research deal with causal relations among constructs
goal: generalizing relationship that is inferred to exist among the constructs
sample selection: homogenous sample (to minimize variation in people’s responses that undermines observations of theoretical relations)
selecting IV: IV to infer how constructs operate in a nomological network
selecting DV: include measures that vary in the psychological proximity to IV measure impact of IV (measures of some state)
definition: an abstract notion that cannot be observed
abstractions that have implications for underlying processes
inferred from examination of relationships between variables
can be always represented by multiple variables
For example, love is an invisible abstract concept that can be inferred through actions that are supposed to represent love. Therefore, love can be represented by various variables (frequent encounters, heart beatings). However, as variables can mean various constructs, it can has alternative explanations. That is, an alternative explanation occurs when variable is inferred to mean another construct(ex. heart beating after seeing thief), and this threatens the construct validity.
when: construct is hypothesized to cause some effect in another construct
what: specifies how the constructs of the cause determine the construct of the effect
descriptions of events (≠construct)
represent (one or more) constructs, concrete
represent or influence a causal agent (independent variable)
Parsimony ▶ Discussion Question 1 & 2
definition: the explanation requiring the fewest constructs that can account for all of the
when there are several explanations that are equally efficient/parsimonious, then additional data are needed to discriminate between the explanations (e.g. searching existing literature, conducting further research)
only momentary - new theoretical propositions to test and new ways to test existing propositions
Dependent Variable Triangulation ▶ Discussion Question 4
showing only one explanation can efficiently account for disparate outcomes
Independent Variable Triangulation
selecting independent variable that vary in many ways from one another but share a particular characteristic that offers a unique explanation for observed outcomes
Post Hoc ▶ Discussion Question 5
definition: analyses that were not specified before seeing the data
if explanation is made post hoc, researcher should conduct another experiment in which a successful prediction is made
Manipulated vs Measured IV
manipulated IV: introducing multiple levels or conditions of a variable, randomly assigning participants to different levels
measured IV: research participants naturally vary, or self-select themselves to a level of IV
-most proximal measure to IV
-to assess the likelihood that the state intended by an IV was induced
Intermediate State Check
-more distal to IV, more proximal than criterion measure
-most distal to IV
-represents an indicator of the outcome deduced from theory
IV introduced in an impoverished background
all other factors eliminated or held constant
appropriate for theoretical explanation approach: reduces Type II error
artificial, difficult to generalize -> not for generalization purpose
identifying nature of causal relationships among constructs
to ascertain how much of particular variables are needed to produce a strategically desired outcome (which variables, and at what levels)
Discussion Question 1: Construct Validity
The author emphasizes parsimony and implies that research ought to be reduced to the “most parsimonious’ explanation. In addition, when there are multiple equally parsimonious explanations, additional data must be gathered until one explanation turns out to be dominantly parsimonious. The author indicates that this implies the achievement of construct validity.
However, the general definition of construct validity refers to it as the “degree to which inferences can legitimately be made from the operationalizations in your study to the theoretical constructs on which those operationalizations were based”. In simpler words, construct validity shows that the experiment is actually measuring the construct it claims to be or “the degree to which a test measures what it claims, or purports, to be measuring"
Then, what is the process by which parsimony imply the achievement of construct validity? How is parsimony related to construct validity?
Discussion Question 2: “Most Parsimonious”
Parsimonious rule: The rule is that the study is completed when there is only one unique explanation for the phenomenon, If there are alternative explanations, it is necessary to continue collecting the data and repeat the experiment.
The value of parsimonious leads to two major implications in this paper. First, it is a prerequisite for judging construct validity. That is, multiple descriptions of one construct are considered lack of construct validity. The absence of construct validity indicates that a particular conclusion can not be derived from the causal relationship between the theoretical constructs.(dq2)
Second, stressing out parsimony as a key factor in judging the robustness of the research reduces negative attitude toward post hoc. This is because as long as it achieves the highest research value of a unique explanation, whether theory is built before or after data gathering does not matter.(dq 5)
Generally, alternative explanation is regarded as a possibility of existence of other underlying mechanisms which are different to researcher’s hypothesis. However, in this paper, it refers variable’s possibility of indicating different construct from researcher’s intended construct. What is the difference of these two relationships? how it can be explained or understood?
In other words, securing parsimonious means rejecting the alternative hypothesis? Or does it mean to make variables represent a unique construct? If these two have same meaning, alternative explanation is all about lack of construct validity? How does this differ from the possible underlying mechanism (mediation)?
Discussion Question 3: Intermediate state check (p. 216) & proper order of DVs
What is an intermediate state check (ISC)? What is example of ISC? What is the difference between ISC and the mediator? What order is most appropriate when measuring DV ? Can DV be measured before mediation?
Intermediate state check: more distal to the causal agent than a manipulation check but more proximal to it than the criterion measure
Why we need this variable? Since manipulation check does not prove that an independent variable created only the desired state, additional measure or multiple independent variables are necessary to rule out alternative interpretations and to check whether intended process is occuring or not after manipulation. We thought that such measures are referred as intermediate state checks.
Discussion Question 4: Dependent variables and Independent variables triangulation
The purpose of triangulation in social science is to increase the credibility and validity of the results. Triangulation is a powerful technique that facilitates validation of data through cross verification from two or more sources. In particular, it refers to the application and combination of several research methods in the study of the same phenomenon.
When the focal dependent variable is not a specific behavior, but is instead a category of response or behaviors, it may be helpful to vary not only measures along the behavior dimension, but also the types of measures taken in order to triangulate and build stronger empirical support.
Independent variables would be vary in many ways from one to another but share a particular characteristic, so triangulation could offer a unique explanation. (P203)
Not only can a construct be operationalized by many variables, but variables also can represent many construct. So there are many rival interpretations for every dependent and independent variables. So the triangulation method is useful to provide unique explanation.
Discussion Question 5: Post Hoc
Post hoc explanation refers to analyses(hypothesis) not specified before seeing the data. Post hoc receives many criticism, with an preference for prediction. This opinion is held in the light that “theoretical relationship is only rigorous when the theory uniquely predicts an outcome.”
However, in this paper, when considering parsimony as the criterion for judging the rigor of tests of theoretical relationship, post hoc explanation are just as compelling as predicted explanation. Authors assert that prediction as the criterion for a successful experimental program can be questioned from these following reasons. (1) Practically, there is no way to determine whether a hypothesis has been built before or after it has collected data. (2) It is difficult to say that prediction is more reliable than post hoc (ex. interpersonal expectancy effect *). (3) When post hoc exists, it is more likely that rival explanation have an opportunity to be a better explanation than prediction is preceded.
Regarding the third reason, “when a prediction is tested, the opportunity to assessing alternative explanations is limited, as opposed to post hoc, where alternative explanations have the opportunity to be assessed”, why would this be the case? How does post hoc explanation provide greater chance for the alternative explanations to be assessed? Is not the post hoc function that makes a variety of explanatory possibilities deviate from the parsimonious rule?
*Interpersonal expectancy effects(Rosenthal, 1969) : 행동연구자들이 특정한 결과를 피험자로부터 기대하면, 그들이 자신도 모르게 피험자를 자신이 예측한 반응의 가능성이 증진되는 방식으로 피험자를 대하는 현상. 즉, 경험주의자도 데이터의 수집과정에서 먼저 세워진 예측에 의해서 영향을 받을 수 있기 때문에 더 post hoc보다 신뢰할 만한 접근 방법이라고 말할 수 없을 논하기 위해 사용한 예시.
# 2 Keeping it Real in Experimental Research - Understanding When, Where and How to Enhance Realism and Measure Consumer Behavior
Discussed the importance of the realistic experimental designs and measuring actual behavior.
Highlighted when, where, and how consumer researchers could inject experimental realism and utilize more behavioral measures in their experimentation.
Theory-focused study: less important to enhance experimental realism and employ measures of actual behavior
Effects papers: more naturalistic experimental design with actual behavioral measures
Experimental realism in the IVs: The more these variables involve realism and entail a more naturalistic setting in which to measure or observe responses, the easier the generalization.
Experimental realism in the DVs; Anything that a participant does in an experiment may be considered behavior. It is therefore imperative to define more concretely what we mean, when we refer to a dependent measure as behavioral.
Field experiments: The IVs were high on realism, and the DVs involved actual behavior
Realistic experiments in the field: Lower on the experimental realism dimension than field experiments, but higher than lab experiments
Lab experiments: IVs range from artificial to more realistic and their DVs can vary from hypothesis to real behavior
Discussion question 6: Is it triangulation or a threat of construct validity?
Considering the various measurements of dependent variables (“the richness of potential behavioral DVs may lead the reader to conclude that everything is behavior.”), it implies that extending the definition of behavior can also provide benefits in terms of the triangulation discussed in the first paper. However, from the perspective of construct validity, can we say that such variables represent a particular construct? Could the expansion of the criteria for DV be rather a threat to construct validity?
Discussion question 7: Is awareness is important to distinguish the type of design?
In Zhao et al(2012)’s study(p.472), which is introduced as a lab experiment, participants were neither aware of the experimental manipulation, nor, aware that their decision of whether to stay or to leave the session was being recorded. In terms of awareness of the participant’s are important to distinguish whether the experiment is lab experiment or field experiments, Why can this experiment be called a lab experiment? What criteria can be used to classify this experiment as a lab experiment?