Lab Experiments and Quasi (Field) Experiments 6 Lab Experiments and Quasi (Field) Experiments 中央大學.資訊管理系 范錚強 mailto: ckfarn@mgt.ncu.edu.tw 2012.04 updated

Learning Objectives Understand . . . Uses for experimentation. Advantages and disadvantages of the experimental method. Seven steps of a well-planned experiment. Internal and external validity with experimental research designs. Three types of experimental designs and the variations of each. 中央資管：范錚強

研究效度間的取捨 學術性研究 實務性研究、應用研究 在進行研究設計時，研究者必須清醒的對各種效度的考量，做全盤的取捨 內部效度構念效度統計結論效度外部效度 實務性研究、應用研究 內部效度外部效度依變數的構念效度統計結論效度自變數的構念效度 在進行研究設計時，研究者必須清醒的對各種效度的考量，做全盤的取捨 例如：在研究管理者決策訂定的研究時，應該用學生為樣本（高可行性、高內部效度、低外部效度）；還是在真實情境用真實的經理人員（低可行性、低內部效度、高外部效度） 中央資管：范錚強

實驗室實驗 Lab Experiments 在一個嚴格「控制」的實驗室情境之下進行 先操弄自變數，後衡量因變數 盡可能排除干擾因素、排除可能的效度威脅 實驗室和和「真實世界」不同 外部效度可能不高 先操弄自變數，後衡量因變數 因果關係確定：內部效度 操弄manipulate自變數 有操弄，就需要考量倫理問題 衡量因變數、其他變數 中央資管：范錚強

Time order of occurrence Causal Evidence Agreement between IVs and DVs Time order of occurrence An experiment is a study involving intervention by the researcher beyond that required for measurement. The usual intervention is to manipulate some variable in a setting and observe how it affects the participants or subjects being studied. There is at least one independent variable (IV) and one dependent variable (DV) in a causal relationship. There are three types of evidence necessary to support causality. First, there must be an agreement between independent and dependent variables. The presence or absence of one is associated with the presence or absence of the other. Second, beyond the correlation of independent and dependent variables, we consider the time order of the occurrence of the variables. The effect on the dependent variable should not precede the manipulation of the independent variable. The effect and manipulation may occur simultaneously or the manipulation may occur before the effect. The third source of support comes when researchers are confident that other extraneous variables did not influence the dependent variable. To ensure that these other variables are not the source of influence, researchers control their ability to confound the planned comparison. Extraneous variables did not influence DVs 中央資管：范錚強

實驗研究的優點 對「因果」關係的確定 排除外在變數的污染 方便性、成本考量 可以複製、重複實驗 可使用自然發生的事件 中央資管：范錚強

實驗研究的弱點 實驗室的人造情境 一些特定樣本可能產生偏誤 成本可能會比較高 針對當前的問題比較有效 對樣本操弄的道德問題 中央資管：范錚強

Conducting an Experiment Specify treatment variables Specify treatment levels Control environment Choose experimental design Select and assign participants Pilot-test, revise, and test These are the seven activities the researcher must accomplish to make an experiment a success. In the first step, the researcher is challenged to 1) select variables that are the best operational definitions of the original concepts, 2) determine how many variables to test, and 3) select or design appropriate measures for the chosen variables. The selection of measures for testing requires a thorough review of the available literature and instruments. In an experiment, participants experience a manipulation of the independent variable, called the experimental treatment. The treatment levels are the arbitrary or natural groups the researcher makes within the independent variable. A control group can provide a base level for comparison. A control group is a group of participants that is measured but not exposed the independent variable being studied. Environmental control means holding the physical environment of the experiment constant. When participants do not know if they are receiving the experimental treatment, they are said to be blind. When neither the participant nor the researcher knows, the experiment is said to be double-blind. The design is then selected. Several designs are discussed on the next several slides. The participants selected for the experiment should be representative of the population to which the researcher wishes to generalize the study’s results. Random assignment is required to make the groups as comparable as possible. Collect data Analyze data 中央資管：范錚強 8

實驗過程：實驗過程的設計影響效度 選擇適當的變數 選擇實驗設計 訂定自變數操弄的劑量（不同的程度） 控制環境變數 選擇和分派樣本 環境控制：blind; double-blind 選擇和分派樣本 隨機分派、配對、配額矩陣 (quota matrix) 先導實驗、調整、測試 分析資料 中央資管：范錚強

實驗研究 實驗室實驗 Lab experiments 現場實驗 Field experiments 模擬實驗 真實實驗 True-experiments 在實驗室中進行 現場實驗 Field experiments 準實驗 Quasi-experiments 在自然情境下的現場中進行 模擬實驗 利用電腦模擬實驗情境 中央資管：范錚強

實驗室實驗和現場實驗的比較 實驗室實驗 現場實驗Field Experiment 人造情境─真實性低 自然情境─真實性高 很少外在變數 很多外在變數 控制嚴密 控制鬆散 低成本 高成本 時程短 時程長 樣本知道參與實驗 樣本不知道參與實驗 中央資管：范錚強 Copyright © 2000 by Harcourt, Inc. All rights reserved

Experiment: Placement of Benefits Module Exhibit 10-2 illustrates an experimental design. 中央資管：范錚強

Measurement Options Options Observation Physiological measures Paper-and-pencil tests Options Scaling techniques Self-administered instruments Researchers have several measurement and instrument options with experiments. 中央資管：范錚強

Validity in Experimentation Internal External Internal validity exists when the conclusions drawn about a demonstrated experimental relationship truly implies cause. External validity exists when an observed causal relationship can be generalized across persons, settings, and times. 中央資管：范錚強

Threats to Internal Validity Maturation History Experimental mortality Threats Testing Statistical regression Instrumentation Selection These seven threats to internal validity can largely be addressed through random assignment. History: During the time that an experiment is taking place, some events may occur that confuse the relationship being studied. Maturation: Changes may also occur within the participant that are a function of the passage of time and are not specific to any particular event. A participant may become hungry, bored, or tired and these conditions can affect response results. Testing: The process of taking a test can affect the scores of a second test. Instrumentation: This threat to internal validity results from changes between observation in either the measuring instrument or the observer. Selection: An important threat to internal validity is the selection of participants for experimental and control groups. The groups should be equivalent in every respect. Statistical regression: This factor operates especially when groups have been selected by their extreme scores. Experimental mortality: This occurs when the composition of the study groups changes during the test. 中央資管：范錚強

Additional Threats to Internal Validity Diffusion of treatment Compensatory equalization Compensatory rivalry Resentful disadvantaged Five additional threats to internal validity are independent of whether or not one randomizes. Diffusion or imitation of treatment: If people in the experimental and control groups talk, then those in the control group may learn of the treatment, eliminating the difference between the groups. Compensatory equalization: Where the experimental treatment is much more desirable, there may be an administrative reluctance to deprive the control group members. Actions to compensate the control group may confound the experiment. Compensatory rivalry: This may occur when members of the control group know they are in the control group. This may generate competitive pressures, causing the control group members to try harder. Resentful demoralization of the disadvantaged: When the treatment is desirable and the experiment is conspicuous, control group members may become resentful that they are deprived and lower their cooperation and output. Local history: The regular history effect already mentioned impacts both experimental and control groups alike. When one assigns all experimental persons to one group session and all control group people to another, there is a chance for some peculiar event to confound results. Local history 中央資管：范錚強

Threats to External Validity 樣本的代表性 利用學生樣本 但是，如果是網路購物、網路學習… External validity is concerned with the interaction of the experimental treatment with other factors and the resulting impact on the ability to generalize to (and across) times, settings, or persons. Three major threats to external validity are presented in the slide. Reactivity of testing on X: The reactive effect refers to sensitizing participants via a pretest so that they respond to the experimental stimulus (X) in a different way. For instance, people who participate in a web survey like the one from Kohl’s shown in the slide may then be sensitized to store displays and organization. Interaction of selection and X: The process by which test participants are selected for an experiment may be a threat to external validity. The population from which one selects participants may not be the same as the population to which one wishes to generalize the results. Other reactive factors: The experimental settings themselves may have a biasing effect on a participant’s response to X. An artificial setting can produce results that are not representative of larger populations. If participants know they are participating in an experiment, there may be a tendency to role-play in a way that distorts the effects of X. Another reactive effect is the possible interaction between X and participant characteristics. 中央資管：范錚強

Experimental Research Designs Pre-experiments True experiments Field experiments Experimental designs vary widely in their power to control contamination of the relationship between the independent and dependent variables. Experiments can be categorized as pre-experiments, true experiments, and field experiments based on the characteristic of control. Various designs in each category are presented on the following slides. 中央資管：范錚強

Selecting and Assigning Participants Random assignment Matching Random assignment uses a randomized sample frame for assigning participants to experimental and control groups. Matching is an equalizing process for assigning participants to experimental and control groups. Matching employs a nonprobability quota sampling approach. The object of matching is to have each experimental and control participant matched on every characteristic used in the research. Quota matrix is a means of visualizing the matching process. If matching does not alleviate assignment problems, a combination of matching, randomization, and increasing the sample size may be useful. Examples of each are provided on the following slides. 中央資管：范錚強

Quota Matrix Example 中央資管：范錚強 Exhibit 10-3 presents an example of a quota matrix. One-third of the participants from each cell of the matrix would be assigned to each of the tree groups. 中央資管：范錚強

實驗設計的一些慣用符號 X: manipulation 操弄 O: observation 觀察 R: random assignment 隨機分派 M: matched pair assignment 配對分派 中央資管：范錚強

實驗設計：假實驗 Pre-experimentation (One-Shot) After-Only Case Study X O X refers to the treatment or manipulation of the independent variable (more than one X refers to a different level of treatment). O refers to the observation or measurement of the dependent variable. An example is a media campaign about a product’s features without a prior measurement of consumer knowledge. Results would reveal only how much target consumers know after the media campaign, but there is no way to judge the effectiveness of the campaign. The lack of a pretest and control group makes this design inadequate for establishing causality. Dryel was test-marketed prior to full commercialization. Its placement in a market followed by observations of sales is another example of this design. 中央資管：范錚強 Pre-experiment

One Group Pretest-Posttest Design O1 X O2 This design meets the threats to internal validity better than the one-shot case study, but it is still a weak design. For example, a researcher examining the effect of a commercial on brand liking would begin by taking a pre-test to determine current levels of brand liking among the participants. The commercial would be shown. Then a post-test would measure brand liking after the commercial. A comparison between the post-test and the pre-test shows the change in liking. However, any changes in liking are not necessarily due to the commercial. The act of giving a pre-test could have influenced liking (testing effect). 中央資管：范錚強 Pre-experiment

Static Group Comparison X O1 O2 This design provides for two groups, one of which receives the experimental stimulus while the other serves as a control. For example, imagine that a new type of cheeseburger is being introduced, and an ad is run in one-fourth of the 210 designated market areas (DMAs) in the United States. After the ad airs, those who remember seeing it would be in the experimental group (X). Those who have no recall of the ad would be in the control group. The intent of each group to purchase the cheeseburger would be measured. The main weakness of this design is that there is no way to be certain that the two groups are equivalent or that the individuals are representative. Pre-experiment 中央資管：范錚強

真實驗 True Experimentation Posttest-Only Control Group Design R X O1 R O2 In this design, the pretest measurements are omitted. Pretests are well established in classical research design but are not really necessary when it is possible to randomize. The experimental effect is measured by the difference between O1 and O2. Internal validity threats from history, maturation, selection, and statistical regression are controlled adequately by the random assignment. Different mortality rates could cause a problem. Buick dealerships wish to determine the effectiveness of a special “test-drive” incentive. Buick dealerships nationwide are randomly assigned to either the control group or the experimental group. Those in the experimental group use a promotion to encourage test drives. The control group does not use any such promotions. The number of test drives throughout are measured and compared to determine if the promotion resulted in significantly more test drives. True experiment 中央資管：范錚強

Pretest-Posttest Control Group Design R O1 X O2 R O3 O4 Effect= (O2- O1) vs (O4 - O3) Deal well with the major internal validity problems True experimental designs use randomly assigned groups to ensure equivalence. In this design, groups are randomly assigned and there is a control group for comparison. The effect of the experimental variable is E = (O2-O1) – (O4-O3). This design deals with many of the threats to internal validity, but local history, maturation, and communication among groups can still lead to problems. External validity is threatened because there is a chance for a reactive effect from testing. For example, this meat counter may wish to test the effectiveness of adding a display of recipes for use with different cuts of meat. Participants are randomly assigned to either the control group or the experimental group. Those in the control group are not exposed to the display of recipes. Sales are measured for the control group at the start of the study and again the following week. Sales associated with the experimental group are also measured at the start of the study and again after exposure to the recipe display. True experiment 中央資管：范錚強

真實驗2 Completely Randomized Design E.g., 3 treatment levels R O1 X2 O3 中央資管：范錚強

真實驗3 Completely Randomized Design E.g., 3 treatment levels R O1 X2 O3 中央資管：范錚強

真實驗4 Randomized Block Design Blocking Factor Active Factor High Medium Low 7 cents R X1 12 cents X2 17 cents X3 中央資管：范錚強

真實驗5 Latin Square Design Customer Income Store Size High Medium Low Large X3 X1 X2 Small 中央資管：范錚強

真實驗6 Factorial Design Covariance Analysis Price Spread Unit price information? 7 Cents 12 Cents 17 Cents Yes X1Y1 X1Y2 X1Y3 No X2Y1 X2Y2 X2Y3 中央資管：范錚強

Field Experiments: Quasi- or Semi-Experiments Non Equivalent Control Group Design Separate Sample Pretest-Posttest Design Group Time Series Design 中央資管：范錚強

Nonequivalent Control Group Design O1 X O2 O3 O4 This is a strong and widely used quasi-experimental design. It differs from the pretest and posttest control group design because the test and control groups are not randomly assigned. There are two varieties: intact equivalent design and self-selected experimental group design. In the intact equivalent design, the membership of the experimental and control groups is naturally assembled. The self-selected experimental group design is weaker because volunteers are recruited to form the experimental group, while non-volunteer participants are used for control. A comparison of the pretest results for each group is one indicator of the degree of equivalence between test and control groups. For example, children from two different classes in school may be asked to test a toy. Participants are pre-tested on their interest in the toy. The experimental group spends time playing with the toy while the control group is not exposed to the toy. A post-test then measures interest in the toy. 中央資管：范錚強 Field experiment

Separate Sample Pretest-Posttest Design R O1 (X) R X O2 This design is most applicable when we cannot know when and to whom to introduce the treatment but we can decide when and whom to measure. The parenthesized treatment (X) is irrelevant to the purpose of the study but is shown to suggest that the experimenter cannot control exposure to the treatment. This is not a strong design because several threats to internal validity are not handled adequately. History can confound the results. There is no way to restrict the application of the treatment. For example, an new advertising campaign for a prescription drug is introduced on television. Awareness of the brand name is measured prior to the campaign introduction. After the campaign ends, awareness is measured again. 中央資管：范錚強 Field experiment

Group Time Series Design R O1 O2 O3 X O4 O5 O6 R O7 O8 O9 O10 O11 O12 A time series design introduces repeated observations before and after treatment and allows participants to act as their own controls. The single treatment group design has before-after measurements as the only controls. There is also a multiple design with two or more comparison groups as well as the repeated measurements in each treatment group. This format is especially useful where regularly kept records are a natural part of the environment and are unlikely to be reactive. The time series approach is also good way to study unplanned events in an ex post facto manner. The internal validity problem for this design is history. To reduce this risk, we keep a record of possible extraneous factors and attempt to adjust the result to reflect their influence., For example, if the federal government were to begin price controls, we could still study the effects of this action on gasoline prices later if we had regularly collected records for the period before and after the advent of price control. 中央資管：范錚強 Field experiment

Job Enrichment Quasi-Experiment 中央資管：范錚強

主效果和互動效果 Main effect The influence of a single independent variable on a dependent variable. Interaction effect The influence on a dependent variable by combinations of two or more independent variables. 中央資管：范錚強

2 x 2 Factorial Design Men Women Ad A Ad B 65 > 70 60 55 75 85 45 70 60 Main Effects of Gender Main Effects of Ad > 55 75 85 45 中央資管：范錚強

變數互動 100 90 Women 80 70 Believability 60 50 40 30 Men 20 10 Ad A Ad B 中央資管：范錚強

強設計： Lab Experiments 實驗室設計能排除最多的效度威脅 強設計：內部效度高 很多批評 系統開發的結果 一般而言，內部效度最高 外部效度偏低 強設計：內部效度高 最能夠放心地做因果關係的結論 很多批評 樣本無真實性、外部效度低 系統開發的結果 模擬實驗 中央資管：范錚強