In this essay, a fictitious research was evaluated in determining if the design of the study was reliable. To determine the researchers design credibility, an assessment was made on the independent and dependent variables to determine if they supported the hypothesis. Assessments were made on the types of variations and the method of manipulation the independent variables were subjected to. It was determined that the design structure used a repeated measure design and the appropriateness of this choice was discussed. After a thorough enquiry into the structure of this design, a low level of internal validity was found to be present. Further investigations revealed that the design also failed to identify any control measures in place to rectify any extraneous variables which may impact on the study. Upon completion it was decided that the design failed to meet the requirements of a reliable investigation and could not be used to support the tested hypothesis.
When undertaking a research project, research can be investigated in three ways either as an experimental or an observational research or as a combination of both which is called a quasi-experiment. In this case the researcher has decided to undertake an experimental method because she wants to see what results were obtained by her intervention of the incentive scheme. In an observational research there is no intervention (Hartung & Touchette, 2009).
The researcher has attempted to investigate three main hypothesises in her research. Firstly, she has hypothesised that the new incentive scheme developed would be more effective than the existing incentive scheme in producing a higher job satisfaction and productivity rate. Secondly, she hypothesises that the new incentive scheme will be more effective, regardless of the industry being tested. Lastly she believes that the new incentive scheme would be more effective for junior staff than it would be for senior staff.
When researchers want to investigate a hypothesis, the first step in preparing for the research is discovering what the independent and dependent variables are. The independent variables (IV) of a study are the variables that the hypothesis predicts as the causal factor of a theory that you are investigating. While the dependent variables (DV), are the results that occur due to the manipulation of these IV’s (Christensen, 2007; Keppel & Wickens, 2004).
Describe IV and DV and critically assess their operationalisation
The researcher hypothesises there are three main IV which are going to manipulate the study. The first IV that will affect the study is the old and new incentive schemes. The variation that this IV takes on can be classified as a categorical variable since it’s either identifying the variable as being an old or new incentive. The second IV manipulating this research is the industry type, however the researcher has hypothesised that with the new incentive scheme the effects should be the same regardless of the industry. This IV is also a categorical variable because for this research it’s either the automotive industry or banking industry being tested. The third IV being tested is the effect the salary position (junior or senior staff) has. This IV is looking at two main salary divisions and therefore would also be classified as categorical.
The DV the researcher believes that will be affected, and tested by the IV, are the job satisfaction and productivity of the employees. Although the DV can take many classifications, the DV is classed as being one that tests scores or ratings of participant’s responses. This is because the researcher is evaluating the satisfaction and productivity level of each participant before and after an IV is altered.
The relationship between the IV and DV can be illustrated using the following 2 x 2 x 2 full factorial design. Drawing up a factorial diagram also enables the researcher to evaluate the number of participants that will be needed in order to complete the study.
Incentive Scheme
Incentive scheme Current Incentive New Incentive
Industry Automotive Banking Automotive Banking
Salary Junior Senior Junior Senior Junior Senior Junior Senior
As mentioned, the research will evaluate three levels of IV. These being the type of incentive schemes, the industry it’s applied to and the salary group difference. In the incentive level, the method of manipulation that will be used by the researcher will be an experimental manipulation, which is when the researcher can control the type of incentive a participant will be given whether it’s the old or new incentive. However at the industry and salary level, the type of manipulation used will be an individual difference manipulation. This is because when presented with a participant she can’t allocate them as being from an automotive industry when in fact they’re from banking or she can’t allocate a participant as being a senior staff when in fact they’re junior staff. Therefore, at the industry and salary level a participant will be allocated based on their own characteristic.
The researcher has decided to operationalise her research by subjecting automotive participants for three months to the old incentive scheme and after three months subjecting them to the new incentive scheme. She will than measure the DV of job satisfaction through questionnaires along with recording productivity to show, if an effect was caused by the incentive scheme.
The researcher will try to support her theory and results by repeating the same procedures with a banking industry in New Zealand. This sort of design structure is referred to as a repeated measure because the same participants are tested and then retested at each IV change (Harsha & Knapp, 1990).
This sort of operationalising is not very appropriate because the researcher has conducted this research with only two industries. For him to support her hypothesis that the incentive scheme will be more effective regardless of industry, she should have conducted her study with more industry types. Another limitation is the fact that she has compared industries from two different countries. This would be an issue of concern because cultural differences, country policies and industry practices may differ between countries. A more beneficial way that she may have tested the two industries she has selected may have been to test the results of the automotive and banking industry in Australia and then compared it to the automotive and banking industry of New Zealand. This is known as a between group studies, whereby the automotive and banking industry in Australia were undertaking the study at the same time as the same industries in New Zealand were undertaking it.
Another benefit would have been to use a control group who had no incentive scheme presented at all from the initial six month period. This would allow you to see if after three months, did the old incentive scheme show any sort of benefit at all compared to a group who had no incentives. If the old incentive and new incentive schemes were beneficial, results could have been compared to see which scheme benefited more when compared to a no incentive group
(i) Identify all extraneous variables. (ii) Critically assess any control measure used to deal with them. (iii) Propose solutions to any problems you have identified with the design
The researcher has not contemplated the effects of any extraneous variables which may have taken place when the results were obtained. Extraneous variables are any other factors that might have caused the DV to be affected (Campbell & Stanley, 1966; Christensen, 2007; Hartung & Touchette, 2009).
The researcher’s reliability regarding the results obtained from doing a repeated measure is unreliable, in that the new incentive results obtained regarding job satisfaction and productivity have been undertaken only once after the new incentive scheme was implemented. It’s possible that at the time of recording there may have been other issues which influenced a person to give such scores on the questionnaire. A possible control might have been to take a second or even third round of appraisals later to eliminate this statistical regression.
Another extraneous variable that may have taken place is order effects in terms of prior learning about incentive schemes (Campbell & Stanley, 1966). Participants have already been exposed to an incentive scheme in order to increase productivity and satisfaction. Therefore participants might form the attitude that the last incentive type wasn’t really attractive to them and therefore the new incentive won’t be either. A confounding variable in this issue may have been the appeal or attraction of the incentive. Appeal could alter the results of how seriously the incentive scheme is undertaken by staff or how satisfied or productive they were in the workplace. The researcher has not undertaken any methods to control this extraneous variable. One way of doing so would have been to make sure the incentive used is something that would be attractive no matter who undertook the study, such as cash incentives.
The method of using questionnaires to record scores also has its limits. Depending on the range of scores given on the questionnaire, a person who scores a 7 as their level of job satisfaction on the old incentive and than answers 7 for the new incentive, doesn’t really show which incentive had a greater effect. Keppel and Wickens (2004), described this as being a floor or ceiling effect. A floor effect is where a majority of scores received can be at a minimum and a ceiling effect is where a majority of scores can be at a maximum response.
Another order effect (practice) that might be present is caused by employees that do the same routine tasks everyday and develop shortcuts to improve productivity. This could impair results in that true productivity rates might be exaggerated. Therefore, these shortcuts can be considered confounding variables because these shortcuts could be misinterpreted as a result of the new incentive scheme. A way to control this issue which the researcher has overlooked is perhaps by monitoring process so that all jobs are done identically each time.
A participant trying to present themselves in a positive light for researchers is another extraneous variable and therefore what is known as the Hawthorne effect may take place. According to Hartung and Touchette (2009), when a person is being watched, they will improve their performance in order to bring themselves into positive light. An ethical aspect to consider is when participants are given questionnaires regarding job satisfaction and have their productivity observed, they may feel threatened that the results maybe used in order to replace staff. Once again no control measures have been undertaken and a way to control this may have been a briefing session at start about how the results were confidential and not used for any other method except, statistical data in evaluating the researcher’s hypothesis. A further control method here would be to have a control group with no incentive presented. This would alert the researcher that someone was improving just because they were being watched rather than an incentive.
The control of selection bias as an extraneous variable is the biggest concern of this research since no attempt at controlling this has been made. Having a comparison just between the automotive and banking industry will in no way give you the reliable results you want, in order to declare that your incentive schemes will obtain the same results in all industries. The researcher has confounded the industry IV with cultural differences by comparing results with an Australian automotive industry and a New Zealand banking industry. In order to control this factor the researcher should have first tested industries within one country along with expanding the number of industries tested prior to testing New Zealand.
Internal validity of the experiment
For internal validity to be high within a study, it must be shown that the researchers design has controlled any extraneous variable that could suggest that anything except the IV has caused the effect on the DV (Campbell & Stanley, 1966; Christensen, 2007). The researcher has made no attempt at controlling any of the extraneous variables present. There are also concerns as to her declaration that the scheme would benefit any industry since only two industries were tested. As a conclusion this research design has low internal validity and would not be recommended as a reliable study.
