Ethics, reliability and validity issues for vocational assessment research papers example

Vocational Assessments

Introduction
In psychology and career counseling, vocational assessments play an important role in career planning. Vocational assessments are most often used as metric for children who are transitioning from school to work and adult life, and are meant to determine student expectations for the transition planning process, as well as how these students perceive the future (Dutta et al., 2008). Through vocational assessment, children are also meant to receive practical advice or realistic responses to their own aspirations and goals, as well as indicate how the child might best be able to serve in a particular field or career. The importance of finding employment for these individuals is substantial, and vocational assessments are the chief metric used to facilitate this goal. Vocational assessments are currently being used for a variety of purposes, including the assessment of children with disabilities and adults encountering vocational rehabilitation (VR); there are several changes that can be made to vocational assessment as it is currently used in order to increase its effectiveness and validity.
Applicable Research Areas and FieldsVocational assessments are shown to be regularly used in the context of children and students with disabilities (Dutta et al., 2008). Vocational rehabilitation (VR) is an extremely prevalent context for which vocational assessments are used as one of several components; VR is defined as services that provide individuals with disabilities, veterans or other eligible participants with the ability to change or restore their ability to work in particular fields (Winther & Klotz, 2013).
In a state vocational rehabilitative context, vocational assessment is defined by Dutta et al. (2008) as:
“ Services provided and activities performed to determine an individual’s eligibility for VR services, to assign an individual to a priority category of a state VR agency that operates under an order of selection, and/or to determine the nature and scope of VR services to be included in the individual plan for employment (IPE); included in this category are trial work experiences and extended evaluation” (p. 329).
Vocational assessment typically works by helping participants to find their action competencies, measuring the specific fields and tasks that people are more likely to be skilled in so they might focus their efforts on one particular field they theoretically excel in (Winther & Klotz, 2013). Vocational assessments offer individual recommendations by providing information about action competencies carried by the individual based on an individual-centric, customized assessment based on disability and preferred fields of work (Targett et al., 2005).
The effectiveness of vocational assessment on employment outcomes seems to be largely supported by evaluative research. Dutta et al. (2008) explored the effect of vocational rehabilitation services on providing employment to people with disabilities in the United States. This particular case study examined the large-scale data set results of the United States Department of Education, Rehabilitation Service Administration Case Service Report, offering “ personal history, types of services, and employment outcome information” on clients benefiting from vocational rehabilitation services in the United States (p. 327). Among the services provided by state VR agencies includes assessment (p. 329). This vocational assessment focuses primarily on whether or not people with disabilities receiving vocational rehabilitation services effectively received gainful employment after the fact, as opposed to their potential ability to be employed later in life.
The purpose of this intervention was to determine whether or not the allotment of state funds to vocational rehabilitation services improves their employment outcomes, as well as determining which services in particular provide direct correlation to these better outcomes (p. 327). According to the results, state vocational rehabilitation services do have a strong association with employment outcomes, implying the effectiveness of vocational assessment as a tool for improving prospects for adults with disabilities in the United States (Detta et al., 2008). To that end, the authors note that more individuals should be given access to VR services, and occupational therapists can utilize these findings to promote further use of assessment.
Example 1 – Targett et al. (2005)
William’s assessment involved him working at a cardiology department of a willing hospital (as he had been a cardiologist pre-injury) to determine residual skills, with a skilled and experienced consultant overseeing his further assessment by discussing current skills and knowledge in an interview process. After determining that his disability would prevent him from performing the tasks needed to be a cardiologist, William was eventually guided to become a Cardiac Catheterization Technician at one of the hospitals selected (Targett et al., 2005). As a result, William had been, as of the publication of the study, a full-time employee at that hospital for over a year, providing a positive outcome for William as a recipient of vocational assessment.
Example 2 – Cook et al. (2005)
Cook et al. (2005) provides further evidence for the effectiveness of vocational assessments and other services in vocational rehabilitation (VR). Discussing the issue of patients with severe mental illness, the researchers examined seven sites in order to determine the effectiveness of vocational assessment on work outcomes, the experimental group engaging in a program emphasizing integrated psychiatric and vocational service delivery. The researchers wished to determine whether or not more highly-integrated participants would receive better employment outcomes, as well as how many vocational services received directly correlated to increased vocational outcomes. According to the results, “ supported employment models in which psychiatric and vocational service delivery are highly integrated produce better vocational outcomes” (Cook et al. 2005, p. 1954). Participants who received more vocational services (including assessment) had better outcomes for employment, thus furthering the idea that vocational assessment is a successful, positive method of improving employment outcomes.

Despite the ostensibly positive assessment of the helpfulness of vocational assessment in helping its target populations achieve higher employment outcomes, there are still a few issues of ethics, reliability and validity that exist within the practice.. In a study of numerous vocational assessment models, Winther & Klotz (2013) note a number of problems with these assessments. For one, test practices involved in vocational assessment tend to provide students with little opportunity to actually apply the knowledge they possess to a process-oriented working context. In terms of validity, content validity is found to be low in vocational assessments; the vast majority of examination curriculum focuses on the goods and services domain, but this does not match the proportion of content on this topic on the test itself (Winther & Klotz, 2013). This presents problems with representative validity, as many vocational assessments will not represent the curriculum being taught to participants in VR programs.
Johnson (2006), in his examination of validity in vocational assessment research, finds that much of the problem with the broad application of vocational assessment lies in the number of policy goals this single metric is expected to fulfill. These goals include “ introducing young people to occupational options, raising workforce skills at a national level, and engaging disaffected learners” – a matrix of goals which may not be accomplishable with the same metrics and assessments (p. 36). Furthermore, performance descriptors in VA tests may also be sources of invalid inferences, as some implicit skills may be ignored in favor of behaviors that are explicitly codified, and some attributes that have nothing to do with action competencies may be involved in the evaluation process. These threats to validity makes these assessments distinctly unfair, and offers a disservice to patients.
While there are issues of validity, there are also problems with the reliability of vocational assessments. Winther & Klotz (2013) state that many VA assessments and measurement items become less accurate in measuring the competencies of people with above-average or below-average competence level, being most accurate when participants have an average competence level. These issues contribute to an overall problem with vocational assessment as it has been executed in many different models, focusing on a “ fractured, subject-specific, content structure” over a process-oriented structure as intended by VA (Winther & Klotz 2013, p. 12). Vocational assessments often suffer from inconsistent assessment judgments, which can be directly attributed to assessor skill levels and experience; the less experienced an assessor or evaluator is, the less likely they are to give a patient an accurate evaluation (Johnson, 2006). The use of standardized assessment tools can be problematic when applying them to multiple contexts at once without variation – the varying strengths and weaknesses of each assessment method makes it difficult to find the right tool for the right assessment. Furthermore, there is a distinct lack of recent reliability research on vocational assessment, necessitating the need for more complete and up-to-date evaluations of VA (Johnson, 2006).
Given the importance of employment in determining people’s quality of life, vocational assessment and rehabilitation has a number of substantial ethical issues it has to compensate for. For example, the evaluator/client relationship must remain professional and distant, so as to allow for the most accurate and objective appraisal of a client’s vocational possibilities. Evaluators may be confronted with scenarios in which their personal feelings about a client may interfere with their ability to administer the test or properly assess their clients. Furthermore, there may also be conflicts between the evaluator’s desire to make their client financially secure (by securing them a job that will make them as much money as possible) and their desire to get the client a job however possible – this can manifest itself in situations in which clients are recommended to receive a lower starting salary than they are currently receiving from unemployment or Social Security. Attorneys may also pressure evaluators to provide clients with lower or higher disability ratings than they received in initial assessment to boost their ability to receive compensation or misrepresent their work ability. The financial pressures from attorneys and insurance providers can provide pressure to evaluators to affect eligibility for rehabilitation services, thus providing another ethical issue (Johnson, 2006). These problems must be addressed thoroughly in order to provide the best possible vocational assessment that will assist patients.
Vocational assessment being chiefly used in children of younger age and adults with disabilities (physical or mental), there are some demographic issues to also be raised. The social perception of disability may provide people with an added emotional incentive to disproportionately rate or discount the action competencies of people with disabilities, including epilepsy (a condition which receives a high level of discrimination in the workplace) (Chaplin, 2005). Furthermore, validity of tests may be affected by the specific impairment that the patient may have – hearing or visual impairments, as well as motor skill impairments, can dramatically affect a patient’s ability to accurately respond to their assessment (Johnson, 2006).

Future of Vocational Assessments

Despite the problems inherent to validity, reliability and ethics with regards to vocational assessments, there are still many reasons to continue the practice from a psychological standpoint. Vocational assessment is claimed to produce the best results in resolving employment problems related to physical disabilities such as epilepsy (Chaplin, 2005). Research indicates that, in a broad sense, vocational assessments still provide a substantial increase in employment outcomes for patients with disabilities, thus ensuring its applicability as a tool for treating these populations in an effective way (Dutta et al., 2008; Cook et al., 2005).
According to Winther and Klotz (2013), among others, serious steps must be taken to address the problems with relatability and validity currently found in vocational assessments. Some proposed steps to be taken include providing more accurate items in assessments to relate to goods and services and acquisition content areas, providing test situations that are more complex and accurate to process-oriented settings, providing a diverse set of situations in assessments to test varying action competence qualities and make test scores more interpretable, increasing reliability by reducing item difficulty, and a gradually-more-complex competence model used throughout the learning process in vocation assessment (p. 12).

Conclusion

Vocational assessments are currently being used in psychology chiefly to facilitate vocational rehabilitation in adults with disabilities or mental illness. In order to restore patients’ abilities to return to work following an injury or disability, customized vocational assessments are provided, which invariably include an interview process, residual skills assessments, and other metrics to determine vocational ability and increase employment outcomes. Currently, vocational assessment is thought to bring about positive incomes in patients with disabilities; however, there are still several threats to validity, ethics and reliability that must be addressed before the effectiveness of vocational assessment can be increased. Future research should address the potential ethical gaps in evaluator/patient relationships, the incentivizing of outside forces (attorneys, insurance providers) to provide inaccurate assessments of vocational ability, and improve the broad effectiveness of VA in light of the many different goals it is meant to accomplish.

References

Chaplin, J. (2005). Vocational assessment and intervention for people with
epilepsy. Epilepsia, 46(s1), 55-56.
Cook, J. A., Lehman, A. F., Drake, R., McFarlane, W. R., Gold, P. B., Leff, H. S., & Grey, D.
D. (2005). Integration of psychiatric and vocational services: a multisite randomized, controlled trial of supported employment. American Journal of Psychiatry, 162(10), 1948-1956.
Dutta, A., Gervey, R., Chan, F., Chou, C. C., & Ditchman, N. (2008). Vocational rehabilitation
services and employment outcomes for people with disabilities: A United States study. Journal of occupational rehabilitation, 18(4), 326-334.
Johnson, M. (2006). A review of vocational research in the UK 2002-2006: Measurement and
accessibility issues. International Journal of Training Research, 4(2), 48-71.
Johnson, S., Johnson, R., Miller, L., & Boyle, A. (2013). Reliability of vocational assessment: an
evaluation of level 3 electro-technical qualifications.
Targett, P., Wehman, P., McKinley, W. O., & Young, C. (2005). Functional vocational
Winther, E., & Klotz, V. K. (2013). Measurement of vocational competences: an analysis of the structure and reliability of current assessment practices in economic domains. Empirical Research in Vocational Education and Training, 5(1), 1-12.