Critical analysis review techniques

Critical literature review techniques In order to qualify as a ‘ critical’ review you must go beyond the simple description of the sources you read. If you are writing a review with reference to specific research questions or objectives then you might consider: – the breadth of different pieces of work – how wide a range of the possible subject matter defined by your question(s) or objective(s) do different pieces of work cover? – the depth of the different pieces of work – how detailed is the analysis of the subject matter in each piece of work? – the relevance of each piece of work to your specific question(s) or objective(s) – how much of the subject matter you are focusing on do different pieces of work cover? Are subject(s) missed out in different sources? Are the research methods adopted in one source more useful in answering your question(s) than those adopted in another? – and so on. – gaps in the form of relevant questions that do not appear to have been tackled by the authors you read. – contradictions and inconsistencies, both within single pieces of work , and as a result of making comparisons between the work of different authors The preceding suggestions may be helpful in obtaining the lower pass-level grades in assessed work, but the award of higher grades demands the application of more sophisticated forms of criticism. These are, in the main, associated with the identification of logical flaws in the literature you are reviewing. Consequently you will need to display the ability to: – recognise un-stated and invalid assumptions in arguments. – distinguish facts from hypotheses. – distinguish facts from opinions. – distinguish an argument’s conclusions from the statements that support it. – recognise what kind of evidence is relevant and essential for the validation of an argument. – recognise how much evidence is needed to support a conclusion. – distinguish between relevant and irrelevant statements and evidence. – identify logical fallacies. NB if you are unsure what is meant by any of the terms in the last list you would be well advised to invest in a textbook on critical reasoning. There are a variety of useful books available on the market including: Brown, N. & Keeley, S., (1997), Asking the Right Questions: A Guide to Critical Thinking, Pearson: London (ISBN: 0131829939) Cottrell, S., (2005), Critical Thinking Skills, Palgrave: Basingstoke; ISBN: 1-4039-9685-7 Cottrell is particularly good on assessing the quality of arguments and evidence and a wide variety of source materials, while Brown & Keeley is excellent on the analysis of the various parts of arguments. With primary sources it may also be possible to critically comment upon: – the suitability of the research design. – the effectiveness of the data collection process. – the validity of the sample selection process . – the appropriateness of the chosen research methodology to the subject being researched. and so on. Illustrative Examples The extracts shown below were taken from a mixture of student assignments and published critical literature reviews. This is not an exhaustive list, merely a sample of the many different ways one can go about critically assessing published work. You should also note that the examples were not selected for the quality of their written English. We are not recommending or endorsing any particular writing style or format by quoting these examples. We would like you to focus on the various methods of constructing criticisms of published work. The examples illustrate the following types of criticism: – Flawed understanding of a phenomenon – Challenge to a statement of ‘ fact’ – Excessively narrow subject focus – Poor data collection methods – Omission of potentially relevant information – Lack of evidence to support conclusions – Lack of current data – Challenge to the validity of the conclusions – Exaggerated and unsubstantiated claims – Insufficient detail to permit the assessment of the validity and reliability of the conclusions – Challenge to the generalisability of the conclusions – Possible confusion between correlation and causation Most of the criticisms refer to defects in the structure of the arguments the authors have offered in their published work, in other words, to faults in the authors’ reasoning. If you do not fully understand what is meant by the phrase ‘ critical reasoning’ you are advised to remedy this gap in your learning as soon as possible. You can make a start by going to the heading of ‘ Critical Thinking’ elsewhere on this website. Flawed understanding of a phenomenon Neiss (2002) contends that foreign creditors would have suffered huge and immediate losses had the IMF not intervened. However, these are naive comments from a member of an organisation that promotes the free market approach. Foreign creditors would have known the structure of the financial systems in South East Asian countries and the close links businesses would have with their respective governments. [In other words, they would have been well aware of the risks and did not need protection from the Fund. JR] Challenge to a statement of ‘ fact’ 1. In this article The Economist is also responsible for portraying the IMF as an organisation that had no choice other than to implement the policies it implemented. Since the Fund had an enormous range of possible courses of action in response to these events this portrayal is clearly not true. 2. There appears to be an error, or at least a discrepancy when the data on waterway freight growth – 9 million tonnes (DFRA, 2002, p. 29) is compared with the figure of 12 million tonnes cited in AINA (2001). Excessively narrow subject focus Only Krippendorf (1999) fails to discuss in detail the economic impact, favouring instead a discussion of some of the other effects of tourism. One might argue that the tight concentration by other authors on the purely economic effects may result in unnecessarily narrow, incomplete and possibly biased conclusions. Poor data collection methods 1. However, Stanton and Bardoni’s (1972) research can be questioned in that whilst they carried out an anonymous questionnaire to enlisted males under 25 years of age in Vietnam, they left the respondent to define ‘flashback’ for themselves. This personal definition is clearly unsatisfactory and does not enable comparisons between studies, nor does it allow for an examination of the nature of such ‘flashback’ reports. Consequently a major variable was not controlled for or quantified. 2. Horowitz (1969) interviewed a smaller sample (n = 31) of individuals said to be representative members of a drug-taking community, found that 32% (that is 8 individuals) reported having flashbacks. However there was poly-drug use amongst this group, which could have confounded the results. 3. The problem of possible participant bias also plagued another study: the Boris and Mandel (1994) study which used 26 consecutive ADHD children referred to their allergy practice. …. The study has been criticized because Boris is an allergist and the group of participants was small. Omission of potentially relevant information When defining culture as an acquired element, the authors omit consideration of an individual’s personality and genetic makeup, although both if these factors interact with the culture in which they are brought up to create their observed behaviour. Lack of evidence to support conclusions The main shortcoming of this text is the author’s reliance on personal opinion rather than empirical data. For example, on page 325 there is a critical attack on the government’s policy towards private eye hospitals, but the author offers no data to support the argument he is making. In the absence of any supporting evidence there is no way of judging the validity or reliability of his conclusions and this seriously undermines the value of the work. Lack of current data Recent years have seen the publication of a number of critiques of changes in funding regimes, for example: Anderson (2003); Phillips (2002). However this text was published before the recent wave of reforms took place. Consequently, although much of the book’s contents remain valuable, the conclusions on this particular aspect of the problem are at best partial in nature, and at worst no longer relevant. Challenge to the validity of the conclusions — alternative conclusions drawn from the same supporting evidence Research by the EU (European Commission, 2002) has also shown that the most important factor impeding SMEs of all sizes from going digital is the belief that e-commerce is not applicable to their type of products or services and the lack of perceived commercial benefit. However this may be due to the survey’s focus on e-commerce rather than e-business. ICT skills gaps seem also to be more important for small enterprises than for larger and medium ones. Exaggerated and unsubstantiated claims When creating this theory of JIT in the late 1980’s Harrison and Tersine (1989) p. 234 made the completely unrealistic claim that it is possible through the pursuit of total quality objectives to achieve production with the creation of no waste. Insufficient detail to permit the assessment of the validity and reliability of the conclusions A lot of primary research has been conducted to help support policy recommendations in this subject area, but Haywood (2002) is lacking depth in its explanation of both the data collection process and sample selection criteria. Greater elaboration of the research process was offered in an earlier paper: Greensmith and Haywood (1999), but in the current work, in the absence of adequate detail, it is extremely difficult to assess the validity and reliability of the findings. Challenge to the generalisability of the conclusions Recent research by Actinic (2002) shows that, in their sample of UK SMEs, 72% believe their e-commerce site is profitable, and that increased sales remain the main justification for e-commerce adoption, …However, it should be noted that ‘ early adopters’ may have done so because they could obtain these benefits and it does not follows that all SMEs will benefit in the same ways. Possible confusion between correlation and causation Boniface describes Africa having: … a low level of economic development. Most African countries fall into the ‘ least developed’ category with only a few having breached the intermediate level of development. Boniface, 2002, p. 241 In contrast to this, he describes Latin American countries as being at an intermediate stage of development, although tourism in Brazil, for example: …accounts for less than one per cent of GDP. Boniface, 2002, p. 327 This ties in with the previous discussion in which Lumsdon (2001) and Youell (1999) argued that the extent to which a country benefits from its tourism activity depends on the success of its other industries. Thus Latin America is noticeably more developed than Africa, but it can be argued that this may be a result of the possession of more successful industries, rather than higher levels of tourism activity.