Computer Science Assignment Answer

Evidence-based Software Engineering

Why evidence is important in software engineering

Initially it is worth while considering why evidence would be beneficial to software developers, users and other stakeholders e.g. public purchasing bodies, certification bodies and the general public. EBSE is potentially important because of the central place software intensive systems are starting to take in everyday life. For example, current plans for advanced life-critical system such as drive-by-wire applications for cars and we are able medical devices have the potential for immense economic and social benefit but can also pose a major threat to industry, to society and to individuals. If systems are reliable, usable and useful, the quality of life of individual citizens will be enhanced. However, there are far too many examples of systems that have not only wasted large amounts of public money but have also caused harm to individual citizens (e.g. the automated command and control system for the London Ambulance Service). Individual citizens have a right to expect their governments to properly administer tax revenues used to commission new software systems and put in place controls to minimise the risk of such systems causing harm. There are many strategies to improve the dependability of software involving the adoption of 'better' software development procedures and practices. At a high level, the Capability Maturity Model and SPICE suggest procedures for improving the software production process. In addition, the professional bodies are establishing procedures for certification of individual software engineers. However, the high level process and the individual engineers are constrained by the specific technologies (methods, tools and procedures) they use. In most cases software is built with technologies for which we have insufficient evidence to confirm their suitability, limits, qualities, costs and inherent risks. Thus, it is difficult to be sure that changing software practices will necessarily be a change for the better. It is possible that EBSE can provide the mechanisms needed to assist practitioners to adopt appropriate technologies and to avoid in appropriate technologies.

The Goal Of EBSE

The goal of evidence-based medicine (EBM) is 'the integration of best research evidence with clinical expertise and patient values'. By analogy, we suggest that the goal of evidence-based software engineering (EBSE) should be:to provide the means by which current best evidence from research can be integrated with practical experience and human values in the decision making process regarding the development and maintenance of software. Thus EBSE would provide:

• A common goal for individual researchers and research groups to ensure that their research is directed to the requirements of industry and other stakeholder groups.
• A means by which industry practitioners can make rational decisions about technology adoption.
• A means to improve the dependability of software intensive systems, as a result of better choice of development technologies.
• A means to increase the acceptability of software intensive systems that interface with individual citizens.
• An input to certification processes.

Practicing EBSE

Steps that are needed to practice evidence-based medicine. Although there are some detailed medical references, it is easy to reformulate the steps to address evidence-based software engineering. In fact, we would hazard a guess that at least part of the attraction that evidence-based medicine has for other disciplines is the ease with which the basic steps can be adapted to other fields. However, it is important to remember that even if high-level process steps for evidence-based practice appear to be similar for medicine and software engineering, this does not guarantee that the underlying scientific, technological and organizational mechanisms that support evidence-based medicine apply to evidence-based software engineering.For this reason we consider each step in more detail below. The first point to note is that Sackettet al. consider EBM from the viewpoint of an individual medical practitioner who needs to decide how to treat a particular patient exhibiting a particular set of symptoms. For EBSE our viewpoint is likely to be somewhat different. In software engineering organizations, individual developers seldom have the option to pick and choose the technologies they are going to use. Technology adoption is often decided either by project managers on a project by project basis, or by senior managers on a departmental or organizational basis. Furthermore, in software engineering, our concern is not usually the specific task to which the technology is applied but the outcome of the project of which the task is a part Evidence-Based Software Engineering for Practitioners.

The Aim And Methodology Of EBSE

EBSE aims to improve decision making related to software development and maintenance by integrating current best evidence from research with practical experience and human values. This means we don't expect a technology to be universally good or universally bad, only more appropriate in some circumstances and for some organizations. Furthermore, practitioners will need to accumulate empirical research about a technology of interest and evaluate the research from the viewpoint of their specific circumstances. This aim is decidedly ambitious, particularly because the gap between research and practice can be wide. EBSE seeks to close this gap by encouraging a stronger emphasis on methodological rigor while focusing on relevance for practice. This is important because rigor is necessary in any research that purports to be relevant. Moreover, because most SE research hasn't influenced industrial practice,5 there's also a pressing need to prevent SE research from remaining an ivory tower activity that emphasizes academic rigor over relevance to practice. So, although rigor is a necessary condition for relevant SE research, it isn't sufficient. Medical evidence is based on rigorous studies of therapies given to real patients requiring medical treatment; laboratory experiments are not considered to provide compelling evidence. This implies that SE should n't rely solely on laboratory experiments and should attempt to gather evidence from industrial projects, using observation studies, case studies, surveys and field experiments. These empirical techniques don't have the scientific rigor of formal randomized experiments, but they do avoid the limited relevance of small scale, artificial SE experiments. Furthermore, there are substantial problems with accumulating evidence systematically and not only because accumulating evidence from different types of studies is difficult. A specific challenge in practicing EBSE is that different empirical studies of the same phenomenon often report different and sometimes contradictory results. Unless we can understand these differences, integrating individual pieces of evidence is difficult. This point to the importance of reporting contextual information in empirical studies to help explain conflicting research results. EBSE involves five steps:

1. Convert a relevant problem or information need into an answerable question.
2. Search the literature for the best available evidence to answer the question.
3. Critically appraise the evidence for its validity, impact and applicability.
4. Integrate the appraised evidence with practical experience and the customer’s values and circumstances to make decisions about practice.
5. Evaluate performance and seek ways to improve it.

However, EBSE isn't a standalone activity. Much of what’s needed to practice EBSE already exists in the concept of technology transfer and software process improvement. SPI involves several steps (each researcher and consultant has his or her own view of how many)—for example

1. Identify a problem.
2. Propose a technology or procedure to address that problem.
3. Evaluate the proposed technology in a pilot project.
4. If the technology is appropriate, adopt and implement it.
5. Monitor the organization after implementing the new technology.
6. Return to step 1.

Thus, EBSE provides mechanisms to support various parts of SPI. In particular, EBSE focuses on finding and appraising an appropriate technology for its suitability in a particular situation. This is an area where SPI is usually Supplementary Guidelines, Assessment Scheme and evidence-based evaluations of the use of Evidence. Based Software Engineering Overview to Evidence Based Software Engineering

Evidence Based Software Engineering (EBSE) has recently been proposed as a methodology to help practitioners improve their technology adoption decisions given their particular circumstances. In simple terms, EBSE first recommends the conduct of a Systematic Literature Review (SLR) to identify and appraise evidence relevant to the problem or technology under consideration. EBSE then recommends that the evaluators integrate the results of the SLR with (their) practical experience, circumstances and (professional) values.

The Five Steps Of EBSE

Step Description

1. Convert a relevant problem or information need into an answerable question
2. Search the literature for the best available evidence to answer the question
3. Critically appraise the evidence for its validity, impact and applicability
4. Integrate the appraised evidence with practical experience and the customer’s values and circumstances to make decisions about practice
5. Evaluate performance and seek ways to improve it

During the first step of EBSE, the evaluator constructs an EBSE question comprising five components: a statement of the intervention (i.e. the instrument of change, such as the introduction of a new tool) of interest, a statement of a baseline against which the performance of the intervention is compared, a statement of the users of the intervention (e.g. the users' experience and skills), a statement of the situation within which the users apply the intervention (e.g. the type of project, type of application) and a statement of the particular outcome that intervention is expected to change and ideally improve. Relating this back to the Borland advert, a simple example of an EBSE question is:

Does the Borland CaliberRM Requirements Management Tool [intervention] more effectively trace requirements [outcome], when used by experienced requirements analysts [users] in the development of a large software system [situation], when compared to the use of no formal requirements management tool [baseline]? A formal EBSE question is typically constrained to compare one intervention against one baseline e.g. comparing two RMTs, or comparing one RMT against not using any RMT.

In step 4, the evaluator attempts to make a recommendation on whether or not to adopt the intervention and this recommendation is made on the basis of the evaluator's conclusions from steps 3 and 4. It may be that the evidence is contradictory or inconclusive and a recommendation cannot be made.

The EBSE methodology was initially developed with reference to the widely-adopted Evidence Based Medicine (EBM) methodology, but more recently EBSE has drawn on other evidence based disciplines. Kitchenham et al. in particular recognise the problems with ‘transplanting’ an EBM-like methodology into software engineering.

Dybå et al. and Jørgensen et al. report anecdotal evidence on the success of teaching EBSE to undergraduate students at a Norwegian university and we have reported on two studies of the use of EBSE by students at a UK university. Janzen and Ryoo propose the development and population of the SEEDS1 repository (http://www.evidencebasedse.com), a web database of EBSE studies critically analyzed by students. Such a database would complement, for example, the NSF Center for Empirically- Based Software Engineering database (www.cebase.org), the EBSE database maintained at Durham University (http://www.dur.ac.uk/ebse/) and the US Department of Defence’sBest Practices ClearingHouse. EBSE has been the subject of two workshops, Realizing Evidence Based Software Engineering (REBSE), both co-located with the International Conference on Software Engineering (ICSE).

The indications are that EBSE has been used almost exclusively by researchers, with some application of EBSE by undergraduate students. We are not aware of any published, empirically-based evidence of professional software practitioners directly using EBSE, or SLRs, to make technology adoption decisions in their projects or organisations.

What constitutes 'best evidence'?

• Research, based on empirical evidence, published in academic journals and conferences
• Industry evaluations, based on empirical evidence, published by consultants, companies etc.
• Industry-academic collaborations, based on empirical evidence and published somewhere e.g. in the 'grey literature'

Summary

• Reviewed progress on the module
  • Main argument and a developing model
  • Your engagement on this module
• Briefly compared different methods of evaluation
  • Generation of new evidence vs. integration of existing evidence
• Focused on EBSE:
  • Purpose of EBSE
  • As a methodology for combining different kinds of knowledge and evidence
  • The steps of EBSE
  • EBSE and SPI

A preliminary empirical investigation of the use of evidence based software engineering by under-graduate students

A BRIEF OVERVIEW OF EVIDENCE-BASED SOFTWARE ENGINEERING

Since the emergence of software engineering in the late 1960's, researchers and practitioners have recognized that most of software engineering research has little practical justification. This is manifested by the fact that very few of the results from software engineering research lead to an uptake of tools or change of practice in industry. There has been a recurring debate over how to make software engineering research more accessible and, indeed, practical to software practitioners. Much of this debate has advocated empirical work. EBSE 'aims to improve decision making related to software development and maintenance by integrating current best evidence from research with practical experience and human values.' As a methodology, it is intended to direct research to the needs of industry, help practitioners to make more rational decisions about technology adoption, enable better choice of development technologies and increase the acceptability of software intensive systems that interface with users. The methodology involves five steps:

1. Convert a relevant problem or information need into an answerable question
2. Search the literature for the best available evidence to answer the question
3. Critically appraise the evidence for its validity, impact and applicability
4. Integrate the appraised evidence with practical experience and the customer’s values and circumstances to make decisions about practice
5. Evaluate performance and seek ways to improve it

Make the point that EBSE complements software process improvement (SPI) and that EBSE is particularly beneficial in an area where SPI is traditionally weak i.e. finding and appraising an appropriate technology, prior to inclusion of that technology in an SPI programmed. A related distinction between EBSE and SPI is that EBSE concentrates on improving practitioners' use of existing findings in order to make decisions. In contrast, SPI concentrates on the introduction, assessment and management of a new intervention. It follows that EBSE is not intended as an approach for generating new evidence. Rather, the focus of EBSE is on gathering and appraising existing evidence. As suggested above, EBSE places a heavy emphasis on the use of systematic reviews, which are a means to provide a fair evaluation of a phenomenon of interest by using a trustworthy, rigorous and auditable methodology. EBSE can be located within a broader context of software engineering research that has:

1. Recognized the current limitations of software engineering research.
2. Advocated the use of empirical evidence to evaluate technologies, particularly the use of quantitative evidence for these evaluations requires Level 4 companies to incorporate metrics into their decision-making; and Kitchenham and others have proposed guidelines for improving software engineering research.
3. Proposed and used other methodologies e.g. the Goal Question Metric framework and DESMET
4. Recognized the disparity between evidence-based approaches to technology evaluation and adoption as advocated by professional researchers in the field, against the actual approaches to technology evaluation and adoption commonly undertaken by software practitioners.
5. Recognized the inherent difficulties of software engineering and the problems that these cause when adopting practices from other well-established disciplines.

Introduction

In this tutorial, we’ll start to review the EBSE methodology. We will use the primary article on EBSE that has been published for practitioners. I plan to consider stages 1 and 2 of EBSE today and, if time permits, to also consider stage 3. Next week we’ll look at stages 3, 4 and 5 of EBSE. I need to emphasize that EBSE is not an ‘easy’ methodology to use. (There are a number of places in the article where the authors state that the stage or procedure isn’t easy.) The methodology tries to encourage both rigors of analysis and also relevance of analysis to a particular situation.

A guide to the tutorial

1. Read through the section entitled 'Critically appraise the evidence'
2. What 'difficulties' are there when critically appraising literature?
   • Published research isn't always of good quality
   • Methodology may have weaknesses
   • Problem being studied may not be relevant to practice
   • Evaluating an article's scientific quality can be difficult.
   • But evaluating the relevance can be more important
3. What is the most convincing form of evidence in EBM?
   • Systematic reviews:
   • They are growing practitioners must do this stuff themselves.
4. Read the box, entitled
   • Notice that the figure title refers to published studies.
   • Are there any particular issues raised in the box that you think are relevant to the coursework or EBSE?
   • Is there any vested interest?
   • Did the study focus on the outcomes of interest?
   • Does the study provide guidelines for practice?
   • Are the guidelines well described and easy to use?
   • Will the benefits of using the guidelines outweigh the costs?
   • What implications do these raise when reviewing non-published articles e.g. blogs, newspaper articles etc.?
   • Reuse as much as possible of the Figure/Checklist
   • Draw on the Supplementary Guidelines
5. Read the section 'Apply the evidence'. Any comments?
   • There is little specific guidance like the Step 3 Checklist or the EBSE question structure. Consult the supplementary guidelines.
   • Integrate the evidence from Step 3 with your practical experience, your customers' requirements and your knowledge of the situation and then apply it. This procedure isn't straight forward!
   • In your c/w you are not asked to apply the evidence: you are asked to make a recommendation on adoption.
   • A software developer must commit to actively engaging in a learning process
   • Make individual judgements in a given situation rather than simply conforming to approved standards and procedures
6. Read the section 'Evaluate performance'. Any comments?
   • Remember that this was the step that previous students have been weakest at.
   • Reflect on your use of EBSE.
   • As with step 4, little specific guidance is given on how you do this: so consult the Supplementary Guidelines.
   • Ask yourself how well you are integrating the appraised evidence with practical experience. If you are not integrating, why not? Perhaps you lack the experience?
   • Consider the use of AARs and PMAs. The 8 questions asked there are useful for the coursework and also your final year project!

Teaching Evidence-Based Software Engineering to University Students

Graphically displays the elements of Toulmin's model. The primary elements of an argument, according to Toulmin's mode, are in bold letters and the secondary elements in italic Toulmin's model of argumentation can be viewed as a layout of argument. This layout of argument, we believe, is useful for the students as a mental model when evaluating and/or constructing arguments. In particular, we find the model useful during the exercises on the evaluation of arguments. The checklist in Appendix is based on Toulmin's model of argumentation. We recommend that the student start with the identification of the claims or conclusions made by the authors. These are normally found in the conclusion section of the papers or in the abstract, but may be found other places as well. Poor papers may, in fact, have no explicit claims at all. The students are requested to evaluate the claim, e.g., whether the claim is circular or vague. We also ask the students to identify the qualifiers, i.e., statements about the strength of the claim and the reservations, i.e., statements about the limitations of the claim. These are important when later evaluating the relevance of the evidence and the connection between evidence and claim. For example, a claim that is qualified with 'this weakly indicates a cause-effect relationship' should be evaluated differently from the claim 'there is a cause effect relationship.' Then, we ask the student to look for the data, i.e. the evidence supporting the claim. In particular, we ask them to evaluate the relevance of the evidence. We frequently find that the students are surprised by how little relevant evidence a lengthy software engineering paper contains. Finally, we ask the students to look for the warrant, i.e., the supporting connection between the data and the claim. This is frequently the most difficult part of the evaluation of the argumentation, where the critical appraisal ability and analytical skill of the students is most important. The students are requested to evaluate the degree to which the relevant data supports the claim. To support this evaluation step, the students are taught guidelines for how empirical studies should be conducted [4] and general guidelines on ethical issues pertaining to such studies when conducted on human subjects. The warrants may have a backing, i.e., an argument that supports a connection of confirmation or deduction between the data and the claim. When it is not obvious that the connection between data and claim is valid (or invalid), we ask the students to search for elements that the authors use to support it (the backing). This may, for example, consist of analytical argumentation or evidence supporting the specific interpretation of data conducted by the authors. We have experienced that familiarity with the model just outlined does more than change how the students evaluate argumentations. It also seems to lead to a more efficient and critical reading of software engineering texts that may be useful for the student in other contexts. For example, we have observed that most of them replace the mechanical reading of papers from the first to the last page, to a more information seeking, flexible reading strategy. If the information about potential vested interests and the claims are found in at the last page of a paper, it is natural to start the reading there. In addition, more knowledge about the elements of argumentation may have enabled the students to improve the quality of their own arguments.