Jizzakh state pedagogical institute named after abdulla kadiri foreign languages faculty

2.1. DATA COLLECTION METHODS
Data Collection Tips
Best practices recommend defining four to six student learning outcomes (SLOs) for academic programs and no more than three SLOs for certificate programs. Each SLO should have both a direct and indirect measure of student learning. Data collection should be a collaborative and inclusive process that provides useful information for both the individual faculty member and the academic program. If possible, use or modify existing assessment practices so that
the data collection process is feasible and accessible for all faculty members.
Direct Methods
Direct methods require students to demonstrate their learning so that observers can assess how well their work fits into the overall program expectation. Observers can then assess and evaluate the student work based on the program SLOs.
Direct methods can include but are not limited to:
• student performances
• research projects
• conference presentations
• capstone projects
• final portfolios
Indirect Methods
Indirect methods capture the attitudes, perceptions, or opinions of students’ learning and the efficacy of the factors that foster that learning. Indirect methods alone provide an inadequate picture of student learning, and therefore ought to be complemented with direct methods. They can, however, expand the discussion among faculty to evaluate the teaching and learning process. Indirect methods can include but are not limited to:
• individual student interviews
• alumni surveys
• student surveys
• interviews
The Benefits of a Multiple Methods Approach
Best practices recommend a multiple methods approach to academic program level assessment since a single method can restrict the interpretation of student learning. However, the limitations of one method may prompt the selection of other methods. Altogether, multiple methods provide a more accurate frame for assessing student learning. More so, a combination of quantitative and qualitative assessment methods adds reliability and a more comprehensive approach to assessment. Using a multiple methods approach to academic program level assessment has several advantages:
• Minimizes potential limitations of data collection and analysis inherent in a single method
• Provides alternative methods for students to demonstrate learning outcomes that may not have been possible in other methods
• Provides a more complete understanding and interpretation of student achievement
• Values the diversity of different learning methods
Data collection methods for impact evaluation vary along a continuum. At the one end of this continuum are quantative methods and at the other end of the continuum are Qualitative methods for data collection.
Data are usually collected through qualitative and quantitative methods. Qualitative approaches aim to address the ‘how’ and ‘why’ of a program and tend to use unstructured methods of data collection to fully explore the topic. Qualitative questions are open-ended such as ‘why do participants enjoy the program?’ and ‘How does the program help increase self esteem for participants?’. Qualitative methods include focus groups, group discussions and interviews. Quantitative approaches on the other hand address the ‘what’ of the program. They use a systematic standardized approach and employ methods such as surveys and ask questions such as ‘what activities did the program run?’ and ‘what skills do staff need to implement the program effectively?’
Both methods have their strengths and weaknesses. Qualitative approaches are good for further exploring the effects and unintended consequences of a program. They are, however, expensive and time consuming to implement. Additionally the findings cannot be generalised to participants outside of the program and are only indicative of the group involved.¹²
Quantitative approaches have the advantage that they are cheaper to implement, are standardised so comparisons can be easily made and the size of the effect can usually be measured. Quantitative approaches however are limited in their capacity for the investigation and explanation of similarities and unexpected differences.1 It is important to note that for peer-based programs quantitative data collection approaches often prove to be difficult to implement for agencies as lack of necessary resources to ensure rigorous implementation of surveys and frequently experienced low participation and loss to follow up rates are commonly experienced factors. Mixed Methods Is there a way to achieve both the depth and breadth that qualitative and quantitative methods may achieve individually? One answer is to consider a mixed methods approach as your design, combining both qualitative and quantitative research data, techniques and methods within a single research framework¹³.2
Mixed methods approaches may mean a number of things: ie a number of different types of methods in a study or at different points within a study, or, using a mixture of qualitative and quantitative methods.3,4
Mixed methods encompass multifaceted approaches that combine to capitalise on strengths and reduce weaknesses that stem from using a single research design.4 Using this approach to gather and evaluate data may assist to increase the validity and reliability of the research.
Some of the common areas in which mixed-method approaches may be used include:
• Initiating, designing, developing and expanding interventions;
• Evaluation;
• Improving research design; and
• Colloborating findings, data triangulation or convergence.¹⁴
Some of the challenges of using a mixed methods approach include:
• Delineating complementary qualitative and quantitative research questions;
• Time-intensive data collection and analysis; and
• Decisions regarding which research methods to combine.
These challenges call for training and multidisciplinary collaboration and may therefore require greater resources (both financial and personnel) and a higher workload than using a single method. However this may be mediated by identifying key issues early and ensuring the participation of experts in qualitative and quantitative research.
Mixed methods are useful in highlighting complex research problems such as disparities in health and can also be transformative in addressing issues for vulnerable or marginalised populations or research which involves community participation.3 Using a mixed-methods approach is one way to develop creative options to traditional or single design approaches to research and evaluation.¹⁵
Substantial description and documentation, often referred to as “thick description”, can be used to further explore a subject. This process provides a thorough description of the “study participants, context and procedures, the purpose of the intervention and its transferability”. Thick description also includes the complexities experienced in addition to the commonalities found, which assists in maintaining data integrity.
The use of documentation provides an ongoing record of activities. This can be records of informal feedback and reflections through journals, diaries or progress reports. The challenge of documentation is that it requires an ongoing commitment to regularly document thoughts and activities throughout the evaluation process
Creative strategies
Drama, exhibition, and video are imaginative and attractive alternatives to the written word.8These imaginative new approaches can be used to demystify the evaluation process. Using creative arts in evaluation offers opportunities for imaginative ways of understanding programs and creating evaluation knowledge. The creative arts may be used in designing, interpreting, and communicating evaluations.¹⁶ The direct perception and understanding a creative arts approach brings is helpful to the evaluator in gaining a deep understanding of the program. In addition, this approach is a useful means of connecting with participants’ experience in an evaluation.
Creative strategies are advantageous as they:
• provide an opportunity for participants to portray experience through different art forms which often reveals insights that they may not have been able to articulate in words;
• accommodate for people who learn in different ways, who have different cultural backgrounds and/or who are less articulate, it can be a most useful means of engaging them in an evaluation and offering them a voice;
• can use a combination of arts-based approaches in the evaluation process; and
• can be used in conjunction with more traditional approaches.
Challenges arising from creative strategies include:
• Participants are often fearful of engaging with art. This may be as a result of past negative experiences of art in school or lack of belief in their own abilities. The challenge is to assure them that they or their final product are not being judged. It is the process of engaging with art that often elicits valuable data.
• The success of such an approach can often rely on the interest levels of the participants; the task needs to be defined clearly, emphasising the reasoning behind it.
There are multiple forms of creative strategies which you can explore here.
Triangulation
Triangulation is used to address the validity of the data.10 Triangulation methods use multiple forms of data collection, such as focus groups, observation and in-depth interviews to investigate the evaluation objectives. Utilising multiple data collection methods leads to an acceptance of reliability and validity when the data from the various sources are comparable and consistent. Using more than one person to collect the data can also increase its reliability. This, however, will significantly increase the cost of the evaluation. Additionally, theory triangulation provides new insights by drawing on multiple theoretical perspectives.
There is skill required and involved in successful qualitative research approaches - which requires careful consideration and planning
Good quality qualitative research involves:
1. Thought
2. Preparation
3. The development of the interview schedule
4. Conducting and analysing the interview data with care and consideration