If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Multiple-mini interviews (MMI) are increasingly used as part of the admission process into healthcare degrees. Evaluations have found MMIs to be a fair assessment tool in terms of reliability and validity and viewed positively by those involved in the MMI process. The use of MMIs in midwifery is novel and evaluation is lacking.
Aim
To evaluate the use of MMIs as part of the admission process for the Bachelor of Midwifery in one Australian university.
Methods
A basic convergent mixed methods study design was utilised. Data included linked data sets, Likert scale responses to survey questions, focus groups and open-ended survey questions. Integration took place at the interpretation and reporting stage.
Findings
Participants viewed the MMI experience positively. The study confirmed the reliability of the MMIs as an assessment tool. Most variance in MMI scores was attributed to the candidate at 31.4 % with the interviewer and the interview station having less influence on the MMI score at 11 % and 6.4 % variance. Older applicants on average achieved higher MMI scores, and those who spoke a language other than English at home or were first in family to attend university had lower on average MMI scores. Being born overseas did not impact an applicant’s MMI score. The overall experience was seen as fair, offering further opportunity to gain entry into the Bachelor of Midwifery.
Conclusion
MMIs were viewed positively and findings support the use of MMIs as part of an admissions process for the Bachelor of Midwifery.
MMIs as part of the admissions process is a new approach in choosing applicants to undertake the Bachelor of Midwifery. Evaluation of the use of MMIs as part of the admission process into the B Mid is lacking..
What is already known about the topic
MMIs are considered a fair assessment tool in terms of reliability and validity and there is broad agreement across stakeholders that the experience is positive. A small number of studies have found that applicants from certain ethnic and social backgrounds achieve lower MMI scores.
What this paper adds
MMIs as part of the selection process for entry into the B Mid was viewed positively and found to be reliable. Interviewers from a variety of backgrounds were considered an advantage. Older students had higher on average MMI scores. Students who spoke a language other than English at home or were first in family to attend university, had lower on average MMI scores. Rather than being biased, these findings can be explained by other potential influences.
1. Introduction
The Multiple Mini Interview (MMI) is an admissions assessment increasingly used for entry into health care professions [
Predictive validity of multiple mini interview scores for future academic and clinical placement performance in physiotherapy, occupational therapy and speech and language therapy programmes.
]. As an entry requirement, it is important that MMIs provide a valid and reliable assessment tool, a fair and equitable process, and that the perceptions of key stakeholders are considered.
Due to their extensive use in medicine, the evaluation of MMIs has predominantly taken place in that discipline [
]. With other healthcare professions now more frequently using MMIs, evaluations have expanded. An international systematic review on the utility of MMIs published in 2016 [
Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37.
] that included 41 studies, 32 from medicine, six from veterinary science, three from dentistry, one from pharmacy and one from nursing, concluded that MMIs do not appear biased against applicants based on age, gender, or socio-economic status. The authors did however note, that in a small number of studies, applicants of certain ethnic and social backgrounds achieved lower MMI scores when compared to other applicants [
Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37.
]. In an evaluative study of MMIs at a Canadian French speaking medical school, published the year following the above review, the potential influence of socio-demographics was however identified [
]. This Canadian study found a slight discrimination in entry success favouring applicants who were of older age, female gender, higher socioeconomic status, and French-speaking [
]. When examining the fairness, predictive validity and acceptability of MMIs used for entry into a medical school in Ireland, the authors found non-European students and those for whom English was not a first language, achieved significantly lower MMI scores than their European and English-speaking counterparts [
]. Similarly, evidence of the influence of language and ethnicity on MMI scores was identified in a cohort of 204 nursing students in the United Kingdom (UK), with non-British students underperforming on the MMI when compared to British students [
Further to ensuring a fair admission assessment, understanding the experience of those involved in the MMI process is also important in the evaluation of MMI’s [
Medical students' experiences of the benefits and influences regarding a placement mentoring programme preparing them for future practice as junior doctors: A qualitative study.
]. A narrative synthesis of literature published between 2000 and 2014 established that while evidence quality was low, there was broad agreement across stakeholder groups regarding the positive experience of MMIs [
]. While MMIs are viewed positively overall, the evaluation of MMIs for entry into midwifery is limited and there is a lack of understanding of how they are perceived by those who administer and undertake them.
2. Aim
This research forms part of a larger study, investigating the use of MMIs as part of the selection process for entry into the Bachelor of Midwifery (B Mid) at Western Sydney University, New South Wales (NSW), Australia. In our first study, we explored the impact of the use of MMIs on the attrition and academic outcomes of midwifery [
Sheehan A., Thomson R., Pierce H., Arundell F. The impact of Multiple Mini Interviews on the attrition and academic outcomes of midwifery students. Women and Birth 2021.
The aim of this current study was to evaluate the use of MMIs as part of the admission process into the Bachelor of Midwifery in one Australian university.
Specific objectives were:
o
To examine the influence of age, family first at university (FiF), country of birth (COB) and language other than English spoken at home, on MMI scores and academic ranking
o
To examine the reliability of MMI’s by comparing the variance in MMI scores attributable to the applicant, interviewer and station, using Generalisability theory
o
To explore the experience of the MMIs from the perspectives of key stakeholders (students, applicants, midwifery clinical educators, and interviewers).
3. The MMI process at WSU
MMIs as part of the application process for entry into the B Mid at Western Sydney University (WSU) were introduced in 2015 for the 2016 cohort. Based on academic ranking scores provided by the Universities Admission Centre (UAC) [
], eligible applicants are invited to participate in the MMIs. In the first year there were five interview stations however, following a review of the process, the number of stations increased to six. Midwifery academics develop questions for each station, to assess non-cognitive skills considered valuable to the practice of midwifery including empathy, teamwork, and ethical behaviour. In addition to the specified non-cognitive skill assessed at each station, interviewers also rate the applicant’s interpersonal communication skills. Various media, including videos, case study vignettes, and stories, are used to create scenarios on which to base interview questions. Using a seven point Likert scale, applicant responses are ranked, with anchor responses used to promote consistency in scoring. The duration of each individual interview is six minutes’ with a two-minute break between each interview. Applicants sit separately to each other between each interview.
All interviewers are volunteers from a variety of backgrounds, including for example, community representatives, academic and clinical nurses and midwives, administrators, librarians and other non-healthcare professionals. Interviewer training is mandatory. Training includes information on the rationale for the use of MMIs; the focus and intent of the interview questions; strategies to reduce interviewer unconscious or conscious bias; and the correct process for completing the score sheets. A briefing session with interviewers takes place each morning of the MMIs to restate the purpose of the interviews and to reinforce correct process for completion of the score sheets. Interviewers are required to sign a confidentiality agreement to prevent disclosure of questions, and arrangements are in place to ensure an interviewer does not interview someone they know. Debriefing of the interviewers occurs at the end of each day.
All invited applicants are provided written information on the MMI process prior to attending. Information includes directions on how to book an interview; what to bring to the interview, including personal identification; a map of the campus with the MMI location; and contact numbers for any further queries or emergencies on the day. Applicants also sign a confidentiality agreement. On the day of the MMI, a group briefing session occurs, with applicants given information outlining the MMI procedure, and an opportunity to ask questions. Similarly, a group debriefing occurs after the MMIs.
4. Methods
4.1 Study setting and design
A basic convergent mixed methods study design collecting both quantitative and qualitative data [
] was used to evaluate the use of MMIs for entry into the B Mid at WSU, Australia. A mixed methods design enabled a more comprehensive exploration of the objectives of this study [
]. Quantitative data were collected using linked data sets and Likert scale responses to survey questions and qualitative data were collected through focus groups and open-ended survey questions. Integration of the qualitative and quantitative data took place at the interpretation and reporting stage.
5. Quantitative methods
5.1 Data collection
Quantitative data included pre-existing data sets, collected and held by the university for students, provided the following variables: university Grade Point Average (GPA) scores, student attrition rates, academic ranking scores (for entry to the B Mid), age, country of birth (COB), language spoken at home, whether the applicant was first in family at university (FiF) and the MMI scores. Further details regarding data linkage can be found in Sheehan et al., 2021 [
Sheehan A., Thomson R., Pierce H., Arundell F. The impact of Multiple Mini Interviews on the attrition and academic outcomes of midwifery students. Women and Birth 2021.
]. In this study, we used linked data to examine the association between; age, FiF, COB and language spoken at home, with the MMI scores and academic ranking for students enrolled between 2016 and 2018.
To evaluate the MMI process from the perspective of the applicants, B Mid applicants who attended MMIs in 2019 were invited to complete an anonymous paper-based survey post MMI. To assist applicants in feeling safe to answer survey questions honestly, no demographic data were collected, and applicants only indicated whether they were a direct school leaver or a non-school leaver. Feedback on the MMI process was collected using five statements. Applicants indicated their level of agreement with each statement on a five-point Likert scale: “strongly agree”, “somewhat agree”, “neutral”, “somewhat disagree”, and “strongly disagree” (see additional material below for further detail of the survey questions).
6. Data analysis
6.1 Student background and MMI
To examine the association between background characteristics and MMI scores (on average), a linear regression model was implemented, with the MMI score treated as the dependent variable. The sample was limited to those students who enrolled in the course. Associations were examined between mean MMI scores and (a) the student's age, (b) whether students were/were not born in Australia and (c) whether students did/did not speak English at home and (d) whether students were/were not the first in family (FiF) to attend university. Percentage variation explained (using R2) was reported to determine whether students’ background influenced MMI scores. Another model was fitted using the academic ranking score as the dependent variable (instead of the MMI score) and R2 was calculated for this model as well. By comparing the R2 of each model, the influence of student background on the MMI score could be compared to the influence on the academic ranking score. Students, whose FiF status was unknown, were grouped with those who were FiF. In the analyses, the MMI scores for the 2016 cohort (scored out of a possible 105) were adjusted proportionally to be out of a possible score of 126 so they are directly comparable with the scores of the 2017 and 2018 cohort.
] was employed to measure the variation in MMI scores between applicants, interviewers, and stations. Ideally, the variation in MMI scores would be much less between interviewers and stations, compared to the variation between applicants. The reliability statistics, generalisability (G) and dependability of the MMI scores were estimated using the G-theory model. These statistics range from 0 to 1, where a value closer to 1 represents better generalisability and dependability. The R library gtheory in the statistical software, R, was used to implement generalisability theory [
Responses to the Likert scale questions were manually loaded into an Excel spreadsheet, which was checked for accuracy by a second person. Data were then analysed with simple descriptive statistics using Excel software. Because of the structure of the survey, and based on their qualitative responses, it was evident that in ten cases the respondents had answered the scale backwards. For the purposes of analysis, these were reversed.
7. Qualitative methods
7.1 Data collection
Qualitative data were collected through focus groups held in 2019 with students, hospital-based educators, and interviewers, and responses to two opened-ended questions in the applicant survey (2019). Semi-structured interview questions guided the focus group discussions, to explore the experiences of these stakeholders. Applicant survey questions were as follows: “What advantages/disadvantages do you see to participating in a multiple mini-interview as part of the admission process for the Bachelor of Midwifery?” “Do you have any ideas about how the MMIs might be improved?” One hospital based educator who had previously volunteered as a MMI interviewer only participated in the educator focus groups. Further details on qualitative data collection is described in Sheehan et al., 2021 [
Sheehan A., Thomson R., Pierce H., Arundell F. The impact of Multiple Mini Interviews on the attrition and academic outcomes of midwifery students. Women and Birth 2021.
Focus group discussions were audio-recorded and transcribed using a transcribing service. Text responses to the two open-ended questions were entered into a Microsoft Excel spreadsheet. Following the analysis and reporting of our first study’s objectives [
Sheehan A., Thomson R., Pierce H., Arundell F. The impact of Multiple Mini Interviews on the attrition and academic outcomes of midwifery students. Women and Birth 2021.
], a secondary analysis of the pooled focus groups and open-ended survey responses was undertaken. Analysis was based on the approach to qualitative content analysis described by Graneheim and Lundman (2004) [
To gain an overall picture of the data, focus group transcripts and survey responses were reread. The unit of analysis was ‘the experience of participating in the MMI process’ and data relating to this were extracted. Data were then analysed into meaning units, which were labeled with a code. These codes were compared and contrasted using constant comparison and sorted into themes, which were then arranged into three overarching content area themes and their related subthemes [
]. The process was undertaken separately by two of the authors, then checked and discussed with a third author. Final consensus was reached with all authors.
7.3 Ethical considerations
This study was approved by the University Human Research Ethics Committee (ID H13400). All potential participants were provided with information sheets, where the voluntary nature of participation in the study was emphasised. Assurance was given to both applicants and students that participation or non-participation in the study would not affect their relationship with the university. A member of the research team with no teaching relationship conducted student focus groups. B Mid applicants were advised that their survey responses would be anonymous.
8. Quantitative results
8.1 Student background and MMIs
Of the 279 students who enrolled in the B Mid course, 123 were from the 2013, 2014 and 2015 cohorts (non-MMI) and 156 completed MMIs for the 2016, 2017 and 2018 cohorts. In this sample, 214 (76.7 %) students were born in Australia, 170 (60.9 %) were first in family at university (FiF) and 56 (20.1 %) spoke a language other than English at home. Overall, the average age of the MMI sample (n = 156) was 29.6 years (SD 9.0) and the average MMI score achieved by these students was 100.7 (SD 11.2) out of a possible score of 126. MMI scores were on average significantly lower for those students who were younger, FiF, or spoke a language other than English at home. These differences, however, were small compared to the range of MMI scores (range MMI (69−123), (see Fig. 1). There was no significant differences in MMI scores for students who were born overseas (see Fig. 1). Academic ranking score was not influenced by student background (Fig. 1).
Fig. 1Student Background: MMI Scores and Academic Ranking.
For the MMIs conducted between 2015 and 2018, there were 70 different interviewers, and as previously stated, five interview stations in 2015 and 6 stations in 2016, 2017, and 2018.
Using generalisability theory, estimated variance for the examined potential influences on the MMI scores, were 31.4 % for the applicants’ ability, 11% for the interviewers, 6.4 % for the stations with 51.2 % residual variance. Here the residual variance represents the variation in MMI score for any given applicant (not explained by the interviewer or station). The variance component of the interviewers was small, suggesting that the interviewers gave relatively consistent ratings between them. The variance component of the stations was even smaller, suggesting that the variability in difficulty of the stations was low. The generalisability was G= 0.79 and the dependability was 0.73.
8.3 Applicant survey results of the MMI experience
Applicants, who presented for the MMIs in 2019, were invited to complete a survey following their MMI. There was a response rate of 99 %; fifty-six applicants identified as a school leaver and 45 as a non-school leaver, with 15 unknowns. The majority of applicants agreed (strongly or somewhat; n = 85, 80 %) that the information provided prior to the interview was clear and on the day of the interview was helpful (n = 92, 87 %). Most applicants (n = 80, 76 %) also agreed that the time allocated at each station was sufficient to respond to the questions asked and that the questions were easily answered (n = 73, 68 %). The majority of applicants agreed (strongly or somewhat; n = 89, 84 %) that efforts were made to help them feel relaxed (see Fig. 2).
In all, eight focus groups with between 1 and 10 participants were conducted in 2019. Five groups consisted of current students who had participated in the MMI process, two groups with MMI interviewers, and one group with hospital-based educators. Data from the focus groups were combined with data from the open-ended survey responses. Three main themes were identified in the data relating to the overarching unit of analysis, ‘the experience of participating in the MMI process’. These main themes were 'the organisation of the MMIs’, ‘the interviews’ and ‘the overall experience’. Within each of these three themes, there were also subthemes (see Fig. 3).
The administration of the MMIs was viewed positively overall. Two main subthemes formed the basis of this theme, ‘the structure of the MMIs’ and ‘the administrative organisation of the MMIs’.
9. The structure of the MMIs
9.1 Short segments with a variety of interviewers
Participants highlighted the benefit of the interviews being broken into small segments, making statements such as ‘I liked how it was run, and I liked that it was a variety of things, and it was broken up into pieces’. (Student FG 5). Participants liked that there were different people interviewing them and the subsequent advantage this offered in providing varying interviewer perspectives,
I liked the fact that it wasn't just, it was a mix of who it could've potentially been, there was educators, there was mothers apparently, there was midwives. and I think that's good because it's a multitude of perspectives (Student FG 1).
The structure of the interviews involving short segments with different interviewers meant that if the applicant felt they had not done well in an interview, they were able to regroup. Advantages are that it is a new interviewer each round so you have a fresh start (Non-school leaver applicant).
Other participants talked about how the segmented format, combined with the opportunity for applicants to ‘regroup’, made the experience more relaxing. ‘Gives a more relaxed way of answering your questions. Peace of mind if you don’t do well in one area you can excel in another’ (Non-school leaver applicant).
While the majority of participants saw being able to regroup as a positive, there were a couple of participants who felt that rather than being able to regroup they found it hard to move on.
I had a moment…where I just couldn't even articulate properly on that. I'm like, well.And I found it really hard to front up to the next one (Student FG 1).
9.2 Time allocated to answer the questions
While the small segments and variety of questions were viewed positively, there were varied comments raised in regards to the matter of timing. For a number of participants, the amount of time offered to answer questions was adequate, ‘It was enough time for you…’ (Student FG 5). For some applicants they wanted more time to answer the questions ‘Longer response times by 1 min’ (Non-school leaver applicant) and for others it was particular questions that needed extra time ‘having different time restraints for different questions (School leaver applicant). The interviewers also raised the issue of time, stating that they felt the time was generally sufficient but what was helpful was when they prepared the applicant by advising them how many questions would be asked in the interview "I'm going to ask you four questions," so that they know that they've got four questions to answer (Interviewer FG 2).
9.3 The administrative organisation of the MMIs
9.3.1 Organised and efficient
The MMIs were considered to be well run and organised by both applicants and interviewers, I feel like overall it was really well executed (Student FG 2) and ‘every time I've done it, it's been so organised and so well structured, and everybody is well informed and knows everything they've got to do’. (Educator FG).
9.3.2 Feeling supported
Another point in relation to the overall administration was that the applicants were made to feel comfortable and supported, ‘beautifully supported. Everyone made to feel as comfortable as you ever possibly could in that situation. (Student FG 2).
While the administration of the MMIs was viewed positively by the majority of participants, one participant felt it was too structured, stating ‘it was just too military, I felt like it was too detention like. (Student FG 2).
9.3.3 The Interviews
There were a number of comments about the interviews, and these fell into three main subthemes. These were ‘the questions’, ‘getting the best out of the applicants’ and ‘preparation for the interviews’.
9.3.4 The questions
The questions were perceived to be ‘very relevant…. (Student FG 5) and the variety of question topics cited as an advantage of the MMIs in the survey, ‘the variety of questions (Non-school leaver applicant).
Questions were also viewed as effective in drawing out the applicants’ passion for midwifery, ‘I have been able to glean that passion for midwifery from most students’. (Interviewer FG), even if English was not the applicant’s first language.
And I think those questions like the mini-interview questions draw that passion, that desire to be a midwife and that can come across even if English isn't your first language. (Educator FG).
Despite this, one of the points raised by some of the applicants and student participants was that they would have liked another question where they could talk about why they wanted to become a midwife and more about themselves, ‘I think what…could have improved was maybe like one extra interview where you could spill out anything to sell yourself. (Student FG 2).
9.3.5 Getting the best out of the applicants
The interviewers talked about wanting to get the best out of the applicants. They made statements such as.
What you're trying to do in this process, you're not trying to trick them. What you want to try and do is get a genuine, accurate assessment, unbiased assessment, of their capabilities and their ability, against the criteria that you've got in front of you (Interviewer FG 1).
Student participants gave positive examples of how interviewers did this. One participant relayed a story of how the interviewer sought to gain understanding when she had misunderstood a question.
I really don't know whether it was the way that I understood the question or… the way he asked it. halfway through the interview, he was saying, do you think this is appropriate?… and then I said, well I understand that this is a diary entry, is it not?. And he's like, no… he gave the opportunity to start again… (Student FG 5).
9.4 Interviewer training
To get the best out of the applicants, interviewers talked about the importance of the interview training making statements such as 'I feel that you wouldn't be prepared enough if you didn't undertake the training…. So, it is definitely like a mandate' (Interviewer FG2). Understanding the need to be unbiased and to focus on the answers was seen as a benefit of the training; ‘that's what the training did for me. It just helps me focus on what I need to focus on in interviews’ (Interviewer FG1) and ‘the training's quite structured and focused like that to prevent you from entering any of your biases’ (Interviewer FG1).
While the feedback on the interviewers was positive, one participant did say the neutral facial expression on a couple of the interviewers made her nervous.
Those two interviewers that had poker faces and I couldn't read them and I think that threw me off …But that was just my reaction and my response, I'm not sure if everyone would have had that (Student FG 2).
9.4.1 Preparing for the interviews
A number of participants talked about preparation for the interview questions and there were three different perspectives on this. Some of the participants in the study highlighted that it would have been good to be able to prepare for the interviews by having some idea of what to expect in the questions. They made comments such as ‘Give a topic/ topics to research (Non-school leaver applicant) and ‘More information given about questions i.e. sample questions to ensure participants can feel prepared’ (School leaver applicant). For some it was considered that being able to prepare in this way would have reduced stress levels ‘possibly some information regarding the questions beforehand would improve stress levels’ (School leaver applicant).
For other participants not being able to prepare was seen as less stressful and meant they could be more genuine, ‘Yeah. But like I said earlier, in a way, preparation can set up more anxiety and a little bit of like answering not so authentically’ (Student FG 3). Another participant shared:
I liked that we weren't told too much about it, because I feel like I didn't, I know maybe everyone's different, but like, I feel like I didn't get too anxious about it, because I didn't know too much what to expect. (Student FG 4).
For some participants they felt they had been able to prepare saying ‘I'd done some preparation by a couple of my friends who now got into medicine.. so some of it was quite similar' (Student FG 3) and ‘I did research on the internet and found that there, that format of interview existed in various capacities. So, I felt like I did preparation’ (Student FG 2). Another participant felt they were prepared based on attending open day at the university saying, ‘the questions themselves were actually quite similar to what I had expected having come to the open days and talked to academic staff, they didn't give me the questions, but I felt prepared’ (Student FG 1).
9.5 The overall experience
There were a number of statements made in relation to the overall experience of attending the MMIs and these were grouped into three subthemes, ‘nervous’, ‘a positive experience’, and ‘it was fair’.
9.5.1 Nervous
Participants talked about being nervous participating in the interviews,
Interviews are always..Nerve wracking, especially when you want something so much like the place in the course. If you care, naturally…, you're going to be nervous and stressed because you're thinking of the outcome if you blow it. I thought I blew it. (Student FG 1).
It was also considered that nerves could affect an applicant’s performance at the interview, ‘one of the negatives is that there are people who might be fantastic in all of those aspects, but the nerves disable them’. (Interviewer FG 1).
One of the participants did describe how she felt stress during the interviews had impacted on her ability to gain entry into the midwifery degree saying ‘The interview made me feel stressed and made it like impossible to get into the midwifery course’ (Unidentified applicant).
While a number of participants raised the issue of nerves, participants also pointed out that nerves are part of participating in any interview and particularly if you really want something. Nerves therefore, were considered unavoidable and to be expected, ’A bit stressful... But I mean you can't avoid that…. It’s going to be a thing with any competitive course, wondering whether you're going to get in’. (Student FG 1).
9.5.2 A positive experience
Even though several participants talked about being nervous, the MMI was considered overall, a positive experience by most participants ‘I think it is an extremely positive experience’ (School leaver applicant) and.
I don't really feel there's any disadvantages. I'm glad it happened the way it did, even though I did feel super nervous before (Student FG 1).
Finally, participants talked positively about the opportunity the MMIs provided for gaining entry into the degree. 'I honestly do feel like I wouldn't have gotten in if it weren't for those interviews' (Student FG 2).
9.5.3 Fair
Several participants remarked on the structure of the process in terms of fairness and equity. Making statements such as, ‘I feel that it gives everyone a fair opportunity to express their interests (School leaver applicant) and they ‘provided a fair evaluation for all participants (School leaver applicant).
Suggested reasons for considering the MMIs as fair and equitable included; ‘All the participants are asked the same questions by the same interviewer’ (Non-school leaver applicant); ‘Fair for each participant as they are given scenarios on the spot’ (School leaver applicant) and ‘Each person gets their chance with a set time period’ (School leaver applicant).
9.6 Integration of findings
A simultaneous bidirectional framework for convergent mixed methods design was used [
] with quantitative and qualitative results given equal weight. Quantitative and qualitative findings were examined in a back and forth process, looking for confirming and disaffirming elements related to the fairness, equity, and experiences of stakeholders. Two of the research team initially examined the data separately and then together, discussing findings to reach agreement for interpretation. Further discussion with all authors occurred with final consensus reached. Qualitative results and quantitative results supported and complemented each other in a number of instances and were able to provide an enhanced understanding of the MMI processes and stakeholder experiences. Integration demonstrated that the MMI as part of the selection process for the B Mid provided a fair and equitable experience. The key themes derived from the qualitative content analysis provided a useful framework to support and explain the integrated findings, with the merged data presented as a joint display (Table 1).
Table 1Integration-Joint display.
Major theme
Subthemes
Summary of qualitative data
Quantitative findings
Mixed methods interpretation
The organisation
The structure
Short segments with a variety of interviewers I liked that it was a variety of things. it was broken up into pieces. (Student FG5).
it was a mix… I think that's good because it's a multitude of perspectives…(Student FG1).
. I liked the fact that you could step out of a room, go "okay, hold on, maybe I didn't do that as well as I'd like to, I can pick it up the next time…” (Student FG1).
I found it really hard to front-up to the next one (Student FG1)
Participants viewed the structure and organisation of the MMIs positively, The short segments with different interviewers meant that the majority of participants felt they could start afresh if they considered they had not done well in one room. Only, a couple of participants said they found it hard to move on. These two participants were both students who gained entry into the degree; therefore it could be argued that they did well in the interviews despite their concerns. Participants particularly liked that the interviewers were not all midwives or academics but were drawn from a broader range of society and thus provided a broad range of perspectives. Applicants themselves rather than interviewers provided the broadest variance in the MMI scores, suggesting a broad range of perspectives were able to draw out differences in the applicants whilst maintaining consistency across scoring. There were different perspectives in relation to the time allocated to the interview stations. Despite this, the majority of applicants (80%) reported there was sufficient time to answer questions, suggesting overall, the time allocated to each station was sufficient. A suggested strategy to ensure maximum use of time was to let the applicant know at the beginning of the interview, the number of questions they would be asked.
Time allocated to answer questions ‘timing was good’ (Non-school leaver),'
Longer response times by 1 min’ (Non-school leaver)
‘Different time restraints for different questions' (School leaver applicant)'
I'm going to ask you four questions, so that they know that they've got four questions to answer' (Interviewer FG2)
Q3 Applicant survey 80% felt that the time was sufficient to answer questions
The administrative organisation
Organised and efficient' I feel like overall it was really well executed'(Student FG2)'
Every time I've done it, it's been so organised and so well structured, and everybody is well informed …'(Educator FG)'
It was just too military' (Student FG2)
Q1 Applicant survey 85% agreed information sent prior to the MMI’s gave clear instructions for booking and attending the MMI.
The MMI’s were viewed as well organised and well-structured with information both prior to and on the day of the interviews beneficial for preparing applicants. One participant felt they were too structured, that it was too military, and viewed this as negative. However, this also supports the claim that the MMI’s were structured to be consistent and as such all applicants were treated the same. Both the qualitative and quantitative data demonstrated that organisers made efforts to support the applicants. Support was provided on the day, including briefing prior to the interviews and debriefing of applicants after the interviews. This demonstrated that the organisers were aware that the applicants could feel overwhelmed, and efforts were made to ameliorate this.
Feeling supported' I remember them being very supportive and checking in and asking if we needed anything…And I could see that the facilitators were taking specific interest in people who might be a bit overwhelmed' (Student FG2)
Q2 Applicant survey 92% agreed that information on the day was helpful
Q5 Applicant Survey 89% agreed efforts were made to help them feel relaxed
The interviews
The questions
Relevant
‘It was very relevant… (Student FG5)'
I think those questions…draw that passion, that desire to be a midwife… even if English isn't your first language' (Educator FG)'
One extra interview where you could spill out anything to sell yourself '(Student FG2)'
The variance in the MMI score was mostly attributed to the candidate at 31.4%, followed by the interviewer at 11%and 6.4%, the station
G= 0.79 and the dependability was 0.73.
Q4 Applicant Survey 73% agreed that the questions asked were easily answered
Key variance in MMI scores was derived from applicants. There was very little variance between interviewers, or stations. Qualitative findings supports that the interviewers were overall supportive, wanting applicants to do their best.
Two participants felt that a couple of interviewers were stern/cold and felt this may have affected their performance but acknowledged that this may have been their own perception and not the feelings of other applicants.
Interview training was structured to reduce bias.
There were different perspectives in relation to applicant preparation for the MMIs; overall, however, the majority of applicants considered the questions easily answered.
Getting the best out of the applicants
You want them to give their best performance in six minutes…. to give each of these individuals the best possible opportunity' (Interviewer FG1).'
… he then said let's start again. …, he gave the opportunity to start again…' (student FG5)'
one guy was a little bit cold and I think that threw me off…But that was just my reaction'(student FG2)
‘the training's quite structured and focused like that to prevent you from entering any of your biases’ (Interviewer FG1)'
Preparing for the interviews
More information given about questions i.e. sample questions to ensure participants can feel prepared’ (school leaver applicant)' preparation can set up more anxiety and a little bit of like answering not so authentically’(student FG3)'
I did research on the internet. So, I felt like I did preparation’. (student FG2)
The overall experience
Nervous
‘even if you don't have anxiety, it's very anxiety inducing. It's stressful and that could affect their performance' (student FG2)
‘The interview made me feel stressed and made it like impossible to get into the midwifery course’ (unidentified applicant)'
Interviews are always..nerve wracking, especially when you want something so much like the place in the course. If you care, naturally then you, you're going to be nervous and stressed' (student FG1)'
Age: R2 = 4% (p = 0.01) Country of birth R2 = 0.3% (p = 0.48) Language other than English spoken at home: R2 = 3.1% (p = 0.03) FIF: R2 = 3% (p = 0.03)
Some participants noted being nervous could negatively affect MMI performance. However, despite this concern some participants who raised this concern had in fact been successful at MMI and offered entry into the course, suggesting nerves may not have had a significant impact on them despite their concerns.
Participants also acknowledged that having ‘nerves’ was normal and as such unavoidable.
Despite experiencing nerves, most participants felt that the interviews provided a positive experience, as well as an opportunity to gain entry into the degree. Participants used words such as ‘positive’, ‘an opportunity’, and having ‘no disadvantages’. Similarly other participants described the MMI’s as ‘fair’.
Statistical analysis did however; demonstrate that students who spoke a language other than English at home and who were FIF to attend university had on average lower MMI scores while students who were older had on average higher MMI scores. These findings could suggest older applicants are more likely to gain entry into the degree while applicants who speak a language other than English at home are less likely to gain entry. Because the analysis was based on students enrolled in the course this cannot be confirmed. While there were differences in MMI scores for those students who were FIF, it is likely that this relationship is not causal but due to a confounder (such as language spoken at home). Additional statistical models are needed to better understand what factors influence the MMI scores of a student who is FIF to attend university. Country of birth was not shown to influence the MMI score. No participants raised any of these variables as concerns in the qualitative data.
It was a positive experience
I don't really feel there's any disadvantages'. (student FG1)' I know a lot of the young ones in our class…they really liked that option (the interview),' (Student FG2)'
I wouldn't have gotten in if it weren't for those interviews.' (student FG 2)
It was fair
‘provided a fair evaluation for all participants' (school leaver applicant).
All the participants are asked the same questions…(non-school leaver applicant)
‘Fair for each participant as they are given scenarios on the spot’ (school leaver applicant)
‘Each person gets their chance with a set time period' (school leaver applicant).
While a number of studies have evaluated MMIs, this is the first known study evaluating the use of MMIs when selecting applicants for a Bachelor of Midwifery in Australia. Using a convergent mixed methods design, we collected both quantitative and qualitative data to explore stakeholder experience and to examine aspects of fairness and equity with MMIs. Integration of data enhanced findings.
Similar to one of the objectives of our study, a number of other researchers have specifically looked at applicant demographics in relation to the equity and fairness of MMIs, including age [
The predictive validity of Multiple Mini Interviews (MMIs) in nursing and midwifery programmes: Year three findings from a cross-discipline cohort study.
Exploring sociodemographic subgroup differences in multiple mini-interview (MMI) performance based on MMI station type and the implications for the predictive fairness of the Hamburg MMI.
Exploring sociodemographic subgroup differences in multiple mini-interview (MMI) performance based on MMI station type and the implications for the predictive fairness of the Hamburg MMI.
Exploring sociodemographic subgroup differences in multiple mini-interview (MMI) performance based on MMI station type and the implications for the predictive fairness of the Hamburg MMI.
]. Our findings demonstrated that on average, older students achieved higher MMI scores, while those who spoke a language other than English at home and who were first in family to attend university, achieved lower MMI scores. No association between MMI score and students who were born overseas was found.
Previous research reports varying results in the association between age and MMI scores. In some studies, age was not found to influence MMI scores [
The predictive validity of Multiple Mini Interviews (MMIs) in nursing and midwifery programmes: Year three findings from a cross-discipline cohort study.
] concluded that the positive association between age and MMI scores could be explained by older applicants having more life experiences and consequently more to discuss at interview. This is in line with other research, indicating applicants who choose to pursue a career in midwifery often do so based on their own personal experiences of giving birth and interacting with midwives [
] identified that non-EU students whose first language was not English achieved significantly lower scores on MMI than EU and English speaking students. It is therefore not unreasonable to expect that the language spoken at home might influence an applicant’s MMI score. Despite these findings, 25 % of accepted students in our study cohort were not born in Australia and one in five spoke a language other than English at home.
The ability to communicate effectively in a country’s native language is an inherent requirement for student learning and clinical success. In support of our findings, differences between language spoken at home and MMI scores, have also been shown in countries where English is not the native language. Leduc et al. (2017) [
] who examined the association between socio-demographics and MMI scores in students admitted into a French-speaking medical school in Canada found that French-speaking applicants were more likely to have higher MMI scores. Similarly, Knorr et al. [
Exploring sociodemographic subgroup differences in multiple mini-interview (MMI) performance based on MMI station type and the implications for the predictive fairness of the Hamburg MMI.
Exploring sociodemographic subgroup differences in multiple mini-interview (MMI) performance based on MMI station type and the implications for the predictive fairness of the Hamburg MMI.
] argue that while language is likely to play a role in the MMI process, language proficiency is also integral in the clinical situation. This is certainly true of midwifery where good communication is an essential component of clinical practice. The Nursing and Midwifery Board of Australia [
] require applicants applying for initial registration as a midwife, to demonstrate competency in both speaking and communicating in the English language. Ultimately Knorr argued, that differences in language proficiency itself might not indicate that the MMI process is unfair because “while vocabulary and grammar do not have to be flawless, recipients should comprehend the intended meaning” [
Exploring sociodemographic subgroup differences in multiple mini-interview (MMI) performance based on MMI station type and the implications for the predictive fairness of the Hamburg MMI.
]. Given the importance of language to the registration requirements for a midwife in Australia, our findings also suggest that differences in language, proficiency might not indicate that the MMI process is unfair but rather related to the need for good communication.
Despite the importance of language to communication, both Knorr and Leduc argued strategies to minimize linguistic bias in the MMI need to be taken into account. Leduc suggested using ‘clear and unambiguous language” [
Exploring sociodemographic subgroup differences in multiple mini-interview (MMI) performance based on MMI station type and the implications for the predictive fairness of the Hamburg MMI.
] (p.243). In our study, the interviewers felt they were able to identify an applicant’s passion for midwifery even if English was not their first language. Questions in our study were viewed as relevant and appropriate to midwifery, and for most applicants easily answered, thus had face validity. The variance component of the stations was also small suggesting the variability in difficulty of the stations for each applicant was low. In our study, generalisability was of a similar or greater level to what others have reported. For example, in the evaluation of MMIs used as part of the entrance process to a medical school in Brazil, Daniel-Fihlo et al. reported a generalisability (G) of 0.743 [
]. Overall, these findings suggest our process was robust.
Despite differences in age and language spoken at home, participants in this study viewed the MMIs positively and a theme emergent from the qualitative content analysis was that the MMIs as part of the admissions process for the Bachelor of Midwifery was fair. A number of other studies using qualitative methods have also found participants support the use of MMIs, consider the process fair [
Sheehan A., Thomson R., Pierce H., Arundell F. The impact of Multiple Mini Interviews on the attrition and academic outcomes of midwifery students. Women and Birth 2021.
]. In this study, participants particularly highlighted that the MMIs were fair because each applicant was asked the same questions, on the ‘spot’ and with the same amount of time to answer. In addition, the MMIs were structured so that all applicants were treated equally, receiving the same information prior to and on the day of the interviews, with the majority of participants reporting they felt supported.
Another key finding from this study was that participants particularly valued that interviewers came from a broad cross section of the community, and were not just midwives or academics. Our statistical analysis supported the validity of using interviewers from diverse backgrounds by demonstrating very little variance between raters. These findings demonstrate that a broad range of perspectives can draw out differences in the applicants whilst maintaining consistency across the scoring. These findings support the work of Baker et al. [
] who identified that while there was some variability in the MMI scores between raters from different backgrounds, greater variability in MMI scores was attributable to the applicant’s performance. What our work adds to the findings of others is that not only does using a variety of interviewers from different backgrounds appear to pose no threat to the integrity of the MMI process; it is preferable from the viewpoint of key stakeholders.
Another possible element relating to consistency of interviewer scores was the requirement that all interviewers in the WSU MMIs receive training. Training in our study was reported as essential to preparation for the interviews and in helping minimise bias. Our findings support other research demonstrating interviewer training reduces bias [
]. In a study using content analysis of focus group data of candidate and interviewer experiences of MMIs, the importance of interview training to promote a shared understanding by interviewers of what is being assessed, and areas of potential interviewer bias was emphasised [
]. Both these content areas are covered in the interview training provided at WSU.
Our findings indicated that having the same time allowance for each station contributed to the fairness of the MMIs however, perspectives varied on the adequacy of time allocated to each MMI station. A recent review of perspectives on MMIs, found adequate reliability of the MMIs with station durations ranging from five to eight minutes [
] explored the perceptions of MMI interviewers at three different time frames; pre MMI training, post MMI training and post-interview, and found that the MMI interviewers considered 6 min sufficient time to assess applicants to a pharmacy degree. Our findings that 76 % of students reported they had adequate time to answer the questions, compares favourably to other studies. For example in a study by Cameron et al., 47 % of applicants rated six-minute stations as ‘‘just right’’ while 50 % considered them ‘‘a bit short’. For the eight minute stations, 50 % of applicants considered the timing ‘‘just right’’ and 43 % ‘‘a bit long’’ [
]. The one station in our study particularly highlighted as needing more time was the video station. A suggested strategy by participants to ensure best use of time was to let the applicant know at the beginning of the interview the number of questions at the station. Follow-up to see if this added strategy has a benefit to maximizing the use of time would be useful.
Finally, participants in this study described feeling nervous, with some concerned that nerves could affect performance at the MMIs. Other researchers have suggested that nerves are a limitation of MMIs [
Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37.
]. Although some participants in this study were concerned that nerves could affect their performance, others reflected that nerves did not affect performance as negatively as anticipated. Participants also acknowledged that nerves were a normal experience particularly when wanting to achieve access into a desired degree. Christensen et al. suggested that raters had a ‘distaste’ for candidates who displayed nervousness [
] but this contrasts with our findings where interviewers reported wanting to get the best out of applicants and organisers were reported as making efforts to support applicants and to help them feel relaxed on the day. Some participants suggested that preparation would have been helpful to reduce stress levels. These comments were however, in contrast to other participants who felt that not being able to prepare made the process less stressful. Previous research has found that preparation does not improve MMI scores [
Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37.
Statistics examining the association between socio-demographics could only be based on students who enrolled in the B Mid. Results therefore, are not reflective of all applicants who were offered a place in the B Mid. Identified relationships between the MMI scores and socio-demographics showed association not causation, therefore, it is possible that there were confounders on the causal pathway that could have impacted findings.
11. Conclusion
This study is novel as it is the first study to use mixed methods to evaluate the fairness and equity of MMIs used for admission into the Bachelor of Midwifery at one university in Sydney, Australia. While an applicant’s age and language spoken at home may influence their MMI score, this finding does not necessarily indicate a lack of fairness or equity, but can be explained by other possible factors such as life experience, and the requirement for language proficiency. The use of a variety of interviewers was an effective strategy, maintaining the integrity of the MMI scoring, with participants deeming this preferable to having all interviewers as midwives or academics. Our findings demonstrate applicants were treated equally, through the organised administration, provision of information prior to and on the day of the MMI, and by efforts made to support students on the day. The applicants considered the MMIs a fair assessment as well as an opportunity to gain entry into the B Mid. Overall, MMIs were viewed positively and considered fair and equitable.
Acknowledgements
This project was funded by a Western Sydney University School of Nursing and Midwifery Partnership Grant with the NSW Ministry of Health: Nursing and Midwifery Office.
References
Kale S.
Kamble M.W.
Spalding N.
Predictive validity of multiple mini interview scores for future academic and clinical placement performance in physiotherapy, occupational therapy and speech and language therapy programmes.
Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37.
Medical students' experiences of the benefits and influences regarding a placement mentoring programme preparing them for future practice as junior doctors: A qualitative study.
Sheehan A., Thomson R., Pierce H., Arundell F. The impact of Multiple Mini Interviews on the attrition and academic outcomes of midwifery students. Women and Birth 2021.
The predictive validity of Multiple Mini Interviews (MMIs) in nursing and midwifery programmes: Year three findings from a cross-discipline cohort study.
Exploring sociodemographic subgroup differences in multiple mini-interview (MMI) performance based on MMI station type and the implications for the predictive fairness of the Hamburg MMI.
00308-0&id=gr1.jpg){kind=link}
00308-0&id=gr2.jpg){kind=link}
00308-0&id=gr3.jpg){kind=link}