Women and Birth
Volume 23, Issue 1 , Pages 36-40, March 2010

An urgent call to implement systematic monitoring of a comprehensive set of quality indicators for maternity services

The University of Queensland, School of Nursing and Midwifery, Herston Campus, Brisbane, QLD 4029, Australia

Received 29 July 2009; received in revised form 6 November 2009; accepted 9 November 2009.

Article Outline

Summary 

This commentary paper proposes that historically the lack of significant system wide reform of the sector may be in part related to the quality of evidence that has informed reviews of maternity services. It calls for the implementation of systematic monitoring of a comprehensive set of quality indicators and makes suggestions about how this goal might be achieved.

Keywords: Maternity reform, Maternity review cycle, Consumer-assessed value, Performance indicators, Perinatal outcomes

 

The safety and quality of maternity services are paramount to a healthy nation. The recent national Maternity Services Review recognised that current arrangements for the delivery of maternity care do not serve all women as well as they could.1 As an Australian midwife I welcomed the 18 recommendations of Improving Maternity Services in Australia and celebrated the recent Federal Budget announcements. I hope the recommendations may be acted upon and structural reform give rise to significant system wide change in the provision of maternity services in this country. However, given previous reviews of maternity care have failed to result in significant system wide reform of the sector, there is a risk that these recommendations may not eventuate.

This commentary paper proposes that historically the lack of significant system wide reform of the sector may be in part related to the quality of evidence that has informed reviews of maternity services. It calls for the implementation of systematic monitoring of a comprehensive set of quality indicators and makes suggestions about how this goal might be achieved.

If one examines the history of maternity services reviews, a cyclical pattern of activities can be seen. Like other social cycle theories, this pattern begins with dissatisfaction being expressed with the status quo, which results in political lobbying and agitation, followed by the announcement and implementation of a review process, the handing down of recommendations for reform and a subsequent period of temporary political appeasement. A perceived lack of implementation of recommendations and reform may stimulate further iterations of the maternity care reform cycle (see Fig. 1).

While it may be argued that the most recent review and reform agenda differs from earlier reviews in that it carries political will, those midwives who have been strong advocates for maternity service reform can attest to the transient nature of political fortunes. It is timely that we pause and reflect upon other factors that may have contributed to the failure of the implementation of previous reviews. This means calling in to question the quality of the evidence used to inform the process of reform.

The national Maternity Services Review is not dissimilar to the previous thirty-seven (37) national, state and regional reviews of maternity services conducted in Australia since 1990. All would lay claim they were informed by ‘evidence’, namely comments and submissions, which have been augmented by research and expertise in order to guide an approach to the change of services. My involvement in the maternity care review cycle reflects the situation in Queensland although I suspect this is not dissimilar to those of other Australian states and territories.

The most recent of the preceding reviews and the resultant report entitled Rebirthing2 was conducted in Queensland and recognised there are limits to evidence with respect to maternity care. These are strongly influenced by social and cultural values held by consumers, care providers and policy-makers alike. Pregnancy and birth are significant and symbolic life events often characterised by strong beliefs and convictions. For many women, interaction with the health sector during episodes of pregnancy and birth may be perceived as a satisfactory experience and they may see no need for reform. These women may be less likely to respond to invitations to inform a review process, than those who are dissatisfied and actively seek reform. This so-called respondent bias3 may be inherent in evidence used to inform review processes. Additionally, expertise used to enlighten the review reflects the professional and personal values of the experts themselves and also subject to a range of biases (for an exhaustive list see Sackett3).

One could argue that balanced against this value-laden evidence are more objective measures of maternity care outcomes. The history of perinatal data collection varies between states. In Queensland the process of perinatal data collection commenced in 19864 when legislative changes required perinatal data be provided for every child born. National Perinatal Data Collection (NPDC) began in 19915 with outcomes reported on live and still births of at least 400g birth weight or 20 weeks gestation. Initiated in 1997, the Perinatal National Minimum Data Set (NMDS) specifies data elements for mandatory collection and reporting at a national level. These items include data relating to the mother, such as demographic characteristics and factors relating to pregnancy, labour and birth, as well as those relating to the baby: birth status; sex; and birth weight.

While more data elements are specified in the NPDC than the NMDS, both datasets have a primary surveillance function and are clinician defined, i.e. the data elements have been identified by clinicians as important perinatal markers. These datasets provide a crude measure of the outcomes of clinical care6 which limits the utility of the data, and it could be argued the datasets lack sensitivity with respect to quality improvement and policy reform. Likewise, the most recent addition to the suite of statistical surveillance programs from the National Perinatal Statistics Unit – the Australian Maternity Outcomes Surveillance System (AMOSS) – focuses on reporting markers and improving knowledge of rare obstetric disorders and their management. Although there is value in surveillance as an end in itself, there is a tendency for surveillance data to be cited as evidence of service effectiveness, quality and safety in the absence of more purposeful evaluation data. If service evaluation is not identified as a goal of surveillance it is unlikely to contribute to quality improvement for maternity services for the majority of consumers.

Tracking crude trends in mortality and morbidity may be useful as part of the evaluation of highly targeted health services.7 However, the episodic nature of maternity care, the long lag time between compilation and release of reports,6 the current plethora of planned reform measures and the significant influence of factors outside the system of care means that there is a heavy reliance on systematic, contextual interpretation of the data. In Queensland the capacity to interpret outcomes or explain emergent trends has undergone a hiatus following the demise of the Perinatal Quality Councils and the establishment of the Maternity Services Clinical Networks. The lack of secure, long-term funding also makes it difficult to examine systematic issues that address prevention.6

Attempts to capture the experiences of consumers vary between states. The most established surveillance work is in Victoria beginning in 1990s8 and continuing with the Healthy Mothers, Healthy Babies research group. This research provides longitudinal mapping of the health of mothers and babies and can be used to assist evaluation of the impact of changing policy and practice. However, in terms of other statewide population-based surveys, little comparative work has been conducted and where attempts have been made, they fall short of expectations. For example, a Post-Birth Survey (PBS) was conducted in Queensland in 2006/2007.9, 10 This is part of a suite of triennial patient satisfaction surveys and focuses on access to advice on baby care, midwifery continuity, care, length of stay, breast feeding choices, anxiety and referral to child health services. This survey demonstrated several limitations despite its intention to provide a consumer voice to birthing services. Restriction of the survey sample to public services, its 3-year cycle and the limited number of items may have inhibited the potential impact of any interpretation of the findings to maternity services, policy and provision. Administration of the survey 3–6 months after birth has resulted in considerable variation in mothers’ recall of having received advice about important aspects of post-birth.11 Gratitude for the care received and fear of closure of smaller units may result in over-estimation of satisfaction. Women are unlikely to express dissatisfaction with services if they are ignorant of alternatives.12 Consequently it would be problematic to use such a survey to shape policy or reform of maternity services.

It seems that the evidence being collected is incomplete and problematic as a starting point on which to base reform, or indeed measure changes in the quality of service provision or the impact of policy changes. Reviews of maternity services tend to be biased in their sampling, focusing on the experiences of the consumer minority and determining whether their needs have been met. Conversely, perinatal data surveillance may sample data from institutional and population health perspectives but it is insensitive to those measures that represent quality of process or experience. How we portray the relative importance of the individual in relation to the population and the processes of care against their outcomes has a far reaching influence on what and how we measure in terms of maternity services provision and on policy direction.

To date, using these two paradigms to measure the performance of the maternity care system has failed to act as a catalyst for reform or adequately measure safety and quality of services or determine outcomes of service improvement. Instead, evaluation of the system has stalled at either recommendations or reported surveillance and falls short of policy-driven reform. The recommendations of the national review recognise the impact of this shortfall: “…there are areas where gaps in our knowledge are resulting in discord among professionals and consumers rather than providing useful feedback to inform policy and practice” (p. 13).

To determine the value of the maternity service system and the impact of planned reforms, attention must be directed towards filling the gaps in current data through evaluating what Duckett and Ward7 describe as ‘patient assessed value’. “At the moment financial/efficiency indicators are most widely available and precise clinical indicators are usually more widely available but often less precise, and patient indicators hardly available at all, often poorly specified, rarely linked to the other dimensions” (p. 7).

A robust set of performance indicators should be developed which include timeliness of access and cost to the consumer.7 Perhaps, in the context of maternity services, these indicators could be more fittingly referred to as consumer-assessed value indicators and these could be linked to financial efficiency and clinical perinatal indicators. The need to engage a broad representation of consumers in the development of a set of consumer-assessed value indicators is self-evident.

Without pre-empting the possible outcomes one could postulate that consumer expectations and experiences might be captured in potential indicators demonstrating that: care choices are available and individual choice is supported; care is safe and feels safe2; individual beliefs are valued and respected; care is coordinated across care disciplines; continuity of care and/or provider is provided; and the uniqueness of the mother and her family's experiences are acknowledged.

The inclusion of consumer-assessed value indicators in a systematic approach to measuring effectiveness of maternity services may complement existing data and provide a more complete assessment of the effectiveness of services. Indeed the recommendations of the Maternity Services Review that the government. “initiate targeted research aimed at improving the quality and safety of maternity services in select priority areas, such as evidence around interventions, particularly caesarean sections and maternal experiences and outcomes, including from postnatal care (p. 57) appears to support this view. Additionally the recommendation for consistent, comprehensive national data collection, monitoring and review” (p. 57) should extend beyond maternal and perinatal mortality and morbidity.

Sound epidemiological research methods should be used to focus on the views, expectations and experiences of women giving birth. The most appropriate study design to achieve this is the time series cross-sectional study. Mixed-mode data collection strategies (telephone interviews, postal surveys, web surveys) could be used to tailor design to maximise response rates and coverage.13 It should be noted that although web and online survey methods are a recent phenomenon the uptake of these has steadily increased with internet access and issues relating to online methods “are transitional in nature in a time of rapid technological change” (p. 1309).14 It is imperative to develop data collection and recruitment methods in order to enhance the representativeness of sampling across all settings and consumer groups.

A number of international studies have employed varied sampling frames. The series of national surveys conducted in the United States (US)15, 16, 17 regarding the birth and postpartum experiences of mothers drew on a very small selective sample (<0.04% of the birthing population) from an existing commercial database. Those women enrolled in a commercial survey database may not be typical of the wider population so results should be generalised cautiously.18

A national Canadian study18 used 2006 census data as a sampling frame to recruit a large (n=8542) sample of women. However, the use of national census data as a sampling frame in Australia would be subject to a long time lag and this would ultimately affect the quality of the data. Birth is a sentinel event and the ability to recall care, services and experiences may vary over time. Therefore, to overcome this, the use of a sampling frame occurring immediately post-birth and at other extended postnatal points should be considered. Additionally, sampling periods should be varied in order to capture populations across seasonal and birth rate variations.

An Australian national study would need to employ varied sampling, recruitment and data collection strategies. Specific strategies would be required to ensure marginalised groups, such as Aboriginal and Torres Strait Islanders, who typically exhibit low response rates and emergent population groups, such as West African migrant women, are represented. Models of engagement for under-represented communities include the recruitment of culturally specific researchers, the development of appropriate recruitment and data collection strategies and recognition that community research partnerships require a longer lead-time to yield data.

Recruitment of mothers into a national approach would be ideally undertaken prior to post-birth discharge. Although this places a burden on service providers, the use of time series cross-sectional design could focus intensive recruitment for specified period each year. All mothers who birth during this time could be recruited to participate in the study. Birth data could be verified by linking surveys with birth register and perinatal data for the period. Response rates from other surveys of birthing mothers indicate that a high level of participation can be anticipated. The Canadian Maternity Experiences Survey reported a response rate of 86% to a pilot survey19 followed by 73% to the actual telephone survey.20 Similarly, a cross-sectional study utilizing a written questionnaire distributed to women birthing in Scotland over a 10-day period reported a response rate of 69%.21 Although recent survey experience in the UK indicates that response rates may be declining.22

Australian studies of birthing mothers demonstrate similar response rates for postal surveys of 71.4%, 62.5% and 67% in 1989, 1994 and 2000, respectively.23 The use of multi-modal response methods could result in response rates in excess of this and extend beyond the use of paper-based surveys and telephone data collection to web-based survey methods. The internet provides significant opportunities for innovation of web survey design24 and may improve the timeliness of reporting and uptake of results. Electronic survey design tools such as e-Cohort Pro have been developed by the University of Queensland to follow large longitudinal cohorts of nurses and midwives (The Nurses and Midwives e-Cohort Study http://nurses.e-cohort.net/) and doctors (The Doctors e-Cohort Study http://doctors.e-cohort.net/) might be extended to undertake national surveys of clinical practice and improve response rates and thus sample size.

A sufficient and representative sample size could allow for analysis based upon the: sector in which care was provided (public or private); model of care and care provider accessed; health service district; and population subgroups.18 Additionally, the concerns of attrition and losses-to-follow-up which impact other longitudinal research methods would not be a consideration as the focus of a national approach would be on tracking trends rather than individuals over time.

A number of existing relevant population-based data collection instruments might be considered in the development of this national approach. Replication of the widely reported Listening to Mothers and New Mothers Speak Out surveys15, 16, 17 should be considered in light of possible international comparison. However, these survey instruments have been developed within the US context and, although regional modification would be possible, the sampling strategy used in these studies may not be representative. As such, the value of comparison may be limited.

International comparison with national surveys conducted in Scotland25 England22 and Canada20 exhibit more robust representative sampling. However, the Victorian work provides a valuable foundation in an Australian context and as well as offering the potential for comparison. Instruments drawn from the Victorian Healthy Mothers Healthy Families Survey,26 could be extended to include consumer-assessed value indicators, modified where appropriate and piloted prior to the commencement of sampling.

Maternity services are on the brink of significant reformation, and this process should occur in an informed manner. Previous reviews of maternity services have fallen short of the reform required for significant system wide change and this may be due to the quality of evidence that has informed these reviews. Data that are currently available is also problematic. Perinatal surveillance data reports crude outcomes of clinical quality that are subject to time lag, while statewide surveys are of variable quality and some may tend to encourage the status quo or have limited application in the direction of service reform or policy development.

The maternity care review cycle should be broken and to do so requires a national population-based approach to generate comprehensive data about birthing services. Existing survey and data collection instruments should be carefully considered in terms of their applicability, validity and potential for international versus national comparison. Clinical and efficiency indicators need to be augmented by consumer-assessed value indicators which give voice to women's views, expectations and experiences of birth and the postnatal period. Indicators of consumer-assessed value of maternity services should be developed and incorporated into the data set and monitored over time to determine the impact of policy and reform on the quality of process, outcomes or experience of maternity consumers.

A comprehensive national approach to data collection will require careful consideration of the sampling frame with respect to balancing data quality and potential biases. Robust strategies for recruitment, sampling and data collection will be imperative to achieve the goal of “improved national data collections and targeted research to support a safety and quality framework and allow the impact of changing models of care to be effectively monitored” (p. 2).1

Back to Article Outline

References 

  1. Commonwealth of Australia. Improving maternity services in Australia: a discussion paper from the Australian Government; 2009.
  2. Hirst C. Re-birthing report of the review of maternity services in Queensland. Brisbane; 2005.
  3. Sackett DL. Bias in analytic research. Journal of Chronic Disease. 1979;32:51–63
  4. Queensland Health. Perinatal statistics Queensland 2006. Brisbane: Queensland Government 2008; March 2008. Contract No.: 9th October.
  5. Lancaster P, Huang J, Pedisich E. Australia's mothers and babies 1991. Sydney: Australian Institute of Health and Welfare; 1994;
  6. Kildea S, Pollock W, Barclay L. Making pregnancy safer in Australia: the importance of maternal death review. Australian and New Zealand Journal of Obstetrics and Gynaecology. 2008;48:130–136
  7. Duckett S, Ward M. Developing ‘robust performance benchmarks’ for the next Australian Health Care Agreement: the need for a new framework. Australian and New Zealand Health Policy. 2008;5(1):
  8. Brown S, Lumley J. The 1993 survey of recent mothers: issues in survey design, analysis and influencing policy. International Journal for Quality in Health Care. 1997;9(4):265–275
  9. Queensland Government. Post-birth consumer survey 2006; 2006.
  10. Queensland Government. Post birth consumer survey 2006. Technical report. Brisbane: Office of Economic and Statistical Research; 2007.
  11. Queensland Government. 2006/07 patient satisfaction—maternal birth survey; 2008. Available from: http://www.health.qld.gov.au/quality/pat_sat_survey/listening.asp [accessed 05.11.09].
  12. van Teijlingen ER, Hundley V, Rennie A-M, Graham W, Fitzmaurice A. Maternity satisfaction studies and thier limitations: “What is, must still be best”. Birth. 2003;30(2):75–81
  13. Dillman DA, Smyth JD, Melani Christian L. Internet, amil and mixed mode surveys: the tailored design method. third ed.. Hoboken, NJ: John Wiley & Sons; 2009;
  14. Huntington A, Gilmour J, Schluter P, Tuckett A, Bogossian F, Turner C. The Internet as a research site: establishment of a web-based longitudinal study of the nursing and midwifery workforce in three countries. Journal of Advanced Nursing. 2009;65(June (6)):1309–1317
  15. Declercq E, Sakala C, Applebaum S, Risher P. Listening to mothers: report of the First National U.S. Survey of Women's Childbearing Experiences. New York: Maternity Centre Association; 2002;
  16. Declercq E, Sakala C, Corry M, Applebaum S. Listening to mothers II: report of the Second National Survey of Women's Childbearing Expereinces. New York: Childbirth Connection; 2006;
  17. Declercq E, Sakala C, Corry M, Applebaum S. New mothers speak out: national survey results highlight women's postpartum experiences. New York: Childbirth Connections; 2008;
  18. Chalmers B, Dzakpasu S, Heamann M, Kaczorowski J, Szakacs HM. Conducting a national survey of women's perinatal expereinces in Canada. Canadian Journal of Public Health. 2007;(July–August):281–283
  19. Dzakpasu S, Chalmers B. Canadian maternity experiences pilot study. Birth. 2005;32(1):34–38
  20. Public Health Agency of Canada. Maternity experiences survey (MES); 2007 [27th October 2008].
  21. Hundley V, Rennie A-M, Fitzmaurice A, Graham W, van Teijlingen ER, Penney G. A national survey of women's views of their maternity care in Scotland. Midwifery. 2000;16:303–313
  22. Redshaw M, Rowe R, Hockley C, Brocklehurst P. Recorded delivery: a national survey of women's experiences of maternity care. National Perinatal Epidemiology Unit 2006; 2007.
  23. Bruinsma F, Brown S, Darcy M-A. Having a baby in Victoria 1989–2000: women's views of public and private models of care. Australian and New Zealand Journal of Public Health. 2002;27(1):20–26
  24. Dillman DA, Smyth JD. Design effects in the transition to web-based surveys. American Journal of Preventive Medicine [Proceedings Paper]. 2007 May;32(5):S90–S96
  25. Scottish Programme for Clinical Effectiveness in Reproductive Health . Maternity care matters an audit of maternity services in Scotland. Aberdeen: Dugald Baird Centre for Research on Women's Health; 1999;
  26. Murdoch Children's Research Institute. Victorian healthy mothers healthy families survey. Available from: http://www.mcri.edu.au/Downloads/HMHF/MothersSurvey.pdf.

PII: S1871-5192(09)00087-0

doi:10.1016/j.wombi.2009.11.001

Women and Birth
Volume 23, Issue 1 , Pages 36-40, March 2010

Article Tools

* Image Usage & Resolution