Written by Sharon Brennan-Olsen
Are you currently developing your application to an MRFF funding opportunity? If so, now is a good time to think about how your application will be assessed.
Earlier this year (March 15th, 2023), MRFF delivered an informative webinar for future applicants ‘Assessing MRFF grants: Insights from assessors’.
In this blog we summarise the key points raised by the webinar panel, with a particular focus on what members of Grant Assessment Committees (GACs) look for during the assessment process.
The big question: when is an application determined as ‘not for further consideration’?
Importantly, a similar assessment process is used for all MRFF schemes regardless of their submission through NHMRC (Sapphire) or the business.gov hub. After the closing date for minimum data, conflicts of interest are managed, and an independent chair (chair does not participate in assessment or scoring) and panel members are assigned to each GAC. Notably, the number of members vary depending on the GAC; although MRFF aims to have four panel members on every GAC, there may be fewer. MRFF aims to ensure that the consumer voice is represented on each of the GACs. Within the panel, spokespeople are assigned to each application.
After applications have been determined as meeting the eligibility criteria, applications are then read a few times: the first read is focused on grantspersonship (clarity in argument, readability, formatting, the use of schematics to leverage the reader’s understanding) and the second reading is where the initial scoring of each application is individually performed by panel members. Those scores are collated, and a rank ordered list produced. A scoring cut off is decided by the relevant grant hub (NHMRC or business.gov). Applications above this cut off proceed to the grant assessment panel for discussion and those below are defined as ‘not for further consideration’ (NFFC).
For applications that progress to the GAC for discussion, the spokespeople speak about their assigned applications, and then open it up for panel discussion. It is the role of the independent chair to ensure that i) the same amount of discussion time is dedicated to each application and ii) discussions are not dominated by any one panel member. Panel members then determine the final score for applications, essentially using the scoring matrix as a rubric.
Project impact: alignment with Measures of Success
When assessing project impact, the other documents that contribute to this are the i) objectives and intended outcomes specified in the guidelines, ii) the MRFF initiative or Missions’ Roadmap and Implementation Plan, and iii) your project’s statement against the MRFF Measures of Success.
Notably, the third/final column of your Measures of Success statement indicates the project’s intended outcomes, so ensure that you discuss the impact that will result from those outcomes.
Much attention was paid to the meaningful engagement of consumers/end-users throughout all stages of the project, with the suggestion of embedding a consumer/end-user map. As with any project, it is very clear to assessors when meaningful collaboration has not been undertaken and where consumers/end-users have not been involved in the conceptualisation of the research idea, prioritising the issues and defining the questions, study design and methodology. The underlying argument is that project impact will be substantially leveraged by meaningful engagement of consumers/end-users across the life of the project: who better to inform what is needed and the best way to do it than those most affected?
Project methods: what scores well?
When assessing methods, the other documents that contribute to the feasibility and sound methodology include the timeline, milestones and the team that has been assembled. The webinar panellists discussed what constitutes a sound methodology, with the key points being:
Alignment with the Measures of Success statement:
The middle column of your Measures of Success statement describes how your project will produce the outcomes.
Clearly articulated research questions and aims:
The project impact section should have set you up to determine your aims.
Ensure that your aims are as specific as possible, and clearly addressable by your proposed study.
Richly detailed methods with a flawless plan:
Your research questions should drive the choice of methods.
Ensure the rationale for your choice of methods is clear.
Detail every step of the research and avoid any vagueness around the methodology that may attract questions about the feasibility.
Share your methods section with your colleagues to see if they understand what you are proposing and how you’ll do it.
Present a coherent package, in terms of scope, design and expertise.
Write succinctly and ensure each element of your methods fit together strongly.
Ensure it is clear how your chosen methods enable you to address the aims.
Use a diagram, if relevant, to illustrate proposed methods.
Being too ambitious may attract a lower score for feasibility.
Assemble a truly multidisciplinary team to do the work:
Clear thought has gone into who is on team, including their expertise and complementarity with other team members.
Clearly identify in the methods who will be undertaking the various aspects of the research, and if junior researchers will do the work, how will they be supported by more senior researchers.
Early engagement with consumers/end-users or policy makers:
It is very obvious when these are a last-minute addition to the proposal.
Ensure they are meaningfully embedded across all stages of your research.
The research journey begins before proposals are developed, so avoid indicating that you ‘will’ engage consumers: consumers should have informed the proposal and not be engaged only once funding is allocated.
Ensure your methods align with best practice principles in consumer engagement.
Capacity, capability and resources
Leadership of the team:
Who is the team and who is CIA?
Is the named CIA the best one to lead the team and why?
Demonstrate what strong leadership means for your project.
Capacity building opportunities:
What are the capacity building opportunities (i.e. is there a postdoc built into grant, etc)?
Who will be mentored, by who and how?
What will be the mentee’s responsibilities in the project and how closely will they work with senior researchers and the CIA?
Is there mentoring of juniors, partner organisations or consumer-researchers?
How much time do the named CIs and other senior researchers dedicate to genuine capacity building?
FTE and alignment with methods:
Has there been enough FTE allocated for CIs to work on this project, including time to upskill and mentor junior researchers?
Is the upskilling, mentoring and capacity building feasible within the skillset?
Do the CIs have sufficient time given their other commitments and responsibilities?
Strengths of the combined team:
Everyone must have clearly defined roles and bring the necessary capacity.
Is there a previous collaborative track record?
To leverage your project’s feasibly, have you included those with specific expertise such as health economics or biostatisticians?
Involvement of consumers/end-users/community in research team:
The research is a need and priority identified by community groups and stakeholders.
How are they involved? (think broader than just advisory committee meetings)
Is there a shared understanding of genuine, clearly defined leadership roles?
Are those with lived experience participating as researchers?
Is there a role for consumers/end-users in the project’s governance?
Relative to opportunity and career disruptions:
Don’t disadvantage your application by not claiming them.
Partnership with organisations:
Meaningful partnerships should be clear in each section of your application
Articulate where the relationships are long-standing to demonstrate commitment.
Overall value and risk: the ‘what if’ scenarios
According to the webinar panellists, the assessment of this section involves taking a broad view look to determine the ways that applicants propose to manage outcomes, risks and the budget in relation to the overall project.
Ensure that you write to the overall value and risk category descriptors and provide contingency plans where necessary that will kick in if things go wrong. A panellist on the MRFF webinar stated that discussions with assessment panels are often ‘…OK, great project, great initiative, great recruitment strategy. But look, they haven't really thought about if something goes wrong.’ That suggests that not adequately discussing your potential risks in this non-weighted section could reduce the overall score of your otherwise well-argued application.
An entwined budget
Demonstrating value for money involves strongly justifying your budget. For example, if you don’t need the full budget and the full timeline that are available to you, then don’t stretch out your project. A more contained project over a shorter period will also likely require a smaller CI team. The alignment of a smaller budget with appropriate methodology still speaks loudly to feasibility and to the overall value and risk assessment. Another example is, if your project budget is 90% salary costs and 10% project costs, ensure that your justification clearly articulates why the salary costs are essential to the research. Also consider if your budget is:
enough to support the successful delivery of your aims
sufficiently detailed and justified
representing value for money to the MRFF
including appropriate costs to enable meaningful consumer engagement
Embed meaningful involvement of consumers/end-users throughout the research journey.
The lack of meaningful involvement of consumers throughout applications is very noticeable to assessors. Think about the phrase, which commonly employed in the inclusive research domain: ‘nothing about us without us’. Consumers and consumer organisations can give strength and added value to your project application. Assessors will look for easily understood plain language applications (indicative of consumer involvement in writing applications), strong consumer engagement during conceptualisation that informed and prioritised the research questions and co-designed the overall project, and consumer/end-user involvement in governance.
Demonstrating consumer engagement before and during conceptualisation of your research project could be coupled with results from targeted consumer surveys or results from focus groups that demonstrate the importance of consumers/end-users on the research focus. Alternatively, it could be informed by meaningful roles for consumers as researchers on the project. That input could also support an argument about cost benefit and risk, for instance, identifying where novel research involves a considered risk, determined by consumers as risk at an acceptable level given the potential social, economic, or community outcomes.
What is the role of consumers on GACs?
Consumers on GACs bring unique perspectives to the consideration of applications in a way that other assessors may not, and may therefore provide exclusive insight to assessment panels as well as inform value for money. For instance, consumer input may be focused on the impact of data collection on participants, appropriate engagement during the early prioritisation of questions, potential (non-clinical) side effects in relation to clinical trials, useability or accessibility of written interventions for population subgroups, among other aspects.
It is important to remember that we are all potential consumers of MRFF-funded research. Consumers on GACs may include retired professionals or semi-professionals, individuals from disadvantaged backgrounds, representatives of different cultures, working individuals, and/or those with lived experience of the condition of interest, amongst other inextricably entwined perspectives.
General points
Grantspersonship. It is critical to present a polished application to avoid risking your science due to a poorly constructed argument. Enlist your colleagues, particularly those who don’t know your work or work in your field, to review and comment constructively (and honestly) on your application. Keep in mind that panel members may not be experts in your area, so write for a general audience and avoid jargon.
Think about your reviewer. Assessors may likely review grants at the end of the day, potentially when they’re tired. Try to ‘…take the reader by the hand and lead them through your grant. Lead them through your story.’
Write to your category descriptors. During the MRFF webinar, it was suggested that applicants have the category descriptors in front of them as they write the proposal. This is because the independent chair is going to bring the panellists and assessors back to those category descriptors during the panel discussion. Self-assess where you are on the assessment matrix; the GAC will focus on the matrix to ensure meaningful discussions around individual projects.
Be involved in an MRFF GAC panel. By contributing to MRFF via a GAC you will attain a greater understanding of the assessment and discussion process including ‘what scores well’ and ‘what to avoid’, thereby leverage your capacity to self-determine if your own applications align with the category 7 descriptors.
What’s the take home message from the webinar?
Although not all questions raised by the webinar panel are relevant to each application, it’s obvious that assessors value applications that are richly detailed in easily accessible language and explain the ‘why’ and the ‘how’ of the research. Assessors use non-weighted sections during the assessment process, and so will be looking for strong alignment between sections. At GrantEd, we strongly encourage you to consider these summarised points from the webinar panel in conjunction with the scheme-specific guidelines. And finally, if you’re interested in watching the MRFF assessment webinar, it can be found here Videos and webinars | Australian Government Department of Health and Aged Care.
Good luck with your MRFF funding applications!
Comments