The purpose of this document is to provide a clear overview of the development of the AGAUR-associated evaluation activities and the calls it manages, both in terms of the competitive actions in research, knowledge transfer and valuation, as well as to improve the university system, which are primarily financed by the Government of Catalonia, as well as other institutions that promote research in Catalonia and around the world.
The evaluation activities carried out by AGAUR mainly relate to managing processes that enable an evaluation, based on public criteria, to be conducted by expert and independent persons. AGAUR staff does not directly evaluate the proposals, but manages the entire evaluation procedure.
Regulations of the evaluation system, approved by the AGAUR’s Board of Management on the 28th of January of 2020.
1. Principles
Within a framework of rapid and effective management of public funds that AGAUR wants to implement in all the actions it carries out; the general principles governing the evaluation activities are:
Transparency. The evaluation process is based on clearly defined rules and procedures that are published in advance of the corresponding grant rules. The outcome of the evaluation of the proposals will be sent to the interested parties.
Excellence. We are committed to a range of actions that will allow for permanent improvement in the management of evaluation processes.
Impartiality. The evaluation process is based exclusively on technical aspects and scientific rigour, regardless of the origin or identity of the applicants.
Proportionality. The evaluation processes have to be adjusted in terms of proportion and investment, according to the nature of the call being evaluated.
Confidentiality. Strict confidentiality must be maintained regarding any information related to the proposals.
2. Standar evaluation procedure
2.1 Main figures
-
Candidates or applicants. Physical or legal persons that participate in a call for scholarships or grants.
-
Experienced and external persons who evaluate the quality of the proposals. They have been selected because of their career and expertise, without conflict of interest and under a commitment to confidentiality.
-
Section coordinators. Experts with acknowledged prestige who review and monitor the evaluation processes for each call.
-
Management staff. AGAUR staff responsible for managing the proposals and processing them properly throughout the different stages of the selection process, in accordance with the grant rules, the specific call and the application regulations. The management staff of the AGAUR Evaluation Area are responsible for processing and ensuring the proper functioning of the evaluation process as defined in these regulations, as well as proposing and carrying out all the relevant actions needed to improve it.
2.2 Evaluators
The evaluators are people external to AGAUR and of recognised international prestige in their fields, who guarantee specialised and general technical knowledge of the fields involved in the proposals.
These evaluators are selected based on their recent scientific career. They come from different scientific, university, industrial, etc. fields, with experience in the subjects and fields to be evaluated and recognised as specialists in their subject matter.
The evaluators must have the appropriate knowledge, high level of professional experience in scientific and technical research and the necessary language skills. Other skills, such as technology transfer and innovation, management or guidance may also be requested, if applicable.
The team of evaluators seeks to be diversely represented with regard to gender, geographical origin, type of institution, age distribution, etc.
The information and details of the entire group of evaluators are in a database owned by AGAUR, whose purpose is to manage them to carry out the evaluations of the proposals presented within the framework of a call. This database is updated with new additions or possible deregistrations. The database contains, as a minimum, each person´s full CV and the information on the field or scientific area they are specialised in, as well as the keywords that they have provided.
AGAUR may, as the case may be, select anyone, with the necessary skills and qualifications that has not been previously included in the aforementioned database, to be subsequently included.
2.3 Section Coordinators
Experts with acknowledged prestige who review and monitor the evaluation processes for each call.
The field coordinators are appointed by the director general of Research from an AGAUR proposal that guarantees the criteria of scientific quality and knowledge of the system necessary to carry out this task. The Section Coordinator candidates must have taken part in evaluation processes for an international body.
The coordinators shall perform this function for a period of two years, which may be extended to a maximum of five years.
The main tasks and functions of this role are to:
-
Review and monitor the specific assessment process for each call.
-
Resolve or propose solutions in the event of divergent evaluations.
-
Assess the evaluators.
-
Recommend improvements for the calls and evaluation processes.
-
Review the previous procedure if appeals are lodged.
The Section Coordinators can review all the evaluations made in their area. They can also set forth the average score awarded by each evaluator.
2.4 Organisation of the evaluation process
Evaluating the proposals is always carried out in accordance with the criteria and requirements expressed in the grant rules and the corresponding call, and never gets underway before reviewing, amending or improving, if applicable, the documentation required to access the evaluation phase.
All applications that go through the evaluation phase have been reviewed and meet the eligibility criteria set out in the calls.
All proposals, depending on the fields of knowledge stated in their application, will be included in a specific area so that they can be provided to the corresponding evaluators.
The list of standard fields and areas of knowledge linked to each field is available and can be consulted on AGAUR´s website.
AGAUR may adjust the evaluation process to different distributions of fields and fields of knowledge if so required by the calls. The fields of application can also be requested for the evaluation of transfer and valuation proposals and taken into account when assigning and making the corresponding classifications.
AGAUR works on introducing keywords in a standardised manner as a mechanism for scoring proposals and assigning the evaluation team, where possible.
A minimum of two evaluations must be made for each proposal. In the case of divergent evaluations (difference of 15% or more between the two final scores) a third evaluation may be carried out.
2.5 Deadline for submitting applications
Evaluators are selected from all those stored in the database, based on their area of knowledge or expertise. The experience in evaluation and transfer processes and the fields in which the proposals are applied will also be taken into account in the calls that so require it.
This selection is made through a random selection process that, at the same time, optimally assigns the files to each evaluator. This assignment process is based on a computer procedure that guarantees transparency at all levels in the assignment, balance and homogeneity in the evaluation task, and also includes diverse representativeness.
Furthermore, the system takes into account the following aspects:
-
Not to overload people by assigning a maximum of 10 files to each of them, while ensuring that they have a sufficient number of files so that they can evaluate more objectively and comparatively.
-
Assignment in successive evaluation processes must be governed by a rotation principle.
In the assignment, an evaluator from a particular institution shall never evaluate the research staff from the same institution.
If the random computer procedure does not assign evaluators to all the files, or if few evaluators are required, they can be chosen according to the criteria of independence and expertise, and under the supervision of coordinating staff.
AGAUR´s management staff must provide the evaluators with all the complementary and necessary information to carry out their tasks, at the beginning of the evaluation process.
The evaluations are made by means of an internet computer application that meets the security requirements and the processing of the corresponding statements. The evaluator can access the complete file and the evaluation form to be filled in for each file, but shall only have access to the assigned files.
2.6 Conflict of interests and confidentiality
All proposed evaluators have been previously reviewed to confirm that they are not part of any submitted application. Evaluators need to sign a statement sheet prior to undertaking their tasks, stating that, in the event of any conflicts of interest, they shall notify AGAUR and refrain from evaluating that proposal, and undertake to maintaining the confidentiality of all the information available to them during the evaluation process. If they do not sign this document, they will not be allowed access to the assigned files. Through this statement, the evaluators authorise AGAUR to periodically publish their identity in aggregate way. This statement sheet is always valid while working for AGAUR.
Applicants may request to exclude evaluators in connection with the evaluation of their proposals.
2.7 Evaluation criteria
The evaluation criteria must be made public and drawn up, beforehand, in the grant rules of the calls or official documents that regulate the awarding of grants from each institution taking part in a call. These documents must also specify the scores or values for each criterion.
These criteria are distinguished from the minimum requirements required for the submission of the application (eligibility criteria), which will be reviewed by the grant management staff. Evaluation staff only values based on the evaluation criteria.
The evaluation criteria and the weight or value of each are reproduced in an evaluation form to be filled out by the evaluators.
Specific assessment guides may be developed to facilitate the understanding of the purpose of the grants and to achieve the maximum consistency when applying the criteria by the individual evaluators.
2.8 Remote evaluation and evaluation panels
The general outline of the AGAUR evaluation procedure shall be as set forth in this Regulation, which establishes the general characteristics that apply. However, in each call for grants or grant rules, some specifics may be specified that require a longer or shorter process, depending on the phases that are applied. This is:
-
A single phase of remote evaluation of applications
-
Remote evaluation of each application and evaluation panels
-
Only evaluation panels
Remote evaluation of applications
Each application will be evaluated by a minimum of two experts, who will each make their own evaluation. This modality is the one that is used in the majority of calls.
These experts remotely evaluate the files that have been assigned to them. They do this individually, separately from other experts who may evaluate the same proposals. They do not discuss the proposals collectively, and provide their evaluations to the AGAUR evaluation area using the criteria and scores described in the call. The result of each evaluation is a form that is collected in the evaluation application, with a quantitative and qualitative assessment of the criteria and a final score, which is an internal document.
Evaluator panel:
Creation of a panel of evaluators to evaluate the different proposals, on the basis of consensus, as a support for the work of the coordinators.
The evaluation panel works on different sessions together. This is a technique that may be more applicable in calls for specific subjects, calls for multidisciplinary projects, valuation and transfer or those in which the curriculum of the research staff or the research group has a very specific weight. Also in those with a very low number of applications or in very competitive calls that require a double evaluation phase. The evaluation panel can also carry out a simple peer review in calls where a comparative view of all applications is deemed appropriate.
When the call establishes a dual modality (remote and panel), it must specify how the final score is established, which will usually be agreed by the evaluation panel.
2.9 Scoring scale
The final scoring system of the evaluations is based on the sum of the different values obtained for each of the blocks evaluated.
The aggregate total is expressed with a value between 0 and 7 (and one-tenth), in which each value has the following meaning:
0 |
Proposal not evaluable, does not meet requirements or does not meet the criteria |
1 |
Deficient proposal in all aspects evaluated |
2 |
Unsatisfactory proposal, with a large number of deficiencies in the different criteria evaluated, that do not guarantee its viability |
3 |
Average proposal, with reservations regarding some important elements or criteria of the valuation and which may affect its viability |
4 |
Acceptable proposal, albeit with some reservation. It does not particularly stand out in any of the aspects evaluated but could be viable |
5 |
Good and viable proposal. It is right in most of criteria evaluated, competitive at national level but it does not contain significant contributions |
6 |
Very good proposal, competitive internationally and which makes significant contributions in its field. It is a national first-line proposal |
7 |
Excellent proposal at the international forefront that ensures a major impact on its field |
These definitions are specified for each call integrating the evaluation criteria that it defines.
Other scales may be used where appropriate. The calls must specify the scale of the adopted score and define the values.
The evaluation team also fills in, apart from a numerical assessment of the different blocks evaluated, qualitative comments that must be consistent with the score awarded. Qualitative evaluation is mandatory and can be used to justify the scores to interested parties, where appropriate.
Field coordinators review the assessments and the evaluation process performed, and resolve or propose solutions to possible incidents in the process; finally they assign the final score for each file.
2.10 Evaluation results
Once the evaluation process is complete, the result of the scientific evaluation is incorporated into a report drawn up by AGAUR from the reports of the independent experts, which can be sent to the email provided by the applicants. The support management teams may individually notify the final score of each file beforehand.
The specific comments and scores of each evaluator who participated in the individual evaluation phase need not necessarily be convergent. The peer review implies that the reviewers are experts, but may have had careers in very different fields. They can emphasise various aspects, highlight some aspects to a greater or lesser extent, and make proposals for improvement according to their expert opinion. This does not necessarily mean that there are contradictions that need to be corrected and that the procedure cannot be deemed to be incorrect.
2.11 Procedure for reviewing the evaluation
Applicants may request review of the evaluation if they deem that the relevant evaluation procedures have not been properly applied to their proposals.
Reviewing the evaluation concerns only the procedural aspects of the evaluation, and not the merit evaluations made by expert evaluators, provided that an explicit error in the treatment of the evaluated information is not detected. Reviewing the evaluation will not lead to a delay in the selection process for proposals that are not being reviewed.
General information on the evaluation processes that have been in development for a year are published on AGAUR´s website at the beginning of the following year.
2.12 General overview of the evaluation process
3. Remunerations
The activity of the evaluators and coordinators is remunerated according to the rates approved by AGAUR´s Board of Directors.
The rate table is published on AGAUR´s website.
When an evaluation process is required which, due to its characteristics and complexity, implies a greater volume than usual, the rate may be reviewed, in accordance with the task performed, and if proposed by AGAUR management.
4. Language
Catalan is the institution's normal language of use.
AGAUR may establish in the calls that the possibility or recommendation of presenting the proposal in English may also be appropriate. The systematic introduction of the English language into applications will favour a more outsourced evaluation process, with the participation of international evaluators who will be selected using the aforementioned procedures. This procedure is especially suitable for calls that, in addition to scientific achievements, are also aimed at the internationalisation of the science and technology system in Catalonia.
All computer applications and instructions for evaluators will also be available in English to facilitate their work.
5. Ethical control
For the type of financial support that so requires it, compliance with ethical standards must be ensured, in accordance with established regulations.
AGAUR outsources to agencies that have already developed expertise in these requirements of research, ethical expert control of proposals and decisions of the Animal Experimentation Commission (CEA).
6. Disclosure evaluators
The names of the AGAUR evaluators must be published in aggregate way on AGAUR´s website annually, with the prior consent of the latter, as a measure of transparency. With regard to the evaluation process of applications with a high number of applications, the names of the evaluators of this call must be published in an aggregate way.
The names of the evaluators assigned to each individual proposal should not be made public. The evaluation is confidential and anonymous, and the name of the person who evaluated them should never be passed on to the applicants.
Furthermore, the identity of the coordinators, who have been in the business for a period of two to five years, and their tasks must also be published systematically on the web.
7. Meta-evaluation and inspection
The general evaluation of the evaluators acting on behalf of AGAUR is carried out through mechanisms of reviewing and monitoring their technical and scientific work. This latest review also includes advice and evaluation from the Section Coordinators, both through the evaluation of each evaluator in the software application as the general evaluations that they may provide by different mechanisms created by AGAUR to achieve this goal.
From time to time, AGAUR may commission external evaluations of its evaluation procedures, or submit them to tender programs for evaluation, in order to improve the evaluation process and adapt it to international standards.
Furthermore, each year internal and external audits are carried out by an independent certification body, throughout the evaluation and selection system of evaluators, in accordance with the requirements of the ISO 9001 standard, for a quality management system.
8. Collaboration with other institutions
AGAUR, as a management and evaluation agency, may establish contacts and proposals for collaboration with other institutions in university evaluation, research and innovation, both in Catalonia and abroad. This collaboration should be used to agree mechanisms, procedures or criteria, as appropriate, or to collaborate on matters that allow the existing evaluation system to be improved.
Collaboration or coordination can entail the creation and participation in debate spaces, the establishment of agreements between institutions, etc.
9. External commissions
AGAUR may meet the needs of institutions, inside and outside Catalonia, in the field of research evaluation, teaching quality and innovation. External commissions must cover the costs incurred by the evaluated process. They will have to follow the standard AGAUR procedure and, in the event that an extra action is ordered or that the evaluation produces difficulties that were not previously anticipated; this must be foreseen in the budget and agreed with AGAUR´s Executive Director.
10. Gender
AGAUR is committed to preventing any gender bias in evaluation processes.
The database will be periodically reviewed to include an increasing number of women evaluators. Furthermore, in the evaluation processes, AGAUR seeks to achieve a balanced participation and aims for this never to be less than 40%.
The review of possible measures, criteria or actions that favour gender equality in the field of scientific evaluation is constantly being worked on, with the participation of AGAUR in the Women and Science Commission of the Inter-University Council of Catalonia, among others.