Regulations of the evaluation system, approved by the AGAUR’s Board of Management on the 30th of June of 2016.
The purpose of this document is to provide a clear overview of the development of evaluation-related activities of AGAUR and calls that it manages both as to competitive actions in research, evaluation and knowledge transfer and to improve the education system, which are mainly funded by the Government of Catalonia, as well as other institutions for supporting research in Catalonia and elsewhere.
In accordance with the general regulations of AGAUR, the general principles governing the evaluation activities are objectivity, independence and quality of evaluations within a fast and effective public-funded management framework that AGAUR would like to bring to all its actions.
In addition, AGAUR will ensure that any application does not infringe any ethical basis and, as appropriate, this could be a reason for exclusion from selection or evaluation process.
AGAUR also promotes to prevent any bias based on gender differences in the evaluation and selection activities, whenever possible. Among other aspects, AGAUR ensures that the selection committees observe gender parity.
The evaluation activities may be developed at different levels and times in order to assess whether public resources are being used appropriately to meet the goals established in the previous planning and in the specific calls.
The AGAUR’s calls include, through their evaluation and selection rules, the involvement of experts who evaluate the submitted applications and ensure that meet the requirements not to be excluded. The selection of these experts or evaluators is based on their recent scientific career coming from different science, university and industrial areas, with experience and high level in their area of expertise, well-known as international experts in their subject matter, and capable of understanding applications in Catalan language, if the call does not indicate otherwise.
The evaluators of the calls for research who usually work in the environment of the Catalan Public Universities are selected taking into account their number of research steps accrued. In this regard, researchers who have completed all possible research steps are considered first.
On the other hand, those who are not in public universities are selected primarily on the basis of minimum bibliometric criteria, established by AGAUR, which characterise each area of expertise (number of documents and citations).
Information and details of evaluators are in a database, in an electronic file maintained by AGAUR, which is updating with new members or those no longer involved. The database includes at least a full CV of each evaluator as well as information about his or her area of expertise and key words.
AGAUR may select as evaluator, if necessary, any individual with the necessary skills and abilities who is not previously included in the database described above.
Calls that require the participation of a large number of evaluators, the latter are selected from among all possible evaluators of the same area of expertise included in the database.
This selection will be made using a random selection process that at the same time optimally assigns applications to each evaluator. This process is based on a computerized procedure that ensures transparency, at all levels of awarding and balance and homogeneity in evaluation task, including gender equality whenever possible. In addition, the system ensures that evaluators are not over-burdened with work by assigning them no more than 10 files to each one. An evaluator from a particular institution never will evaluate research staff from the same institution. Assignment in successive evaluation processes will be governed by the principle of rotation, whenever possible.
If the random computerized procedure does not assign evaluators to all applications, they will be manually selected under the supervision of the evaluation advisory staff.
2.2 Conflict of interest and confidentiality
All proposed evaluators must complete a sheet of statements before evaluating stating that they have no conflict of interest with the applications to be evaluated and they undertake to keep confidentiality of information available to them for the evaluation process. If they do not sign this document, they will not have access to the applications assigned to them. By means of this document evaluators authorize AGAUR to periodically disclose their identity on an aggregate basis. This document shall remain valid as long as evaluators are working for AGAUR.
Evaluators are paid as provided for by the AGAUR’s Board of Management, according to the number of evaluated applications and its complexity.
The remuneration table is available at the AGAUR’s website.
Calls of the Government of Catalonia managed by AGAUR are published in the Official Gazette of the Government of Catalonia (DOGC).
Each call must specify the criteria and the mechanisms for evaluation and selection of applications and, where appropriate, the score or value given to each criterion in the evaluation as well as the minimum limits that must be reached.
Calls distinguish between the minimum requirements for submitting the application, which will be valued by the call’s managers, and the merits, which will be valued by the evaluators.
Any eventualities not provided for in the specific call will be governed by the existing general regulations of AGAUR, which are also published in the Official Gazette of the Government of Catalonia. The calls must specify, whenever possible, that applications should be submitted electronically, without excluding any another way to submit applications, in accordance with the common administrative procedure in force. If an application is received in both paper and electronic formats, only the electronic version will be processed.
AGAUR may specify in its calls, wherever appropriate, the possibility or the recommendation to submit applications in English language. The systematic introduction of the English language in the applications encourages more and more externalised evaluation process, involving international evaluators who will be selected with the aforementioned procedures and whose number will increase significantly in the coming period. This procedure is especially suitable for those calls that, in addition to scientific achievements, are aimed to the internationalization of the Catalan Science and Technology system.
All software applications and instructions addressed to evaluators will also be available in English language in order to facilitate their work.
The evaluation of applications is always carried out on the basis of fulfilment of the criteria and requirements set out in the call. The evaluation never begins before the deadline for submitting applications expires and before documentation is requested.
Firstly, AGAUR reviews information and format of the application. If there are no administrative defects found that lead to the application’s exclusion, the possibility of requesting complementary information is provided for, if necessary.
All applications are usually classified by section for being addressed to the corresponding evaluators. Each section has a coordinator who determines the thematic suitability in the selection of the evaluation team.
AGAUR is working to add key words in a standardized way as a means of classifying applications and for the assignment of the evaluation team, whenever possible. In this sense, an agreement with Termcat and the Catalan Universities to create a thesaurus of key words is being implemented. This new classification may replace or complement the classification by areas of expertise in the calls for research and will improve the assignment of the evaluation team.
5.1 Key figures
The different elements and people that are combined in the evaluation are as follows:
• Call. A set of rules or regulations that defines the procedure for granting an aid by competitive tendering.
• Candidates or applicants. Natural or legal persons who are involved in a call for scholarships and grants.
• Management Staff. Staff responsible for receiving applications from candidates and processing them adequately among the different elements of the selection process, in accordance with the specific call and the applicable regulations. The Staff in the Area of Evaluation and Research Analysis is responsible for processing and ensuring the evaluation process as the latter is defined in these regulations and for proposing and carrying out all actions required to improve it.
• Evaluation team. Independent experts selected on the basis of their research career, with no conflict of interest and bound by a commitment of confidentiality, and who assess the quality of candidates and applications.
• Section Coordinator. A renowned expert, appointed by the President of the Executive Committee of Research Grants (CEAR) for certain representative sections, such as Sciences, Life Sciences, Health and Medical Sciences, Social Sciences, Engineering and Architecture, Humanities, and Knowledge Transfer, who submit to the Selection Committee, together with the Management Staff, a consensus-based proposal of the evaluations made by the Evaluation team depending on budget availability in the calls. The main tasks and functions of this figure are as follows:
‐To review or monitor the specific evaluation process of each call.
‐To resolve or propose solutions in case of divergent evaluations.
‐To assess the evaluators.
‐To recommend improvements for the calls and the evaluation procedure.
‐To review the previous procedure if appeals are lodged.
The Section Coordinators are selected by the General Manager for Research from among a shortlist of three candidates proposed by the Executive Director of AGAUR, which ensures the scientific quality criteria and the understanding of the system necessary to carry out this task.The Section Coordinators are appointed for a period of two years, which can be extended up to a maximum of five years after which they are replaced by others with the same features.
• Selection Committee. A collegiate body, whose composition is defined in the call, that proposes the resolution on awarding scholarships and grants to the President of the corresponding executive grant commission, on the basis of evaluations, reports by the Section Coordinators, reports by the Management team, etc.
The AGAUR procedure, whenever possible, carries out at least two evaluations for each application, one of them done by an evaluator linked professionally to Catalonia, and another one done by an evaluator from outside the Catalan science and technology system. However, as a general rule, the Evaluation team is made up of two people who will assess each application. No member of the team may belong to the same institution as the application being evaluated. In the case of evaluations that are significantly divergent (25% of small difference), the corresponding Section Coordinator, on the basis of the considerations of the potential evaluation team made by the AGAUR Management Staff, may propose a third evaluation to resolve the previous doubts.
5.2 Scoring Scale
The final scoring system of the evaluations is based on the sum of the different values obtained for each of the blocks evaluated. The aggregate total is expressed with a value between 1 and 7 (and one-tenth), in which each value has the following meaning:
Deficient proposal in all aspects evaluated.
Unsatisfactory proposal, with a large number of significant deficiencies in the different criteria assessed that do not ensure its viability.
Average proposal, with reservations about some important elements or criteria of the assessment that may affect its viability.
Acceptable proposal, although with some reservation. It is not particularly outstanding in any of the aspects evaluated but it could be viable.
Good and viable proposal. It is right in most of criteria evaluated, competitive at national level but it does not contain significant contributions.
Very good proposal, competitive internationally and which makes significant contributions in its area. It is a national first-line proposal.
Excellent proposal at the international forefront that ensures a major impact on its area.
These basic definitions are further defined for each call, including the evaluation criteria defined.
The Evaluation team also indicates, apart from the numerical evaluation of the different blocks evaluated, qualitative remarks which must be consistent with the expressed score. The qualitative evaluation is compulsory and may be used to justify the scores obtained to the applicants, if appropriate.
AGAUR should provide, as applicable, the evaluation guidelines of the call in question, in order to define, for the Evaluation team, the object and the criteria of the call and to ensure a maximum coherence between the evaluators acting individually.
The Section Coordinators meet to achieve a consensus based on the notes of the evaluation team and budget availability, and, together with the Management Staff, to prepare a proposal to submit to the Selection Committee. The latter may change the order of the proposal.
The Selection Committee, taking into account the quality and suitability of the applications, and in accordance with the criteria explained in the call, will take a decision that includes the final resolution proposal, which will be forwarded to the President of the corresponding executive grant committee.
In those calls that so state specifically, and in accordance with the general regulations of AGAUR, the Selection Committee will determine the members of a prioritised stand-by list of rejected beneficiaries, who will become beneficiaries of the grant in the case of waiver or replacement of any of the original beneficiaries.
Research activities funded by AGAUR must meet ethical standards. Control systems will be progressively implemented in order to comply with European regulations of Decision 1982/20067EC, article n. 6.
AGAUR outsources, to organisations that have already reputation and expertise in the requirements of expert ethical control of the applications, and also agrees the decisions of the Animal Experimentation Committee (CEA).
No application involving any suspicion of non-fulfilment of any ethical standard will be funded.
By default, the calls for applications should follow the general scheme of the aforementioned evaluation and selection process. This scheme includes a number of features that will be applied to all applications, but the types of calls that are currently being defined by AGAUR may allow the evaluation process of each of them to have singularities in the procedure. The evaluation panels will be created when the calls so require.
On this basis, and always bearing in mind which procedure may be the best one for the evaluation of each call for applications, as well as the above-mentioned general criteria, the applications may be evaluated by means of different techniques such as:
Each application is evaluated by two experts who make their individual assessment. This technique is applied to most calls.
Panel of Evaluators:
Creating a panel of evaluators that evaluates the different applications, on a consensus basis or reviews the previous individual evaluation process already carried out, to support the Coordinators. The Panel of Evaluators work together in different sessions. This technique may be more applicable for those calls on specific topics or which address only one area, or those in which the CV of the research staff or the research group has a very relevant specific weight; also in those in which there is a very low number of applications or in highly competitive calls that require a double evaluation phase. The Panel of Evaluators may also complement a simple peer review in calls in which a comparative vision of all the applications is needed.
The AGAUR’s Management Staff will provide the evaluators with all the information they need to do their tasks when the evaluation process begins. The evaluation sheets are used to follow a generic structure which includes the evaluation criteria and the remarks. The criteria are specific for each line and the evaluator assigns a numerical value to each criterion evaluated. In addition, as previously mentioned, qualitative remarks should be made in each point.
Evaluations are made by means of a web-based software application based on certain personal connection parameters. Evaluators can work from any computer with Internet access. From a specific website and with a user name and password, evaluators can access the list of calls for applications assigned to them. Each call provides access to applications assigned to be evaluated. Evaluators can access full application and the evaluation sheet that has to be completed for each application, but only have access to the applications assigned to them.
The Section Coordinators can review all evaluations in their area. They may also have access the average scores given by each evaluator, in order to standardise the scores of that area if necessary, and to begin a process that must be introduced into the different phases of the call.
AGAUR carries out inspection activities on the different calls for applications in order to document and evaluate in itinere the development of the different projects or actions for correcting inadequate dynamics, if they are detected.
The assessment of results obtained individually in each action provides very significant information for learning how research is carried out and its impact. Thus ex post evaluation actions of the proposals and the funded projects that take different aspects of this process into account are also introduced:
- The suitability of the obtained results for the target specified in the call for applications.
- The scientific evaluation of results
- The development of the methodology required to assess the impact of the results at socioeconomic level in the Catalan context.
The specifities of these two areas of expertise and the demands of the groups that work in them and the different institutions may require the implementation of various actions to improve the evaluation in these areas and promote internationalization and impact, while also respecting the research tradition in the areas of Catalan studies. Particularly, the CARHUS Plus+ project (classification of journals in humanities and social sciences), which updates its data every two years, is also highlighted here as a tool to improve assessment processes of journals and the book and book chapter evaluation project, both of the them with a previous proven track and continuity.
The names of the AGAUR evaluators are published in an aggregate way on the AGAUR’s website every year, with the consent of such evaluators, as a purpose of transparency.
Similarly, the identity of the Section Coordinators, who are engaged for a two-year period, and their tasks, are also published systematically on the website.
The general evaluation of the evaluators who work for the AGAUR is carried out through mechanisms of reviewing and monitoring of their work, both from the technical and scientific area. This latest review also includes advice and evaluation from the Section Coordinators, both through the assessment of each evaluator in the software application as the general assessments that they may provide by different mechanisms created by AGAUR to achieve this goal.
Periodically, AGAUR may commission external evaluations of its evaluation processes in order to submit them to the criterion of experts and to improve and adapt them to the context of the Catalan science and technology system.
AGAUR, as a management and assessment agency, may establish contacts and cooperate with other state and international institutions engaged in research and university evaluation. This collaboration should be used to agree on mechanisms, procedures or criteria, if necessary, or to collaborate in questions with a view to improving the existing system of evaluation (eg, to share evaluators database).
Collaboration or coordination may take the form of the creation and participation in debate forums or signing agreements among institutions, etc.
Moreover, AGAUR will also allow this interaction with institutions from Catalonia that usually participate in the calls for applications and therefore in the evaluation processes.
AGAUR may meet the needs of institutions from inside and outside Catalonia in matters of research evaluation, on the basis of reimbursement for expenses incurred in this activity ensuring the quality of the evaluations, in accordance with the principles expressed herein.
Costs allocated to the evaluations will be calculated using the table in Annex 1 of this document.