Selection and monitoring for ERDF and ESF projects in the 2014-2020 period are still mainly outputs-oriented

Executive summary

I

For the 2014-2020 programming period, the EU has allocated almost €349.4 billion towards achieving its objectives in terms of cohesion policy. This policy aims to support job creation, business competitiveness, economic growth, sustainable development and improvement in quality of life. To achieve these objectives, funding is allocated to projects delivered by beneficiaries in Member States. Authorities in Member States select projects to fund and monitor their implementation. It is critical that this funding is allocated effectively, namely through the delivery of expected results.

II

In order to support this goal, the Commission implemented various measures aimed at increasing focus on results in the 2014-2020 period. In previous reports, we have welcomed the improvements that these have brought. But we have also highlighted serious weaknesses in the effectiveness of these measures, and the quality of the monitoring information relating to spending on Cohesion Policy. This report consolidates and develops these previous findings.

III

We examined how well the focus on results is embedded within the way projects are selected, and how well the Commission and Member States can demonstrate the EU budget is well spent. As such, we have examined the selection and monitoring arrangements of projects using Cohesion funding.

IV

We conclude that, despite longstanding intentions, the design of selection procedures and the processes themselves continue to emphasise outputs and absorption rather than results. Moreover, shortcomings in the monitoring arrangements made it difficult to assess the extent to which EU funding has contributed to the achievement of the EU’s and Member States’ objectives. Specifically, for the OPs we visited, we made the following findings:

• On project selection, potential beneficiaries were well informed and provided with appropriate support to access to EU funding. The procedures we examined were designed to support the selection of projects relevant to the OPs’ objectives, but selection criteria seldom required applicants to define quantified result indicators at project level. Furthermore, most project selection was done on a first-come first-served basis. Applications were only scored and ranked against each other in one selection procedure.
• On monitoring, we found that monitoring systems had only become functional at a late stage, mainly because of delays in adopting the legislative framework. There remain weaknesses in some of the IT systems used for collecting and aggregating monitoring data. Furthermore, slow progress in the audits of the respective monitoring systems introduced the risk that there now may not be enough time to take any necessary corrective action before the performance review in 2019.
• Monitoring information remains mostly outputs-oriented. The Commission’s main report to measure achievements presents progress for the main output indicators alongside the implementation of funding. However, there is only limited information on the achievement of results.

V

We therefore recommend the following:

• In order to ensure a consistent and genuine results-oriented approach to the selection of projects, Member States should ensure that a comparison between project applications takes place, require beneficiaries to define at least one genuine result indicator for each project and carry out an assessment of the expected results and indicators in the assessment report for the applications.
• In order to ensure a results-oriented monitoring, Member States should include one or more genuine and quantified result indicators in the grant agreement, which contribute to the result indicators set at OP level, and the Commission should define common result indicators for the ERDF based on a common definition of “results”.
• The Commission should improve the reporting on performance and ensure that a meaningful performance review takes place in 2019.

Introduction

Cohesion policy is the EU’s main investment policy

01

Cohesion policy is the EU’s main investment policy. It is aimed at supporting job creation, business competitiveness, economic growth, sustainable development and improvements in quality of life. Around one-third of the EU’s budget is allocated to cohesion policy. At current prices, this amounted to around €230 billion in the 2000-2006 programme period, €346.5 billion in the 2007-2013 period and €349.4 billion in the 2014-2020 period.

02

Cohesion policy is delivered through the European Regional Development Fund (ERDF), the European Social Fund (ESF) and the Cohesion Fund (CF). These funds, together with the European Agricultural Fund for Rural Development (EAFRD) and the European Maritime and Fisheries Fund (EMFF), are part of the European Structural and Investment Funds (ESIF). They are implemented through operational programmes (OPs) which set out investment priorities and specific objectives¹. These also describe how funding will be used during the programme period to finance projects. The OPs are implemented by the Member States and their regions. This means that the Member States and their regions select, monitor and evaluate projects.

Performance orientation is a key priority for the Commission and the Member States during the 2014-2020 period

03

The Europe 2020 Strategy is the EU’s ten-year jobs and growth strategy. It was launched in 2010 to create conditions conducive to smart, sustainable and inclusive growth. With a view to meeting the objectives and targets set in the Europe 2020 Strategy, the Commission emphasised in 2010 the need to improve the effectiveness of the cohesion policy by focusing on results².

04

The legislative package for cohesion policy in the 2014-2020 programme period therefore introduced significant changes intended to increase focus on performance³. The main measures are the following:

• The introduction of ex-ante conditionalities (EACs)⁴ which require a Member State to fulfil certain conditions for the effective use of ESI funds (e.g. existence of policy/strategic frameworks) by the end of 2016. One of them, general ex-ante conditionality No 7⁵, requires “the existence of a system of result indicators necessary to select actions which most effectively contribute to desired results, to monitor progress towards results”.
• The introduction of a performance reserve, setting aside part of the EU funding allocated to the OPs, to be released only if a subset of indicators (mainly spending/output indicators) reach pre-defined milestones. The release of this reserve in 2019 will be based on a performance review carried out in 2019⁶.
• A reinforced intervention logic during the programming exercise⁷ requiring Member States to assess their needs and commit to a set of results before considering where and how to spend cohesion funds.
• A more consistent use of performance indicators to measure progress towards achieving results⁸.

The importance of project selection and monitoring to achieve performance

05

In 2017, we concluded⁹ that the Commission and Member States had been successful in developing OPs with a more robust intervention logic. This means that the OPs set out in a consistent way the aims of the interventions (specific objectives/results) and how these will be achieved (required funding, actions to be undertaken and expected outputs).

06

However, to achieve policy effectiveness, it is not only important to set clear goals and consider results during the programming exercise. It is essential that a results-oriented approach is followed through during the implementation phase of the OP, as it is mainly the quality of projects funded and their added value in terms of tangible results which will determine the effectiveness of a policy. Project selection, monitoring and reporting is thus essential for achieving performance. An overview of the implementation cycle of an OP and the bodies concerned is provided in Figure 1. The processes relevant to our audit are highlighted in yellow.

Main participants and their role in project selection and monitoring

07

Projects are selected by the national and regional authorities responsible for managing the OPs. These managing authorities (MAs), or intermediate bodies (IBs) to which they delegate powers, lay down selection criteria, organise selection committees and decide which projects will receive European funding. The Commission is not involved in the selection of projects as opposed to its crucial role in the negotiation of OPs. However, the Commission is involved through its advisory role in the OPs monitoring committees¹⁰ where the methodology and criteria used for selecting projects are approved. It also provides guidance, based on best practices and shared experience with Member States.

08

The responsibility for monitoring projects lies with the Member States. Managing authorities monitor the implementation of OPs and projects, aggregate monitoring information and submit Annual Implementation Reports (AIRs) to the Commission. The monitoring committee also reviews the implementation of the OPs and approves AIRs. Here also, the Commission’s role is limited to issuing guidance, examining the performance of OPs together with the MAs, and making observations on AIRs.

09

In its reporting role nevertheless, the Commission submits its annual synthesis of Member States’ AIRs and strategic reports (in 2017 and 2019) to the discharge authority (European Parliament), and to the European Economic and Social Committee and the Committee of the Regions¹¹.

Audit scope and approach

10

This audit sought to answer the question of whether the selection of projects and monitoring systems for ERDF and ESF projects in the 2014-2020 programme period were results-oriented. Given the level of advancement of the OPs’ implementation, and building on the audits already carried out, this audit allowed us for the first time to follow the “results-oriented approach” through the whole implementation phase as the first results are becoming available. It is also of high relevance to the performance review scheduled for 2019.

11

In the first part, we report on our analysis of selection procedures in place for the audited OPs. We assess the design of these selection procedures, especially the determination of the selection criteria and their result orientation, as well as the phase of evaluation of the applications. In the second part, we examine the management and control systems in place and the monitoring that has been carried out so far by the MAs/IBs of the visited OPs to measure operational performance at programme and national level. We review the reliability of data, its availability and capacity to produce relevant information for management and for reporting to the Commission.

12

In total, we examined 34 projects (see list of projects in Annex I). We focused on the ERDF and ESF, and more specifically on three thematic objectives (TOs)¹² (see a description of the 11 thematic objectives in Annex II):

• TO3, enhancing the competitiveness of SMEs (16 projects),
• TO8, promoting sustainable and quality employment and supporting labour mobility (12 projects), and
• TO9, promoting social inclusion and combating poverty and discrimination (6 projects).

13

We looked at projects selected directly by the MA (or intermediary body) through 20 selection procedures (see list in Annex III), from seven OPs located in four Member States: the Czech Republic, France, Italy and Finland (see list in Annex IV). These OPs and Member States were selected according to the materiality of funding allocated and the stage of the projects’ implementation. We selected the projects based on the amount of grant received and their level of advancement at the time of the audit. The priority axis, investment priorities and specific objectives to which the selected projects correspond are listed in Annex V. The related output and result indicators are listed in Annex VI for the ERDF and Annex VII for the ESF.

14

We also reviewed the actions taken by the Commission in relation to project selection and monitoring of operational programmes for the ESF and the ERDF in general.

Observations

Project selection is insufficiently focused on results

15

The selection procedures put in place by the managing authorities should prioritise the selection of those projects that are the most relevant to the objectives pursued and which have the greatest likelihood of delivering results, as well as ensuring that available funding is spent in line with the rules. We checked the extent to which results had been reflected in the design of selection procedures (e.g. the selection criteria used for assessing an application, the method used for assessing projects and the documentation available to potential bidders) as well as in the implementation of the selection procedures and thus in the projects selected.

16

Previous ECA audit work¹³ showed that the Commission and Member States had been successful in developing OPs with a more robust intervention logic i.e. with a clear link between development needs, specific objectives and result indicators. This intervention logic should be reflected in project selection procedures. We consider that there are a number of key criteria that should be taken into account by MAs when designing procedures to ensure that they are results-oriented:

• the selection criteria applied should ensure that projects are aligned with OP objectives;
• the selection criteria should support the selection of projects that have not only quantified output indicators but also quantified result indicators to measure projects achievements;
• beneficiaries should be encouraged to submit applications to ensure that a high number of projects are available for selection;
• finally, the project selection should be based on a direct comparison of applications to identify and prioritise the projects best suited for funding.

17

The extent to which MAs applied these variables while processing the applications received and how they influenced the selection of results-oriented projects is described in the next paragraphs (see paragraphs 18 to 38).

The selection procedures support the selection of projects relevant to OPs’ objectives

18

Two measures were introduced for the 2014-2020 programme period legislative framework which are designed to trigger the selection of projects relevant to OPs’ objectives:

• OPs must set out “guiding principles” for the selection of projects for each priority axis¹⁴;
• MAs must design selection procedures and criteria “that ensure the contribution of operations to the achievement of the specific objectives and results of the relevant priority”¹⁵.

19

In addition to these, the following requirements were also introduced for the 2014-2020 period to trigger the selection of more results-oriented projects:

• under general ex-ante conditionality 7 (see also paragraph 4), “a system of result indicators necessary to select actions which most effectively contribute to desired results” must be in place;
• MAs must also draw up selection procedures and criteria that “ensure the contribution of operations to the achievement of the […] results of the relevant priority”¹⁶.

20

The “guiding principles” act as guidance for project selection, detailing the main selection criteria to be used. Coupled with the description of the projects to be allocated funding, providing examples of such projects, and beneficiaries, they give a good overview of the type of projects to be funded under each specific objective. These principles are used as a basis for preparing the selection criteria, and may be referred to directly within selection criteria.

21

We found that, in some cases, the principles were of a general nature; however, they all referenced the need for selected projects to be in line with the specific objectives of the OP, as described under each priority axis. The selection criteria, for the selection procedures we examined, were consistent with the guiding principles. Each procedure entailed at least one criterion about projects’ relevance and their contribution to the specific objective.

22

Overall, the selected applications we examined were assessed against the criteria defined within the selection procedure. We found that the 34 projects we examined for this audit were relevant to the OP specific objectives.

The focus on results when assessing project applications was weak, and seldom included quantified results

23

We examined how MAs carried out their selection of projects in practice, in particular the existence in the project proposal of clear expected results and objectives accompanied by clearly defined and quantified output and result indicators (linked to the specific objectives). We also assessed whether the MAs had assessed each applicant’s ability to achieve expected results.

24

We found that 18 out of 20 of the selection procedures we examined included criteria requiring beneficiaries to describe their project’s expected results (see Annex VIII). However, only four out of 20 procedures included criteria requiring the quantification of result indicators at project level (selection procedures 17, 18, 19 and 20). And in three cases, these result indicators did not correspond directly to those defined at OP level (see example in Box 1).

25

Our review of the 34 applications of selected projects corroborates these findings. All the applications provided quantified information for output indicators that corresponded to those at OP level, or such data could be obtained directly from the IT system (for instance for indicators such as number of enterprises receiving support). However, only 14 applications provided quantitative information on expected result indicators. In six of these, the result indicators did not correspond directly to the OP indicators. The other applications mainly provided a narrative description (see Figure 2 and see Annex VIII).

26

Regarding the assessment of applications, we found that the MAs had assessed the overall feasibility and likelihood of the projects to succeed. However, while there was an assessment of the achievability of the expected results for all 18 ERDF projects, this assessment was only performed for 2 of the 16 ESF projects we examined. We also found that the assessment of achievability remained vague in the assessment reports, except in seven cases¹⁷ which included a more detailed description about the relevance of the indicators and the likelihood of the targets being met.

27

In terms of scoring the applications, we found that the provision of expected results was part of the scoring process in only 11 cases. This indicates that, as long as the envisaged projects were aligned with the type of actions set out in the selection procedure, the MAs assumed that it would contribute to the OP’s objectives. However, they did not focus on the magnitude of this contribution.

Beneficiaries were encouraged to apply for funding

28

To ensure timely absorption throughout the programming period and at the same time ensure the focus on results, there is a need for a sufficient number of bidders able to provide good applications. The promotion of a call and related guidance to potential applicants by MAs is an important variable determining the overall effectiveness of selection procedures. A broader applicant population is likely to ensure that better projects are selected.

29

The measures aiming at promoting calls to potential bidders were prepared by the MAs on the basis of the detailed requirements set in the EU regulation. They were submitted to the Monitoring Committees for approval as part of the OP’s communication strategy that is required by the Regulation¹⁸.

30

We found that MAs had encouraged beneficiaries to participate in the selection procedures. First, MAs used several sources to promote a call to potential bidders. These included the involvement of Chambers of Commerce, local business associations, and online tools. MAs also relied on partners to circulate information on upcoming calls within their networks, based on the knowledge they acquired in the monitoring committees.

31

Furthermore, the guidance on funding provided by MAs to applicants was publicly available and, except for one MA at the time of the audit, it set out clear conditions for receiving support, provided detailed information on the procedure for application and financing arrangements, referring explicitly to the selection criteria to be applied.

Selection procedures did not usually involve any comparison between project applications, resulting in a risk that the best projects will not be selected

32

In our review of procedures, we checked that projects submitted for funding were assessed against each other. This is another important measure to ensure that the most results-oriented projects are selected.

33

In the framework of shared management, selection procedures were defined by the MAs and approved by the monitoring committees, based on the requirements in the EU Regulations¹⁹ and the Commission’s guidance²⁰. Neither the ESI Funds Regulations nor any Commission guidance specifies the type of selection procedure to be used; this is left to the MAs’ discretion.

34

Out of the 20 procedures we examined, 10 were temporary calls for proposals, 6 were permanent calls, 3 were direct awarding procedures, and one a reiterated call (see also details in Annex III). We found that in all selection procedures except one (IT, OP Piemonte, No 13) no comparison between project applications had taken place. As a result, funding may not have been allocated to the best projects. This was mitigated to some extent in some of the OPs visited, where an informal (French and Finnish OPs) or a formal (Italy Puglia) pre-selection had taken place, allowing the MAs to limit the submission of applications to the most suitable projects. We found the following:

• Fifteen out of twenty procedures were based on the first-come first served principle, where an application was selected provided that it met the criteria and that sufficient funding was available. Of these, 13 had been subject to a formal or informal pre-selection. We noted that some had been required to reach a minimum score for selection (see selection procedures Nos 4 and 5 for the Czech OPEIC and 16, 17, 18, 19 and 20 for the Finnish OP).
• In the three direct awards, there was, by definition, no competition (see selection procedures No 1, 3 and 12).
• In one case (IT, OP Piemonte, selection procedure No 8)²¹, funding had been awarded to projects already selected during the previous 2007-2013 programming period simply by extending their duration. This allowed the IB to avoid interruptions to the provision of training to young people, and allowed it to start absorbing the funds. But it also prevented potential new applicants from participating.

The quality of monitoring data is at risk, and it is mostly outputs-oriented

35

We checked that the monitoring systems allowed the systematic collection of data on specified indicators, thus enabling the measurement of the progress made at project, programme and EU levels. We also examined whether this had been reported on in a way that allowed the success of the financed projects, and thus the effective and efficient use of the EU budget, to be demonstrated.

36

The following section contains our assessment of the necessary conditions to enable monitoring and reporting on the achievement of objectives. We first checked whether the MAs had set up functional monitoring systems before starting to implement the funds. We also looked at whether the performance information reported by the beneficiaries had been reliable, complete and timely. We also examined the role of the MAs and the Commission.

37

Finally, we consider whether the systems in place produced results-oriented performance information; specifically, whether they produced meaningful aggregated information, as this is a key element to allow reporting on the effective use of funds at Member State and EU level via annual monitoring and reporting.

Assurance on the quality of monitoring data may not be provided in time for the 2019 performance review

38

The monitoring of an OP’s progress towards its objectives is done on the basis of the data collected from various sources. This data is aggregated at OP and Member State level, and forms the basis for the reports sent to the OP’s monitoring committee and to the Commission. In order to present a correct view of the situation at OP or Member State level, it is essential that the data collected is reliable, complete and timely. To achieve this, the MAs have to set up a monitoring system including the collection of data and checks on the source of the data. In order to gain assurance on the quality of the data, the Commission relies on the audit work carried out by the audit authorities at Member State level as well as its own audit work.

39

In the specific context of the performance review, it is also important that the data, and the systems used to produce them, be audited as early as possible, to allow Member State authorities to address any detected weaknesses regarding the data quality well before the end of June 2019. It is then that the 2018 Annual Implementation Reports (AIRs) – the basis for the performance review and the allocation of the performance reserve – will be sent to the Commission.

Extensive EU rules lay down how Member States should set up their monitoring systems for the 2014-2020 period

40

With a view to achieving the systematic collection and monitoring of data produced at OP level, the regulatory framework for the 2014-2020 programme period introduced several changes concerning data collection and monitoring arrangements:

• MAs must put in place a management and control system (MCS) covering adequate systems for reporting and monitoring²².
• MAs are responsible for establishing a computerised system to record all the data related to the OPs indicators²³ for monitoring and reporting purposes.
• Under EAC 7, MAs must put in place “a system of result indicators necessary to select actions which most effectively contribute to desired results, to monitor progress towards [those] results”²⁴. OPs were required to demonstrate fulfilment of all the ex-ante conditionalities (EAC) by 31 December 2016²⁵ (see paragraph 4).
• By 31 December 2015²⁶, Member States were required to ensure that all exchanges of information between beneficiaries and a MA, Certifying Authorities (CA), AAs and IBs could be carried out electronically²⁷.

41

The Commission has also established specific guidance²⁸, for its staff and the Member States’ responsible authorities, on a common methodology for assessing management and control systems in the MS, including the existence of adequate systems for monitoring and reporting.

Performance information is not always reliable, complete and timely, and this has an impact on the implementation reports issued by the Member States

Data reported by the beneficiaries

42

For the ERDF, in many cases, the data for result indicators is collected from national statistics bodies (see examples in Annex VI). The remainder comes from beneficiaries. For ESF, most of the data is reported via data collected from beneficiaries (see Annex VII). Managing authorities check that the data collected from beneficiaries is correct at project completion, when receiving their final report. During project implementation, checks can also be done on-the-spot, on a sample basis.

43

A number of plausibility and consistency checks are carried out automatically by the systems in place on the data provided by the beneficiaries, especially for the ESF. MAs also have tools to monitor the completion of the data provided on participants which allow them to remind beneficiaries who have submitted incomplete data to provide further information.

44

Our audit revealed issues regarding the quality of the data collected. We identified a general issue regarding the completeness and correctness of the data concerning participants to ESF co-financed actions. This data is classified as sensitive in national legislation²⁹, so it is up to participants to decide whether to declare it. It is also difficult for the Member States and the Commission to check its accuracy. An additional weakness regarding the collection of data is presented in Box 2.

45

In its audit of performance data reliability (2014-2020), the Commission reported on other issues regarding the reliability of data, for instance:

• data not collected which resulted in achievements not being recorded and thus not being taken into account in order to determine the progress of an OP, leading to an incorrect understanding of the situation;
• a misunderstanding on the MAs’ part regarding the definition of common indicators (for instance, Common Output indicator 26, 'Enterprises cooperating with research organisations') resulting in an overstatement of the number of companies reported under this indicator;
• double counting/failure to cap the value for indicator 'population served' at the number of area residents, resulting in inconsistent values.

Availability and reliability of performance information

46

The existence of functional monitoring systems at the time MAs start implementing programmes is a key element ensuring that performance data is generated in time, which ensures that monitoring can take place.

47

The late approval, in December 2013, of the legislative framework for the period 2014-2020 at EU level, delayed the development of the IT systems used for monitoring. For the ESF indicators, added complexity concerning the sensitivity of the data and the definition of the indicators, required additional discussions well into 2014. This further delayed the finalisation of the IT systems. In two cases, other factors led to the delays: reorganisation of the regions (France, OP Lorraine et Vosges), and the design of a new single IT system for all 2014-2020 ESI funds in the Czech Republic.

48

The effects of the delay are reflected in the level of implementation of EAC 7 on the existence of a system of results indicators. At the end of 2016, the deadline to complete action plans, 50 plans remained incomplete, i.e. around 11 % of all OPs still did not fulfil this requirement. However, by the end of February 2018, all the action plans related to EAC 7 had been completed.

49

In the first half of 2017, the fourth year after the start of the programme period, we found that some IT systems were still not fully functional; this poses risks to the completeness and accuracy of the data collected (see examples in Box 3). In its audit of performance data reliability (2014-2020)³⁰, the Commission drew attention to the absence of electronic data exchange systems with beneficiaries, resulting in the manual encoding of the data on indicators by the MA into the system, a potential source of clerical errors.

Implementation reports issued by the Member States

50

The MAs must submit to the Commission an Annual Implementation Report (AIR) for each OP, which is approved by the Monitoring Committee. The AIR delivers key information on the implementation of the OP, mostly by providing quantitative and qualitative information on common and programme-specific indicators³². The 2016 AIRs, submitted in June 2017, were the first reports sent to the Commission on the “progress made towards achieving the objectives of the programme”³³. The Commission carries out a quality review of the AIRs (consistency of number of participants with population of the areas covered, for instance).

51

At the end of 2017, 26 AIRs had still not been accepted by the Commission due to missing data for the indicators used in the performance framework, inconsistencies in the data reported with the information provided on the implementation of the OP, or a lack of explanation for some of the data reported. Around 40 % of ERDF OPs had not reported any values for their performance framework indicators in 2017, on which there was no Commission analysis at the time of the audit.

52

The shortcomings identified in the AIRs raise questions about the reliability of the information provided as well as the level of progress reached until now as presented in the documents issued by the Commission based on the AIRs³⁴: the annual synthesis report to the Parliament and the Council, European Economic and Social Committee and the Committee of the Regions and, in 2017 and 2019 only, the strategic progress report, which summarises progress made.

Audits on management and control systems and the reliability of collected data started late, limiting the degree of assurance on the monitoring systems and reducing the time available to address detected weaknesses

53

Appropriate arrangements for the verification and correction of the performance information are necessary to ensure its quality and hence, its use in the context of monitoring and reporting. Good practices requires audits on the systems producing the data - and associated corrective actions - to be carried out and finalised before the data is included in AIRs and strategic reports.

54

Under the legislative framework, both the Member States’ audit authorities and the Commission³⁵ audit the functioning of the monitoring systems. These audits should, in particular, ensure the existence of reliable systems for collecting, recording and storing performance data, and assess whether this performance data is reliable.

Audit authorities’ assessment of the functioning of the management and control systems

55

An assessment of the systems in place at MA level was carried out during the early stages of the implementation of the OPs as part of the designation procedure³⁶. In this procedure, an independent audit body (often the AA) was required to produce a report and an opinion on whether the MAs had fulfilled the criteria relating to the internal control environment, risk management, management and control activities and monitoring³⁷. The designation was required to take place before the first submission of an interim payment for the OPs. Thus, without this designation, OPs cannot be reimbursed. At this stage, the audit covers the existence of systems and procedures, but it does not cover good implementation, which is checked in the context of the audit authorities’ system audits on management and control systems.

56

Due to the late approval of the OPs and late finalisation of the IT systems (see paragraph 47), the designation process was late. This was also affected by national factors (reorganisation of the regions in France in 2015, complex management structures - for instance, the French PON FSE has 13 delegated managing authorities and 120 intermediary bodies). For the OPs examined in this audit, the MAs were designated between September 2016 and February 2018, more than two and four years into the programme period respectively (see Annex IV).

57

Due to the late implementation of OPs and the resulting limited amount of data collected, systems audits were only beginning at the time of our audit visits i.e. during the first half of 2017. Any problems regarding the quality of data thus did not become apparent until the second half of 2017. This was almost four years after the start of the seven-year programme period and one-and-a-half years before the 2018 AIRs, which form the basis for the allocation of the performance reserve, are forwarded to the Commission.

The Commission’s audits

58

In addition to relying on the audit work and systems in place in the Member States, the Commission also carries out its own audits. There are two types: “early preventive systems audits”, aimed at obtaining reasonable assurance at an early stage of the implementation that the management and control systems are functioning well, and “performance reliability data audits”. But given the aforementioned delays, and thus the late finalisation of the monitoring systems as well as the minimal reporting of relevant data, these audits were conducted later than planned. Especially for the ERDF, audits could only start after the 2016 AIRs were received. In total, by the end of 2017, the Commission managed to carry out 23 audits in 12 Member States: 12 performance reliability data audits and 11 early preventive systems audits.

59

For most of these audits, reporting was ongoing at the beginning of 2018. According to the Commission, the preliminary results for the audited systems were rather positive, even though some shortcomings were highlighted. At the same time, however, around 40 % of ERDF OPs did not report any values for their performance framework indicators³⁸ in the 2016 AIRs submitted in 2017. For these OPs, in February 2018, the Commission had not carried out audits to determine whether the absence of reporting was due to delays in implementation or a lack of a functioning data-reporting system.

60

The late undertaking of the systems audits means that the overview of the monitoring systems’s status is not complete. Shortcomings are particularly susceptible to being identified late and there may not be sufficient time to apply the necessary adjustments. This jeopardises the implementation of the performance review in 2019, which will be based on data reported by Member States in the 2018 AIRs by 30 June 2019.

The performance review may not be based on realistic milestones

61

In the OPs we visited, on the basis of the 2015 AIRs or in the monitoring information available at the time of the audit, we noted that there were already cases where OPs had achieved the targets set for 2023, as well as risks of underachievement. Requests for amendments in order to adjust the targets to a more realistic value, had not been submitted for the OPs visited. Only one (Czech Republic, OPEIC) had already asked for, and obtained, amendments, mainly in relation to the modification of allocations within priority axes. On the other hand, according to the Commission, in February 2018, out of the 17 480 indicators used in the existing 430 OPs, the corresponding target value for 2023 had been modified upwards in 1445 instances and downwards in 941 instances.

62

This indicates that a number of milestones³⁹ of the performance framework are no longer realistic. In case milestones need to be revised, this requires that amendments be made to the OP, which can only be done at the Member States initiative⁴⁰. Given the duration of the amendment procedure (set in the Regulation at three months), there is a risk that the values reached at the end of 2018 by the performance-framework indicators are not compared with realistic milestones.

The information generated on the basis of the data collected is still mostly outputs-oriented

63

Performance information is collected so that the Member States and the Commission can use it for reporting purposes, in order to be accountable for what has been achieved with funding.

64

Result indicators and targets laid down in the grant agreements signed between beneficiaries and MAs are essential for measuring the achievement of results. However, several key issues in the set-up of the monitoring systems are affecting the measurement and monitoring of the contribution of projects to the achievement of the OP objectives.

A set-up which relies mostly on output indicators, especially in the early years of implementation for the ERDF, and a lack of quantified result indicators defined at project level

65

The current set-up allows for a more systematic collection of data for a number of well-defined indicators, though the number of indicators is very high. The existence of common indicators also allows for the aggregation of this data at OP, Member State and EU levels. However, in the set-up of the logical framework for the ERDF, many result indicators at OP level are not directly related to the funded interventions. These indicators are often national indicators which take into account the influence of external factors, and their sources are mainly national statistics offices (see examples in Box 4 and see also Annex VI). These result indicators do not isolate the impact attributable to the ERDF interventions, and cannot therefore be considered immediate results of the OPs.

66

To mitigate the fact that results generated by ERDF projects often materialise (well) after project completion, at a time when there is no monitoring of the projects as such, MAs must carry out evaluations at priority axis level⁴¹. At least once during the programming period, these evaluations assess how support from the ESI Funds has contributed to the objectives for each priority. But given the late start of the implementation, the number of evaluations already carried out is still limited. As a consequence, for the time being, the progress of the OP is mainly measured through the output indicators.

67

At project level, we found that result indicators were not systematically mentioned in the grant agreements (see paragraph 25). Only six⁴² out of the 18 ERDF projects examined in this audit contained results indicators in the applications and thus in the grant agreement. However, these result indicators were not the same as those used at OP level. This means that they cannot be used for reporting purposes or to measure the progress made in achieving OP objectives. In four other cases⁴³, the results indicators were not quantified (i.e. no target was given) thus preventing an assessment of their contribution to the OP objective from being made and limiting the measurement of the project’s achievement to its outputs.

68

For the ESF projects, result indicators were mentioned in 7 out of 16 projects. For the remaining projects, the results described were either not related to the OP indicators, or they were not quantified. In any given project, this makes it impossible to compare the value reached with an estimated target. However, in the case of the ESF, even if results and result indicators are not defined at project level or quantified, the measurement of the values for the common result indicators is still taking place, as project beneficiaries have to report those data in the MA’s IT systems.

69

We have previously mentioned the lack of result indicators defined at project level⁴⁴ in our annual reports. This makes it difficult to measure and to monitor the extent to which projects contribute to the achievement of the OP objectives.

The release of the performance reserve is based on indicators that are mostly input- and outputs-oriented

70

Member States can demonstrate progress in programme implementation through milestones (to be reached by 2018) and target values (to be reached by 2023) for three types of indicators: financial, output and result. They can be complemented by key implementation steps (KIS) to measure projects that are on-going or scheduled to be started, but for which it is unlikely that any outputs can be achieved by December 2018. Overall, the indicators used in the performance framework must be representative of the actions under the priority axis. In order to trigger the release of the performance reserve, the values reached at the end of 2018 for these indicators must be between 75 % and 85 % of the milestone value.

71

As shown in a previous ECA report⁴⁵, the vast majority of the indicators used in the performance framework under the cohesion policy related to output (57 %), key implementation steps (9 %) and financial indicators (33 %), with the use of result indicators remaining marginal. Thus, the release of the reserve will to a large extent remain input- and outputs-oriented, and not focused on results (see also paragraph 62).

Most reporting at EU level concerns output indicators

72

The main objective of the Commission’s strategic report 2017⁴⁶, which encompasses the 2017 annual synthesis report drawn up based on the 2016 AIRs, is to report on the achievements of the ESI Funds towards attaining the EU2020 objectives. However, the report mainly presents the level of implementation as well as the values reached for the main output indicators by the end of 2016. It presents no information on the achievement of results, except for the following ESF result indicators: the number of participants that found a job immediately after the training, and the number of participants that gained a qualification immediately after training. For the ERDF, this is also due to the fact that there are no common result indicators; those used at Member State level cannot, therefore, be aggregated.

73

As we have reported recently, there is a fundamental problem affecting result indicators in particular. In fact, ERDF and ESF result indicators measure different things in different ways⁴⁷. Furthermore, the concept of a “result” is interpreted differently in the fund-specific regulations. The ERDF result indicators cannot be used to measure the immediate results of the OPs. In addition, the high number of different performance indicators pose a real challenge to the collection and reporting of performance information⁴⁸.

Conclusions and recommendations

74

Although the set-up of OPs is now more results-oriented, with a stronger intervention logic and an extensive set of indicators, overall we conclude that the selection of projects under the ERDF and ESF is not fully results-oriented and that the monitoring in place is still more outputs-oriented.

Selection procedures were not results-oriented in important aspects

75

For the OPs we visited, selection procedures were used to support the selection of projects relevant to the OPs’ objectives. However, they do not systematically include selection criteria requiring the definition of quantified result indicators at project level corresponding to those at OP level. Consequently, result indicators were rarely included in applications; even when this was the case, they did not necessarily correspond to the OP indicators, or had not been quantified. Furthermore, only one-third of the assessment reports included a specific assessment of the expected results in relation to OP objectives (see paragraphs 18 to 27).

76

We found that MAs had provided comprehensive information on the OPs, on access to EU funding, and on the selection procedures through several channels. Beneficiaries were generally well informed, or could rely on support from specialised organisations (see paragraphs 28 to 31).

77

However, only 1 of the 20 procedures examined involved a scoring and a ranking of applications against each other. Projects were generally approved on a first-come first-served nature. As a result, funding may not have been allocated to the best projects (see paragraphs 32 to 34).

There are risks to the quality of monitoring data, which remains more outputs-oriented rather than results-oriented

78

Following the late adoption of the legislative framework, we found that some of the OPs’ monitoring systems were not operational at the start of the programme nor fully operational at the time of the audit. Consequently, data was input manually and at a later stage. Data collection also occurred outside the IT systems, creating errors, which, in some cases, impacted upon the data presented in the AIRs (see paragraphs 35 to 52).

79

The late adoption of the regulatory framework led, in most OPs, to a late start of the implementation. Until the end of 2016, only a limited amount of data was collected in IT systems. Consequently, the audits necessary to provide assurance on the monitoring systems and the reliability and availability of the data mostly started in 2017. The results of these audits only became available at the end of 2017, four years into the seven-year programme period. This means that assurance obtained upon the monitoring data is, up to now, only partial. The Commission thus does not have an overview of the functioning of the monitoring systems, including IT systems. The performance review will take place in 2019 and, in the remaining time available, there is a risk that any corrective measures cannot be completed in time and that the performance review will not be based on correct information (see paragraphs 53 to 60).

80

We found that a number of milestones of the performance framework are no longer realistic. In case milestones need to be revised, there is no certainty that the necessary OP amendments can be made in time for the performance review (see paragraphs 61 and 62).

81

Regarding result indicators, ERDF and ESF indicators measure different things in different ways. In fact, the concept of a “result” is interpreted differently in the fund-specific regulations. For both funds, we identified the risk that a meaningful aggregation of performance data (in particular regarding results) may not be feasible at EU level⁴⁹. Furthermore, we recommended⁵⁰ that the Commission should define a common definition of “outputs” and “results” (see paragraph 73).

82

For the ERDF, many result indicators often use other sources, such as data from national statistics offices. Evaluations should measure achievements against the OPs’ objectives. However, given the late start of implementation, few evaluations were available at the time of our audit. Consequently, for the ERDF, it is difficult to monitor to what extent the projects contribute to the achievement of the OP objectives (see paragraphs 63 to 69). This problem also applies to the ESF, as many grant agreements do not include quantified result indicators.

83

Similarly, output and financial indicators predominantly form the basis of the performance framework. This means that the release of the performance reserve will be mainly based on these types of indicators rather than on the results achieved and the actual progress made towards the OP objectives. Also, the information published in the Commission’s reporting is mostly outputs-related, except for the ESF where the aggregated value for some result indicators is provided (see paragraphs 70 to 72).

Recommendation 1 – Results-oriented selection

In order to ensure a consistent and truly results-oriented approach to the selection of projects, Member States should:

1. have selection criteria requiring the beneficiaries to define at least one genuine result indicator based on a common definition of what constitutes a “result” for their project, including a baseline and a target. This result indicator should contribute to the result indicators set at OP level;
2. include an assessment of these expected results and indicators in the assessment report for the applications;
3. when deciding on the selection procedures to be used, ensure that a comparison between project applications takes place.

Target implementation date: starting in 2019 for upcoming calls for proposals.

Recommendation 2 - Results-oriented monitoring

1. In order to allow the monitoring of projects’ contributions to OP objectives, Member States should include quantified result indicators in the grant agreement, contributing to the results indicators set at OP level.

   Target implementation date: starting in 2019 for upcoming calls for proposals.

2. In order to make the ERDF monitoring framework more results-oriented, and in particular to allow the aggregation of performance information, the Commission should define common result indicators for this fund based on a common definition of “results”.

Target implementation date: preparation of the next MFF.

Recommendation 3 - Reporting on performance and preparation of the performance review in 2019

To improve the Commission’s reporting on performance and to carry out a meaningful performance review with reliable data and realistic milestones, the Commission should:

1. have an overview of the main weaknesses in and uncertainties about the OPs’ monitoring systems based on audits by both the Commission and the AAs (the necessary systems audits/performance-reliability audits);
2. ensure that OP amendments requested by Member States relating to justified revisions of milestones of the performance framework are processed in time for the performance review.

Target implementation date: before the finalisation of the performance review.

This Report was adopted by Chamber II, headed by Mrs Iliana IVANOVA, Member of the Court of Auditors, in Luxembourg at its meeting of 27 June 2018.

Glossary

Europe 2020 Strategy: The EU’s growth strategy for the 2010–2020 period, aimed at allowing it to recover from the crisis. The strategy is split into five headline targets covering employment; research and development; climate and energy; education; and social inclusion and poverty reduction.

European Code of Conduct on Partnership: A set of principles set out in Commission Delegated Regulation (EU) No 240/2014 to support Member States in organising partnerships for drawing up and implementing partnership agreements and operational programmes.

European Structural and Investment Funds (ESIF or ESI funds): The ESI funds are five separate funds that aim to reduce regional imbalances across the Union, with policy frameworks set for the seven-year MFF budgetary period. The funds include the European Regional Development Fund (ERDF), the European Social Fund (ESF), the Cohesion Fund (CF), the European Agricultural Fund for Rural Development (EAFRD) and the European Maritime and Fisheries Fund (EMFF).

Ex-ante conditionalities: Conditions which Member States must meet before they can receive any funding under the European Structural and Investment Funds. When preparing operational programmes in the 2014-2020 programme period, Member States were required to assess whether these conditions have been fulfilled. If they have not, action plans were required to be drawn up to ensure fulfilment by 31 December 2016.

Financial indicators: Values used to monitor progress in terms of the (annual) commitment and payment of the funds available for any project, measure or programme in relation to its eligible cost.

Indicator: A measurable proxy for an objective, providing information to support an assessment of the degree to which an objective has been met.

Intervention logic: The link between the assessed needs, objectives, inputs (planned and allocated), outputs (targeted and achieved) and results (intended and actual).

Investment priorities: For each priority axis, Member States must set out in their operational programmes the investment priorities and corresponding specific objectives. They are listed in the specific Regulations governing the ERDF, ESF and CF.

Managing authority (MA): A managing authority is a public or private body which has been designated by a Member State to manage an operational programme. Its tasks include selecting projects to be funded, monitoring how projects are implemented, and reporting to the Commission on financial aspects and achieved results.

Monitoring: The regular examination of expenditure, outputs and results, which provides up-to-date information on whether projects/programmes are progressing as intended.

Monitoring committee: A committee which monitors the implementation of an OP. It consists of “representatives of the relevant Member State authorities” (e.g. representatives of the managing, certifying and audit authorities, implementing bodies, employer or employee organisations, and civil society). The Commission also has an observer role.

Operational Programme (OP): A statement of a Member State’s priorities and specific objectives which describes how funding (EU and national public and private co-financing) will be used over a given period (currently seven years) to finance projects. The projects within an OP must contribute to a certain number of objectives specified at the level of the OP’s priority axes. OP funding may come from the ERDF, CF and/or ESF. The OP is drawn up by the Member State and has to be approved by the Commission before any payments can be made from the EU budget. OPs can only be modified during the programme period if both parties agree.

Outputs: Something that is produced or accomplished with the resources allocated to an intervention (e.g. training courses delivered to unemployed young people, number of sewage plants, or a length of road built).

Output indicators: Values used to measure the outputs of the operations supported or the outputs at operational programme level.

Performance framework: A set of milestones and targets defined for each priority axis in an Operational Programme, forming an important pillar of the performance-orientation approach.

Performance reserve: Resources constituting 6 % of the resources allocated to the ERDF, ESF and CF or to the EAFRD and EMFF, to be made available after the 2019 performance review if certain requirements are met or exceeded.

Priority axis: One or more investment priorities associated with one thematic objective. The funding in an operational programme is organised by priority axis.

Results: Immediate changes that arise for beneficiaries at the end of their participation in an intervention (e.g. improved accessibility to an area due to the construction of a road, trainees who have found a job).

Result indicators: Values used to measure the results generated by supported projects, or the results achieved at operational programme level.

Selection/award criteria: Predefined criteria used in selection procedures to assess applicants’ ability to complete the proposed action or work programme.

Specific objective: The envisaged result to which EU action aims to contribute.

List of abbreviations

AA: Audit Authority

AIR: Annual Implementation Report

CA: Certifying authority

CF: Cohesion Fund

CPR: Common Provision Regulation

CSF: Common Strategic Framework

DMA: Delegated Managing Authority

EC: European Commission

ERDF: European Regional Development Fund

ESF: European Social Fund

ESI funds or ESIF: European Structural and Investment Funds

IB: Intermediate body

IP: Investment priority

MA: Managing Authority

MS: Member State

OP: Operational Programme

PA: Priority axis

SMEs: Small and medium-sized enterprises

SO: Specific objective

TO: Thematic objective

Audit team

The ECA’s special reports set out the results of its audits of EU policies and programmes, or of management-related topics from specific budgetary areas. The ECA selects and designs these audit tasks to be of maximum impact by considering the risks to performance or compliance, the level of income or spending involved, forthcoming developments and political and public interest.

This performance audit was carried out by Audit Chamber II Investment for cohesion, growth and inclusion spending areas, headed by ECA Member Iliana Ivanova. The audit was led by ECA Member Ladislav Balko, supported by Branislav Urbanič, Head of Private Office and Zuzana Frankova, Private Office Attaché; Myriam Cazzaniga, Principal Manager; Pekka Ulander Head of Task; Michaela Binder, Seconded National Expert.

Contact

EUROPEAN COURT OF AUDITORS
12, rue Alcide De Gasperi
1615 Luxembourg
LUXEMBOURG

Tel. +352 4398-1
Enquiries: eca.europa.eu/en/Pages/ContactForm.aspx
Website: eca.europa.eu
Twitter: @EUAuditors

More information on the European Union is available on the internet (http://europa.eu).

Luxembourg: Publications Office of the European Union, 2018

PDF	ISBN 978-92-847-0548-1	ISSN 1977-5679	doi:10.2865/390387	QJ-AB-18-018-EN-N
HTML	ISBN 978-92-847-0523-8	ISSN 1977-5679	doi:10.2865/439457	QJ-AB-18-018-EN-Q

© European Union, 2018.

For any use or reproduction of photos or other material that is not under the European Union copyright, permission must be sought directly from the copyright holders.

GETTING IN TOUCH WITH THE EU

In person
All over the European Union there are hundreds of Europe Direct Information Centres. You can find the address of the centre nearest you at: https://europa.eu/european-union/contact_en

On the phone or by email
Europe Direct is a service that answers your questions about the European Union. You can contact this service:

• by freephone: 00 800 6 7 8 9 10 11 (certain operators may charge for these calls),
• at the following standard number: +32 22999696, or
• by email via: https://europa.eu/european-union/contact/write-to-us_en

FINDING INFORMATION ABOUT THE EU

Online
Information about the European Union in all the official languages of the EU is available on the Europa website at: https://europa.eu/european-union/index_en

EU Publications
You can download or order free and priced EU publications: https://publications.europa.eu/en/publications. Multiple copies of free publications may be obtained by contacting Europe Direct or your local information centre (see https://europa.eu/european-union/contact_en)

EU law and related documents
For access to legal information from the EU, including all EU law since 1951 in all the official language versions, go to EUR-Lex at: http://eur-lex.europa.eu

Open data from the EU
The EU Open Data Portal (http://data.europa.eu/euodp/en/home?) provides access to datasets from the EU. Data can be downloaded and reused for free, both for commercial and non-commercial purposes.