Special Report
NO21 2018

Selection and monitoring for ERDF and ESF projects in the 2014-2020 period are still mainly outputs-oriented

About the report For the Cohesion policy funds during the 2014 – 2020 programme period, we examined how well Member States focus on results in selecting projects and how well the Commission and Member States can demonstrate, through their monitoring, that the EU budget is well spent. We found that although the Commission has taken various measures to increase results-orientation, selection procedures continue to emphasize outputs and absorption rather than results. Moreover, shortcomings in the monitoring arrangements made it difficult to assess the extent to which EU funding has contributed to the achievement of the EU’s and Member States’ objectives.

We recommend that, during the selection process, the Member States ensure a comparison of project applications, require beneficiaries to identify at least one real result indicator for each project to be included in the grant agreement and contributing to the OP indicators, and assess expected results and indicators in the evaluation report of the applications. The Commission should define common result indicators for the ERDF, improve reporting on performance, and ensure a meaningful performance review in 2019.

This publication is available in 23 languages and in the following format:
PDF
PDF General Report

Executive summary

I

For the 2014-2020 programming period, the EU has allocated almost €349.4 billion towards achieving its objectives in terms of cohesion policy. This policy aims to support job creation, business competitiveness, economic growth, sustainable development and improvement in quality of life. To achieve these objectives, funding is allocated to projects delivered by beneficiaries in Member States. Authorities in Member States select projects to fund and monitor their implementation. It is critical that this funding is allocated effectively, namely through the delivery of expected results.

II

In order to support this goal, the Commission implemented various measures aimed at increasing focus on results in the 2014-2020 period. In previous reports, we have welcomed the improvements that these have brought. But we have also highlighted serious weaknesses in the effectiveness of these measures, and the quality of the monitoring information relating to spending on Cohesion Policy. This report consolidates and develops these previous findings.

III

We examined how well the focus on results is embedded within the way projects are selected, and how well the Commission and Member States can demonstrate the EU budget is well spent. As such, we have examined the selection and monitoring arrangements of projects using Cohesion funding.

IV

We conclude that, despite longstanding intentions, the design of selection procedures and the processes themselves continue to emphasise outputs and absorption rather than results. Moreover, shortcomings in the monitoring arrangements made it difficult to assess the extent to which EU funding has contributed to the achievement of the EU’s and Member States’ objectives. Specifically, for the OPs we visited, we made the following findings:

  • On project selection, potential beneficiaries were well informed and provided with appropriate support to access to EU funding. The procedures we examined were designed to support the selection of projects relevant to the OPs’ objectives, but selection criteria seldom required applicants to define quantified result indicators at project level. Furthermore, most project selection was done on a first-come first-served basis. Applications were only scored and ranked against each other in one selection procedure.
  • On monitoring, we found that monitoring systems had only become functional at a late stage, mainly because of delays in adopting the legislative framework. There remain weaknesses in some of the IT systems used for collecting and aggregating monitoring data. Furthermore, slow progress in the audits of the respective monitoring systems introduced the risk that there now may not be enough time to take any necessary corrective action before the performance review in 2019.
  • Monitoring information remains mostly outputs-oriented. The Commission’s main report to measure achievements presents progress for the main output indicators alongside the implementation of funding. However, there is only limited information on the achievement of results.
V

We therefore recommend the following:

  • In order to ensure a consistent and genuine results-oriented approach to the selection of projects, Member States should ensure that a comparison between project applications takes place, require beneficiaries to define at least one genuine result indicator for each project and carry out an assessment of the expected results and indicators in the assessment report for the applications.
  • In order to ensure a results-oriented monitoring, Member States should include one or more genuine and quantified result indicators in the grant agreement, which contribute to the result indicators set at OP level, and the Commission should define common result indicators for the ERDF based on a common definition of “results”.
  • The Commission should improve the reporting on performance and ensure that a meaningful performance review takes place in 2019.

Introduction

Cohesion policy is the EU’s main investment policy

01

Cohesion policy is the EU’s main investment policy. It is aimed at supporting job creation, business competitiveness, economic growth, sustainable development and improvements in quality of life. Around one-third of the EU’s budget is allocated to cohesion policy. At current prices, this amounted to around €230 billion in the 2000-2006 programme period, €346.5 billion in the 2007-2013 period and €349.4 billion in the 2014-2020 period.

02

Cohesion policy is delivered through the European Regional Development Fund (ERDF), the European Social Fund (ESF) and the Cohesion Fund (CF). These funds, together with the European Agricultural Fund for Rural Development (EAFRD) and the European Maritime and Fisheries Fund (EMFF), are part of the European Structural and Investment Funds (ESIF). They are implemented through operational programmes (OPs) which set out investment priorities and specific objectives1. These also describe how funding will be used during the programme period to finance projects. The OPs are implemented by the Member States and their regions. This means that the Member States and their regions select, monitor and evaluate projects.

Performance orientation is a key priority for the Commission and the Member States during the 2014-2020 period

03

The Europe 2020 Strategy is the EU’s ten-year jobs and growth strategy. It was launched in 2010 to create conditions conducive to smart, sustainable and inclusive growth. With a view to meeting the objectives and targets set in the Europe 2020 Strategy, the Commission emphasised in 2010 the need to improve the effectiveness of the cohesion policy by focusing on results2.

04

The legislative package for cohesion policy in the 2014-2020 programme period therefore introduced significant changes intended to increase focus on performance3. The main measures are the following:

  • The introduction of ex-ante conditionalities (EACs)4 which require a Member State to fulfil certain conditions for the effective use of ESI funds (e.g. existence of policy/strategic frameworks) by the end of 2016. One of them, general ex-ante conditionality No 75, requires “the existence of a system of result indicators necessary to select actions which most effectively contribute to desired results, to monitor progress towards results”.
  • The introduction of a performance reserve, setting aside part of the EU funding allocated to the OPs, to be released only if a subset of indicators (mainly spending/output indicators) reach pre-defined milestones. The release of this reserve in 2019 will be based on a performance review carried out in 20196.
  • A reinforced intervention logic during the programming exercise7 requiring Member States to assess their needs and commit to a set of results before considering where and how to spend cohesion funds.
  • A more consistent use of performance indicators to measure progress towards achieving results8.

The importance of project selection and monitoring to achieve performance

05

In 2017, we concluded9 that the Commission and Member States had been successful in developing OPs with a more robust intervention logic. This means that the OPs set out in a consistent way the aims of the interventions (specific objectives/results) and how these will be achieved (required funding, actions to be undertaken and expected outputs).

06

However, to achieve policy effectiveness, it is not only important to set clear goals and consider results during the programming exercise. It is essential that a results-oriented approach is followed through during the implementation phase of the OP, as it is mainly the quality of projects funded and their added value in terms of tangible results which will determine the effectiveness of a policy. Project selection, monitoring and reporting is thus essential for achieving performance. An overview of the implementation cycle of an OP and the bodies concerned is provided in Figure 1. The processes relevant to our audit are highlighted in yellow.

Figure 1

Overview of the implementation cycle of an OP

Source: ECA.

Main participants and their role in project selection and monitoring

07

Projects are selected by the national and regional authorities responsible for managing the OPs. These managing authorities (MAs), or intermediate bodies (IBs) to which they delegate powers, lay down selection criteria, organise selection committees and decide which projects will receive European funding. The Commission is not involved in the selection of projects as opposed to its crucial role in the negotiation of OPs. However, the Commission is involved through its advisory role in the OPs monitoring committees10 where the methodology and criteria used for selecting projects are approved. It also provides guidance, based on best practices and shared experience with Member States.

08

The responsibility for monitoring projects lies with the Member States. Managing authorities monitor the implementation of OPs and projects, aggregate monitoring information and submit Annual Implementation Reports (AIRs) to the Commission. The monitoring committee also reviews the implementation of the OPs and approves AIRs. Here also, the Commission’s role is limited to issuing guidance, examining the performance of OPs together with the MAs, and making observations on AIRs.

09

In its reporting role nevertheless, the Commission submits its annual synthesis of Member States’ AIRs and strategic reports (in 2017 and 2019) to the discharge authority (European Parliament), and to the European Economic and Social Committee and the Committee of the Regions11.

Audit scope and approach

10

This audit sought to answer the question of whether the selection of projects and monitoring systems for ERDF and ESF projects in the 2014-2020 programme period were results-oriented. Given the level of advancement of the OPs’ implementation, and building on the audits already carried out, this audit allowed us for the first time to follow the “results-oriented approach” through the whole implementation phase as the first results are becoming available. It is also of high relevance to the performance review scheduled for 2019.

11

In the first part, we report on our analysis of selection procedures in place for the audited OPs. We assess the design of these selection procedures, especially the determination of the selection criteria and their result orientation, as well as the phase of evaluation of the applications. In the second part, we examine the management and control systems in place and the monitoring that has been carried out so far by the MAs/IBs of the visited OPs to measure operational performance at programme and national level. We review the reliability of data, its availability and capacity to produce relevant information for management and for reporting to the Commission.

12

In total, we examined 34 projects (see list of projects in Annex I). We focused on the ERDF and ESF, and more specifically on three thematic objectives (TOs)12 (see a description of the 11 thematic objectives in Annex II):

  • TO3, enhancing the competitiveness of SMEs (16 projects),
  • TO8, promoting sustainable and quality employment and supporting labour mobility (12 projects), and
  • TO9, promoting social inclusion and combating poverty and discrimination (6 projects).
13

We looked at projects selected directly by the MA (or intermediary body) through 20 selection procedures (see list in Annex III), from seven OPs located in four Member States: the Czech Republic, France, Italy and Finland (see list in Annex IV). These OPs and Member States were selected according to the materiality of funding allocated and the stage of the projects’ implementation. We selected the projects based on the amount of grant received and their level of advancement at the time of the audit. The priority axis, investment priorities and specific objectives to which the selected projects correspond are listed in Annex V. The related output and result indicators are listed in Annex VI for the ERDF and Annex VII for the ESF.

14

We also reviewed the actions taken by the Commission in relation to project selection and monitoring of operational programmes for the ESF and the ERDF in general.

Observations

Project selection is insufficiently focused on results

15

The selection procedures put in place by the managing authorities should prioritise the selection of those projects that are the most relevant to the objectives pursued and which have the greatest likelihood of delivering results, as well as ensuring that available funding is spent in line with the rules. We checked the extent to which results had been reflected in the design of selection procedures (e.g. the selection criteria used for assessing an application, the method used for assessing projects and the documentation available to potential bidders) as well as in the implementation of the selection procedures and thus in the projects selected.

16

Previous ECA audit work13 showed that the Commission and Member States had been successful in developing OPs with a more robust intervention logic i.e. with a clear link between development needs, specific objectives and result indicators. This intervention logic should be reflected in project selection procedures. We consider that there are a number of key criteria that should be taken into account by MAs when designing procedures to ensure that they are results-oriented:

  • the selection criteria applied should ensure that projects are aligned with OP objectives;
  • the selection criteria should support the selection of projects that have not only quantified output indicators but also quantified result indicators to measure projects achievements;
  • beneficiaries should be encouraged to submit applications to ensure that a high number of projects are available for selection;
  • finally, the project selection should be based on a direct comparison of applications to identify and prioritise the projects best suited for funding.
17

The extent to which MAs applied these variables while processing the applications received and how they influenced the selection of results-oriented projects is described in the next paragraphs (see paragraphs 18 to 38).

The selection procedures support the selection of projects relevant to OPs’ objectives

18

Two measures were introduced for the 2014-2020 programme period legislative framework which are designed to trigger the selection of projects relevant to OPs’ objectives:

  • OPs must set out “guiding principles” for the selection of projects for each priority axis14;
  • MAs must design selection procedures and criteria “that ensure the contribution of operations to the achievement of the specific objectives and results of the relevant priority”15.
19

In addition to these, the following requirements were also introduced for the 2014-2020 period to trigger the selection of more results-oriented projects:

  • under general ex-ante conditionality 7 (see also paragraph 4), “a system of result indicators necessary to select actions which most effectively contribute to desired results” must be in place;
  • MAs must also draw up selection procedures and criteria that “ensure the contribution of operations to the achievement of the […] results of the relevant priority”16.
20

The “guiding principles” act as guidance for project selection, detailing the main selection criteria to be used. Coupled with the description of the projects to be allocated funding, providing examples of such projects, and beneficiaries, they give a good overview of the type of projects to be funded under each specific objective. These principles are used as a basis for preparing the selection criteria, and may be referred to directly within selection criteria.

21

We found that, in some cases, the principles were of a general nature; however, they all referenced the need for selected projects to be in line with the specific objectives of the OP, as described under each priority axis. The selection criteria, for the selection procedures we examined, were consistent with the guiding principles. Each procedure entailed at least one criterion about projects’ relevance and their contribution to the specific objective.

22

Overall, the selected applications we examined were assessed against the criteria defined within the selection procedure. We found that the 34 projects we examined for this audit were relevant to the OP specific objectives.

The focus on results when assessing project applications was weak, and seldom included quantified results

23

We examined how MAs carried out their selection of projects in practice, in particular the existence in the project proposal of clear expected results and objectives accompanied by clearly defined and quantified output and result indicators (linked to the specific objectives). We also assessed whether the MAs had assessed each applicant’s ability to achieve expected results.

24

We found that 18 out of 20 of the selection procedures we examined included criteria requiring beneficiaries to describe their project’s expected results (see Annex VIII). However, only four out of 20 procedures included criteria requiring the quantification of result indicators at project level (selection procedures 17, 18, 19 and 20). And in three cases, these result indicators did not correspond directly to those defined at OP level (see example in Box 1).

Box 1

Example of result indicators at project level different from result indicators at OP level: project No 32, selected through selection procedure No 17

Result indicators at project level

(see Annex I, project No 32)
  • amount of additional turnover generated
  • amount of additional direct exports
  • number of new jobs created
Result indicators at OP level

(see also Annex VI, PA 1, SO 3d)
  • evolution of the business dynamic index
  • number of growing companies
25

Our review of the 34 applications of selected projects corroborates these findings. All the applications provided quantified information for output indicators that corresponded to those at OP level, or such data could be obtained directly from the IT system (for instance for indicators such as number of enterprises receiving support). However, only 14 applications provided quantitative information on expected result indicators. In six of these, the result indicators did not correspond directly to the OP indicators. The other applications mainly provided a narrative description (see Figure 2 and see Annex VIII).

Figure 2

Information provided in the applications upon expected outputs and results

26

Regarding the assessment of applications, we found that the MAs had assessed the overall feasibility and likelihood of the projects to succeed. However, while there was an assessment of the achievability of the expected results for all 18 ERDF projects, this assessment was only performed for 2 of the 16 ESF projects we examined. We also found that the assessment of achievability remained vague in the assessment reports, except in seven cases17 which included a more detailed description about the relevance of the indicators and the likelihood of the targets being met.

27

In terms of scoring the applications, we found that the provision of expected results was part of the scoring process in only 11 cases. This indicates that, as long as the envisaged projects were aligned with the type of actions set out in the selection procedure, the MAs assumed that it would contribute to the OP’s objectives. However, they did not focus on the magnitude of this contribution.

Beneficiaries were encouraged to apply for funding

28

To ensure timely absorption throughout the programming period and at the same time ensure the focus on results, there is a need for a sufficient number of bidders able to provide good applications. The promotion of a call and related guidance to potential applicants by MAs is an important variable determining the overall effectiveness of selection procedures. A broader applicant population is likely to ensure that better projects are selected.

29

The measures aiming at promoting calls to potential bidders were prepared by the MAs on the basis of the detailed requirements set in the EU regulation. They were submitted to the Monitoring Committees for approval as part of the OP’s communication strategy that is required by the Regulation18.

30

We found that MAs had encouraged beneficiaries to participate in the selection procedures. First, MAs used several sources to promote a call to potential bidders. These included the involvement of Chambers of Commerce, local business associations, and online tools. MAs also relied on partners to circulate information on upcoming calls within their networks, based on the knowledge they acquired in the monitoring committees.

31

Furthermore, the guidance on funding provided by MAs to applicants was publicly available and, except for one MA at the time of the audit, it set out clear conditions for receiving support, provided detailed information on the procedure for application and financing arrangements, referring explicitly to the selection criteria to be applied.

Selection procedures did not usually involve any comparison between project applications, resulting in a risk that the best projects will not be selected

32

In our review of procedures, we checked that projects submitted for funding were assessed against each other. This is another important measure to ensure that the most results-oriented projects are selected.

33

In the framework of shared management, selection procedures were defined by the MAs and approved by the monitoring committees, based on the requirements in the EU Regulations19 and the Commission’s guidance20. Neither the ESI Funds Regulations nor any Commission guidance specifies the type of selection procedure to be used; this is left to the MAs’ discretion.

34

Out of the 20 procedures we examined, 10 were temporary calls for proposals, 6 were permanent calls, 3 were direct awarding procedures, and one a reiterated call (see also details in Annex III). We found that in all selection procedures except one (IT, OP Piemonte, No 13) no comparison between project applications had taken place. As a result, funding may not have been allocated to the best projects. This was mitigated to some extent in some of the OPs visited, where an informal (French and Finnish OPs) or a formal (Italy Puglia) pre-selection had taken place, allowing the MAs to limit the submission of applications to the most suitable projects. We found the following:

  • Fifteen out of twenty procedures were based on the first-come first served principle, where an application was selected provided that it met the criteria and that sufficient funding was available. Of these, 13 had been subject to a formal or informal pre-selection. We noted that some had been required to reach a minimum score for selection (see selection procedures Nos 4 and 5 for the Czech OPEIC and 16, 17, 18, 19 and 20 for the Finnish OP).
  • In the three direct awards, there was, by definition, no competition (see selection procedures No 1, 3 and 12).
  • In one case (IT, OP Piemonte, selection procedure No 8)21, funding had been awarded to projects already selected during the previous 2007-2013 programming period simply by extending their duration. This allowed the IB to avoid interruptions to the provision of training to young people, and allowed it to start absorbing the funds. But it also prevented potential new applicants from participating.

The quality of monitoring data is at risk, and it is mostly outputs-oriented

35

We checked that the monitoring systems allowed the systematic collection of data on specified indicators, thus enabling the measurement of the progress made at project, programme and EU levels. We also examined whether this had been reported on in a way that allowed the success of the financed projects, and thus the effective and efficient use of the EU budget, to be demonstrated.

36

The following section contains our assessment of the necessary conditions to enable monitoring and reporting on the achievement of objectives. We first checked whether the MAs had set up functional monitoring systems before starting to implement the funds. We also looked at whether the performance information reported by the beneficiaries had been reliable, complete and timely. We also examined the role of the MAs and the Commission.

37

Finally, we consider whether the systems in place produced results-oriented performance information; specifically, whether they produced meaningful aggregated information, as this is a key element to allow reporting on the effective use of funds at Member State and EU level via annual monitoring and reporting.

Assurance on the quality of monitoring data may not be provided in time for the 2019 performance review

38

The monitoring of an OP’s progress towards its objectives is done on the basis of the data collected from various sources. This data is aggregated at OP and Member State level, and forms the basis for the reports sent to the OP’s monitoring committee and to the Commission. In order to present a correct view of the situation at OP or Member State level, it is essential that the data collected is reliable, complete and timely. To achieve this, the MAs have to set up a monitoring system including the collection of data and checks on the source of the data. In order to gain assurance on the quality of the data, the Commission relies on the audit work carried out by the audit authorities at Member State level as well as its own audit work.

39

In the specific context of the performance review, it is also important that the data, and the systems used to produce them, be audited as early as possible, to allow Member State authorities to address any detected weaknesses regarding the data quality well before the end of June 2019. It is then that the 2018 Annual Implementation Reports (AIRs) – the basis for the performance review and the allocation of the performance reserve – will be sent to the Commission.

Extensive EU rules lay down how Member States should set up their monitoring systems for the 2014-2020 period
40

With a view to achieving the systematic collection and monitoring of data produced at OP level, the regulatory framework for the 2014-2020 programme period introduced several changes concerning data collection and monitoring arrangements:

  • MAs must put in place a management and control system (MCS) covering adequate systems for reporting and monitoring22.
  • MAs are responsible for establishing a computerised system to record all the data related to the OPs indicators23 for monitoring and reporting purposes.
  • Under EAC 7, MAs must put in place “a system of result indicators necessary to select actions which most effectively contribute to desired results, to monitor progress towards [those] results”24. OPs were required to demonstrate fulfilment of all the ex-ante conditionalities (EAC) by 31 December 201625 (see paragraph 4).
  • By 31 December 201526, Member States were required to ensure that all exchanges of information between beneficiaries and a MA, Certifying Authorities (CA), AAs and IBs could be carried out electronically27.
41

The Commission has also established specific guidance28, for its staff and the Member States’ responsible authorities, on a common methodology for assessing management and control systems in the MS, including the existence of adequate systems for monitoring and reporting.

Performance information is not always reliable, complete and timely, and this has an impact on the implementation reports issued by the Member States
Data reported by the beneficiaries
42

For the ERDF, in many cases, the data for result indicators is collected from national statistics bodies (see examples in Annex VI). The remainder comes from beneficiaries. For ESF, most of the data is reported via data collected from beneficiaries (see Annex VII). Managing authorities check that the data collected from beneficiaries is correct at project completion, when receiving their final report. During project implementation, checks can also be done on-the-spot, on a sample basis.

43

A number of plausibility and consistency checks are carried out automatically by the systems in place on the data provided by the beneficiaries, especially for the ESF. MAs also have tools to monitor the completion of the data provided on participants which allow them to remind beneficiaries who have submitted incomplete data to provide further information.

44

Our audit revealed issues regarding the quality of the data collected. We identified a general issue regarding the completeness and correctness of the data concerning participants to ESF co-financed actions. This data is classified as sensitive in national legislation29, so it is up to participants to decide whether to declare it. It is also difficult for the Member States and the Commission to check its accuracy. An additional weakness regarding the collection of data is presented in Box 2.

Box 2

Example

Issues regarding timeliness of the data

For the French national ESF OP, larger beneficiaries submitted their data for the 2014-2015 period late. This was corrected in 2016, and resulted in significant differences between the values reported in the 2015 AIR and those reported in the 2016 AIR for the year 2015.

45

In its audit of performance data reliability (2014-2020), the Commission reported on other issues regarding the reliability of data, for instance:

  • data not collected which resulted in achievements not being recorded and thus not being taken into account in order to determine the progress of an OP, leading to an incorrect understanding of the situation;
  • a misunderstanding on the MAs’ part regarding the definition of common indicators (for instance, Common Output indicator 26, 'Enterprises cooperating with research organisations') resulting in an overstatement of the number of companies reported under this indicator;
  • double counting/failure to cap the value for indicator 'population served' at the number of area residents, resulting in inconsistent values.
Availability and reliability of performance information
46

The existence of functional monitoring systems at the time MAs start implementing programmes is a key element ensuring that performance data is generated in time, which ensures that monitoring can take place.

47

The late approval, in December 2013, of the legislative framework for the period 2014-2020 at EU level, delayed the development of the IT systems used for monitoring. For the ESF indicators, added complexity concerning the sensitivity of the data and the definition of the indicators, required additional discussions well into 2014. This further delayed the finalisation of the IT systems. In two cases, other factors led to the delays: reorganisation of the regions (France, OP Lorraine et Vosges), and the design of a new single IT system for all 2014-2020 ESI funds in the Czech Republic.

48

The effects of the delay are reflected in the level of implementation of EAC 7 on the existence of a system of results indicators. At the end of 2016, the deadline to complete action plans, 50 plans remained incomplete, i.e. around 11 % of all OPs still did not fulfil this requirement. However, by the end of February 2018, all the action plans related to EAC 7 had been completed.

49

In the first half of 2017, the fourth year after the start of the programme period, we found that some IT systems were still not fully functional; this poses risks to the completeness and accuracy of the data collected (see examples in Box 3). In its audit of performance data reliability (2014-2020)30, the Commission drew attention to the absence of electronic data exchange systems with beneficiaries, resulting in the manual encoding of the data on indicators by the MA into the system, a potential source of clerical errors.

Box 3

Examples of IT systems still not fully functional and related risks

Czech Republic - OPE

  • Problems with transferring data between the Labour Office and the OP. Data only started to be reported electronically to the OP from 31.3.2017. Until then, data was shared via Microsoft Excel files. This jeopardises the transfer of data between systems, especially given the large amount of data involved.
  • The AIR 2016 was the first AIR containing relevant data.

The MA discovered inconsistencies in the data reported by the system for the AIR. This required to recalculate and fill them in manually in the AIR.

Italy - OP Puglia

The tools used for exchanging data with beneficiaries electronically were not working properly at the time of the audit31, and the system for collecting and storing the data was empty. Data was merely stored on staff computers. As a result, the aggregation of data with a view to producing the 2016 AIR (to be submitted to the Commission by 30 June 2017) was done manually, in separate files.

Implementation reports issued by the Member States
50

The MAs must submit to the Commission an Annual Implementation Report (AIR) for each OP, which is approved by the Monitoring Committee. The AIR delivers key information on the implementation of the OP, mostly by providing quantitative and qualitative information on common and programme-specific indicators32. The 2016 AIRs, submitted in June 2017, were the first reports sent to the Commission on the “progress made towards achieving the objectives of the programme”33. The Commission carries out a quality review of the AIRs (consistency of number of participants with population of the areas covered, for instance).

51

At the end of 2017, 26 AIRs had still not been accepted by the Commission due to missing data for the indicators used in the performance framework, inconsistencies in the data reported with the information provided on the implementation of the OP, or a lack of explanation for some of the data reported. Around 40 % of ERDF OPs had not reported any values for their performance framework indicators in 2017, on which there was no Commission analysis at the time of the audit.

52

The shortcomings identified in the AIRs raise questions about the reliability of the information provided as well as the level of progress reached until now as presented in the documents issued by the Commission based on the AIRs34: the annual synthesis report to the Parliament and the Council, European Economic and Social Committee and the Committee of the Regions and, in 2017 and 2019 only, the strategic progress report, which summarises progress made.

Audits on management and control systems and the reliability of collected data started late, limiting the degree of assurance on the monitoring systems and reducing the time available to address detected weaknesses
53

Appropriate arrangements for the verification and correction of the performance information are necessary to ensure its quality and hence, its use in the context of monitoring and reporting. Good practices requires audits on the systems producing the data - and associated corrective actions - to be carried out and finalised before the data is included in AIRs and strategic reports.

54

Under the legislative framework, both the Member States’ audit authorities and the Commission35 audit the functioning of the monitoring systems. These audits should, in particular, ensure the existence of reliable systems for collecting, recording and storing performance data, and assess whether this performance data is reliable.

Audit authorities’ assessment of the functioning of the management and control systems
55

An assessment of the systems in place at MA level was carried out during the early stages of the implementation of the OPs as part of the designation procedure36. In this procedure, an independent audit body (often the AA) was required to produce a report and an opinion on whether the MAs had fulfilled the criteria relating to the internal control environment, risk management, management and control activities and monitoring37. The designation was required to take place before the first submission of an interim payment for the OPs. Thus, without this designation, OPs cannot be reimbursed. At this stage, the audit covers the existence of systems and procedures, but it does not cover good implementation, which is checked in the context of the audit authorities’ system audits on management and control systems.

56

Due to the late approval of the OPs and late finalisation of the IT systems (see paragraph 47), the designation process was late. This was also affected by national factors (reorganisation of the regions in France in 2015, complex management structures - for instance, the French PON FSE has 13 delegated managing authorities and 120 intermediary bodies). For the OPs examined in this audit, the MAs were designated between September 2016 and February 2018, more than two and four years into the programme period respectively (see Annex IV).

57

Due to the late implementation of OPs and the resulting limited amount of data collected, systems audits were only beginning at the time of our audit visits i.e. during the first half of 2017. Any problems regarding the quality of data thus did not become apparent until the second half of 2017. This was almost four years after the start of the seven-year programme period and one-and-a-half years before the 2018 AIRs, which form the basis for the allocation of the performance reserve, are forwarded to the Commission.

The Commission’s audits
58

In addition to relying on the audit work and systems in place in the Member States, the Commission also carries out its own audits. There are two types: “early preventive systems audits”, aimed at obtaining reasonable assurance at an early stage of the implementation that the management and control systems are functioning well, and “performance reliability data audits”. But given the aforementioned delays, and thus the late finalisation of the monitoring systems as well as the minimal reporting of relevant data, these audits were conducted later than planned. Especially for the ERDF, audits could only start after the 2016 AIRs were received. In total, by the end of 2017, the Commission managed to carry out 23 audits in 12 Member States: 12 performance reliability data audits and 11 early preventive systems audits.

59

For most of these audits, reporting was ongoing at the beginning of 2018. According to the Commission, the preliminary results for the audited systems were rather positive, even though some shortcomings were highlighted. At the same time, however, around 40 % of ERDF OPs did not report any values for their performance framework indicators38 in the 2016 AIRs submitted in 2017. For these OPs, in February 2018, the Commission had not carried out audits to determine whether the absence of reporting was due to delays in implementation or a lack of a functioning data-reporting system.

60

The late undertaking of the systems audits means that the overview of the monitoring systems’s status is not complete. Shortcomings are particularly susceptible to being identified late and there may not be sufficient time to apply the necessary adjustments. This jeopardises the implementation of the performance review in 2019, which will be based on data reported by Member States in the 2018 AIRs by 30 June 2019.

The performance review may not be based on realistic milestones
61

In the OPs we visited, on the basis of the 2015 AIRs or in the monitoring information available at the time of the audit, we noted that there were already cases where OPs had achieved the targets set for 2023, as well as risks of underachievement. Requests for amendments in order to adjust the targets to a more realistic value, had not been submitted for the OPs visited. Only one (Czech Republic, OPEIC) had already asked for, and obtained, amendments, mainly in relation to the modification of allocations within priority axes. On the other hand, according to the Commission, in February 2018, out of the 17 480 indicators used in the existing 430 OPs, the corresponding target value for 2023 had been modified upwards in 1445 instances and downwards in 941 instances.

62

This indicates that a number of milestones39 of the performance framework are no longer realistic. In case milestones need to be revised, this requires that amendments be made to the OP, which can only be done at the Member States initiative40. Given the duration of the amendment procedure (set in the Regulation at three months), there is a risk that the values reached at the end of 2018 by the performance-framework indicators are not compared with realistic milestones.

The information generated on the basis of the data collected is still mostly outputs-oriented

63

Performance information is collected so that the Member States and the Commission can use it for reporting purposes, in order to be accountable for what has been achieved with funding.

64

Result indicators and targets laid down in the grant agreements signed between beneficiaries and MAs are essential for measuring the achievement of results. However, several key issues in the set-up of the monitoring systems are affecting the measurement and monitoring of the contribution of projects to the achievement of the OP objectives.

A set-up which relies mostly on output indicators, especially in the early years of implementation for the ERDF, and a lack of quantified result indicators defined at project level
65

The current set-up allows for a more systematic collection of data for a number of well-defined indicators, though the number of indicators is very high. The existence of common indicators also allows for the aggregation of this data at OP, Member State and EU levels. However, in the set-up of the logical framework for the ERDF, many result indicators at OP level are not directly related to the funded interventions. These indicators are often national indicators which take into account the influence of external factors, and their sources are mainly national statistics offices (see examples in Box 4 and see also Annex VI). These result indicators do not isolate the impact attributable to the ERDF interventions, and cannot therefore be considered immediate results of the OPs.

Box 4

Examples of result indicators for which the data is collected from published statistics rather than funded projects

France, OP Grand Est:
  • Priority axis 2; TO 3, investment priority A: development of SMEs:
Specific result indicator: evolution of the region’s industrial SME investment
Source: Banque de France.
Czech Republic, OPEIC:
  • Priority axis 2; TO 3, investment priority 3b: increase in exports of SMEs:
Specific result indicator: percentage of exports in the total turnover of companies
Source: Ministry of Industry and Trade.
Finland, OP Sustainable growth and work:
  • Priority axis 1, TO 3, investment priority 3d:
Specific result indicator: Business dynamics index
Source: National statistics office of Finland.
66

To mitigate the fact that results generated by ERDF projects often materialise (well) after project completion, at a time when there is no monitoring of the projects as such, MAs must carry out evaluations at priority axis level41. At least once during the programming period, these evaluations assess how support from the ESI Funds has contributed to the objectives for each priority. But given the late start of the implementation, the number of evaluations already carried out is still limited. As a consequence, for the time being, the progress of the OP is mainly measured through the output indicators.

67

At project level, we found that result indicators were not systematically mentioned in the grant agreements (see paragraph 25). Only six42 out of the 18 ERDF projects examined in this audit contained results indicators in the applications and thus in the grant agreement. However, these result indicators were not the same as those used at OP level. This means that they cannot be used for reporting purposes or to measure the progress made in achieving OP objectives. In four other cases43, the results indicators were not quantified (i.e. no target was given) thus preventing an assessment of their contribution to the OP objective from being made and limiting the measurement of the project’s achievement to its outputs.

68

For the ESF projects, result indicators were mentioned in 7 out of 16 projects. For the remaining projects, the results described were either not related to the OP indicators, or they were not quantified. In any given project, this makes it impossible to compare the value reached with an estimated target. However, in the case of the ESF, even if results and result indicators are not defined at project level or quantified, the measurement of the values for the common result indicators is still taking place, as project beneficiaries have to report those data in the MA’s IT systems.

69

We have previously mentioned the lack of result indicators defined at project level44 in our annual reports. This makes it difficult to measure and to monitor the extent to which projects contribute to the achievement of the OP objectives.

The release of the performance reserve is based on indicators that are mostly input- and outputs-oriented
70

Member States can demonstrate progress in programme implementation through milestones (to be reached by 2018) and target values (to be reached by 2023) for three types of indicators: financial, output and result. They can be complemented by key implementation steps (KIS) to measure projects that are on-going or scheduled to be started, but for which it is unlikely that any outputs can be achieved by December 2018. Overall, the indicators used in the performance framework must be representative of the actions under the priority axis. In order to trigger the release of the performance reserve, the values reached at the end of 2018 for these indicators must be between 75 % and 85 % of the milestone value.

71

As shown in a previous ECA report45, the vast majority of the indicators used in the performance framework under the cohesion policy related to output (57 %), key implementation steps (9 %) and financial indicators (33 %), with the use of result indicators remaining marginal. Thus, the release of the reserve will to a large extent remain input- and outputs-oriented, and not focused on results (see also paragraph 62).

Most reporting at EU level concerns output indicators
72

The main objective of the Commission’s strategic report 201746, which encompasses the 2017 annual synthesis report drawn up based on the 2016 AIRs, is to report on the achievements of the ESI Funds towards attaining the EU2020 objectives. However, the report mainly presents the level of implementation as well as the values reached for the main output indicators by the end of 2016. It presents no information on the achievement of results, except for the following ESF result indicators: the number of participants that found a job immediately after the training, and the number of participants that gained a qualification immediately after training. For the ERDF, this is also due to the fact that there are no common result indicators; those used at Member State level cannot, therefore, be aggregated.

73

As we have reported recently, there is a fundamental problem affecting result indicators in particular. In fact, ERDF and ESF result indicators measure different things in different ways47. Furthermore, the concept of a “result” is interpreted differently in the fund-specific regulations. The ERDF result indicators cannot be used to measure the immediate results of the OPs. In addition, the high number of different performance indicators pose a real challenge to the collection and reporting of performance information48.

Conclusions and recommendations

74

Although the set-up of OPs is now more results-oriented, with a stronger intervention logic and an extensive set of indicators, overall we conclude that the selection of projects under the ERDF and ESF is not fully results-oriented and that the monitoring in place is still more outputs-oriented.

Selection procedures were not results-oriented in important aspects

75

For the OPs we visited, selection procedures were used to support the selection of projects relevant to the OPs’ objectives. However, they do not systematically include selection criteria requiring the definition of quantified result indicators at project level corresponding to those at OP level. Consequently, result indicators were rarely included in applications; even when this was the case, they did not necessarily correspond to the OP indicators, or had not been quantified. Furthermore, only one-third of the assessment reports included a specific assessment of the expected results in relation to OP objectives (see paragraphs 18 to 27).

76

We found that MAs had provided comprehensive information on the OPs, on access to EU funding, and on the selection procedures through several channels. Beneficiaries were generally well informed, or could rely on support from specialised organisations (see paragraphs 28 to 31).

77

However, only 1 of the 20 procedures examined involved a scoring and a ranking of applications against each other. Projects were generally approved on a first-come first-served nature. As a result, funding may not have been allocated to the best projects (see paragraphs 32 to 34).

There are risks to the quality of monitoring data, which remains more outputs-oriented rather than results-oriented

78

Following the late adoption of the legislative framework, we found that some of the OPs’ monitoring systems were not operational at the start of the programme nor fully operational at the time of the audit. Consequently, data was input manually and at a later stage. Data collection also occurred outside the IT systems, creating errors, which, in some cases, impacted upon the data presented in the AIRs (see paragraphs 35 to 52).

79

The late adoption of the regulatory framework led, in most OPs, to a late start of the implementation. Until the end of 2016, only a limited amount of data was collected in IT systems. Consequently, the audits necessary to provide assurance on the monitoring systems and the reliability and availability of the data mostly started in 2017. The results of these audits only became available at the end of 2017, four years into the seven-year programme period. This means that assurance obtained upon the monitoring data is, up to now, only partial. The Commission thus does not have an overview of the functioning of the monitoring systems, including IT systems. The performance review will take place in 2019 and, in the remaining time available, there is a risk that any corrective measures cannot be completed in time and that the performance review will not be based on correct information (see paragraphs 53 to 60).

80

We found that a number of milestones of the performance framework are no longer realistic. In case milestones need to be revised, there is no certainty that the necessary OP amendments can be made in time for the performance review (see paragraphs 61 and 62).

81

Regarding result indicators, ERDF and ESF indicators measure different things in different ways. In fact, the concept of a “result” is interpreted differently in the fund-specific regulations. For both funds, we identified the risk that a meaningful aggregation of performance data (in particular regarding results) may not be feasible at EU level49. Furthermore, we recommended50 that the Commission should define a common definition of “outputs” and “results” (see paragraph 73).

82

For the ERDF, many result indicators often use other sources, such as data from national statistics offices. Evaluations should measure achievements against the OPs’ objectives. However, given the late start of implementation, few evaluations were available at the time of our audit. Consequently, for the ERDF, it is difficult to monitor to what extent the projects contribute to the achievement of the OP objectives (see paragraphs 63 to 69). This problem also applies to the ESF, as many grant agreements do not include quantified result indicators.

83

Similarly, output and financial indicators predominantly form the basis of the performance framework. This means that the release of the performance reserve will be mainly based on these types of indicators rather than on the results achieved and the actual progress made towards the OP objectives. Also, the information published in the Commission’s reporting is mostly outputs-related, except for the ESF where the aggregated value for some result indicators is provided (see paragraphs 70 to 72).

Recommendation 1 – Results-oriented selection

In order to ensure a consistent and truly results-oriented approach to the selection of projects, Member States should:

  1. have selection criteria requiring the beneficiaries to define at least one genuine result indicator based on a common definition of what constitutes a “result” for their project, including a baseline and a target. This result indicator should contribute to the result indicators set at OP level;
  2. include an assessment of these expected results and indicators in the assessment report for the applications;
  3. when deciding on the selection procedures to be used, ensure that a comparison between project applications takes place.

Target implementation date: starting in 2019 for upcoming calls for proposals.

Recommendation 2 - Results-oriented monitoring

  1. In order to allow the monitoring of projects’ contributions to OP objectives, Member States should include quantified result indicators in the grant agreement, contributing to the results indicators set at OP level.

    Target implementation date: starting in 2019 for upcoming calls for proposals.

  2. In order to make the ERDF monitoring framework more results-oriented, and in particular to allow the aggregation of performance information, the Commission should define common result indicators for this fund based on a common definition of “results”.

Target implementation date: preparation of the next MFF.

Recommendation 3 - Reporting on performance and preparation of the performance review in 2019

To improve the Commission’s reporting on performance and to carry out a meaningful performance review with reliable data and realistic milestones, the Commission should:

  1. have an overview of the main weaknesses in and uncertainties about the OPs’ monitoring systems based on audits by both the Commission and the AAs (the necessary systems audits/performance-reliability audits);
  2. ensure that OP amendments requested by Member States relating to justified revisions of milestones of the performance framework are processed in time for the performance review.

Target implementation date: before the finalisation of the performance review.

This Report was adopted by Chamber II, headed by Mrs Iliana IVANOVA, Member of the Court of Auditors, in Luxembourg at its meeting of 27 June 2018.

For the Court of Auditors

Klaus-Heiner LEHNE
President

Annexes

Annex I

List of projects selected for this audit

(amounts in million euros)

No Project Description Grants received previously Total amount of investment Amount of EU Grant Fund TO (a) PA (b) IP/SO (c) Start & end date Outputs and results achieved Selection procedure No (see Annex III)
Czech Republic - OP Employment (OPE) 2014CZ05M9OP001 (amounts in million CZK)
1 ”New working opportunities” (Call Instruments of Active labour market policy) Yes 4 011 3 304.6 ESF 8 1.1 1.1.1 1.4.2015 ongoing Participants in employment: 14 304 1
2 Child care centre for preschool children #1 (Call Support of creation and operation of day care services for children of pre-school age for companies and public outside the City of Prague) Yes 4.9 4.2 ESF 8 1.2 1.2.1 1.1.2016 - 31.12.2017 Places in kindergarten: 24 2
3 Child care centre for preschool children #2(a) (Call Support of creation and operation of day care services for children of pre-school age for companies and public outside the City of Prague) Yes 4.1 3.4 ESF 8 1.2 1.2.1 1.1.2016 - 31.12.2017 Places in kindergarten: 20 2
4 Child care centre for preschool children #2(b) (Call Support of creation and operation of day care services for children of pre-school age for companies and public outside the City of Prague) Yes 4.9 4.2 ESF 8 1.2 1.2.1 1.1.2016 - ongoing Places in kindergarten: 20
Number of persons using a childcare facilities: 30
2
5 “Support of selected social services (asylum houses, daily centres, etc) in one region” Yes 411.8 350 ESF 9 2.1 2.1.1 1.1.2016 - ongoing Actual realisation: NA yet, project in start up phase 3
Czech Republic - OP Entrepreneurship and Innovation for Competitiveness (OPEIC) 2014CZ16RFOP001 (amounts in million CZK)
6 Support to exports for the companies Yes 1.2 0.6 ERDF 3 2.1 3b
SO 2.2
3.7.2015 - 10.5.2016 Number of participations in exhibitions and fairs abroad 3 (but the same company): one company supported
Grants previously received:
12 projects OPPI 265.2 million CZK,
4 Fair grants 53 million CZK
4
7 Participations in trade fairs abroad in 2016 Yes 3.3 1.6 ERDF 3 2.1 3 b
SO 2.2
9.9.2015 - 21.12.2016 Number of participations in exhibitions and fairs abroad 3 (but the same company): one company supported
Grants previously received:
8 projects from OPPI 117.3 million CZK
1 OPLZZ 5.3 million CZK
4
Czech Republic - OP Entrepreneurship and Innovation for Competitiveness (OPEIC) 2014CZ16RFOP001 (amounts in million CZK)
8 Building of a production hall for woodworking division Yes 8.6 3.8 ERDF 3 2.1 3 c
SO 2.2
1.11.2015 - 25.1.2016 Revitalisation of the company’s site (844 mq)
Number of companies using the revitalised infrastructure: 1
Grants previously received:
7.2 million CZK
5
9 Reconstruction of a Building Yes 21.2 7.4 ERDF 3 2.1 3 c
SO 2.2
7.9.2015 - 6.9.2016 Revitalisation of the company’s site (2721 mq)
Number of companies using the revitalised infrastructure: 1
Grants previously received:
OPPI 12 projects 123.7 million CZK,
OPLZZ project 2.2 million CZK
5
France - OP ERDF/ESF Lorraine et Vosges - 2014FR16M0OP015
10 Take over a printing activity (due to retirement of the previous owner) No 2 0.04 ERDF 3 2 2.3A 18.9.2014 - 12.10.2015 Number of enterprises receiving support: 1
Jobs: + 2
6
11 Acquisition of equipment for production (e.g.: presse 140 tons with bi-injection system) No 0.6 0.05 ERDF 3 2 2.3A 1.1.2014 - 31.12.2016 Number of enterprises receiving support: 1
Jobs: + 2
6
12 Investments for the development of a ski resort No 6.7 1.1 ERDF 3 9 9.3A 1.1.2015 - 31.12.2016 Number of enterprises receiving support: 1
Number of enterprises receiving subsidy: 1
7
13 Acquisition of equipment for production (hydraulic tools) No 1.6 0.02 ERDF 3 2 2.3A 2.7.2014 - 31.2.2016 Number of enterprises receiving support: 1
Number of enterprises receiving subsidy: 1
Jobs: + 6
6
France - National OP ESF (PON FSE) - 2014FR05SFOP001
14 Support to people benefitting from the minimum income in the department Yes 10.7 5.3 ESF 3 9 1.1 1.1.2014 - 31.12.2016 Number of participants expected: 6360 / year or 19 080 in total 8
15 Accompany/ guiding people in an insertion process, including learning French language Yes 2 1 ESF 3 9 1.1 1.1.2014 - 31.12.2016 Number of participants to the action (2014-2016): 657
Positive outcome within 3 months after the project participation: 40 %
9
France - National OP ESF (PON FSE) - 2014FR05SFOP001
16 Professional training of employees from the construction sector (Paris and neighbouring region) Yes 9.5 4.8 ESF 2 8 5.3 1.1.2014 - 31.12.2015 Number of participants to the action (2014-2016): 2 464 10
17 Professionalization of networks about the creation of activities Yes 5.2 2.6 ESF 1 8 3.2 1.1.2014 - 31.12.2016 No participants 11
18 Training activities for unemployed people Yes 80 30.7 ESF 2 8 5.4 1.1.2014 - 31.12.2016 Number of participants to the training: 21 860 (targeted: 22 734) 12
Italy - OP ESF Piemonte - 2014IT05SFOP013
19 Training courses to help insertion in the labour market (Technical e-commerce) Yes 0.69 0.69 ESF 8 1 8.x 1.10.2015 - 15.6.2016 Technical e-commerce course (600 hours) 13
20 Training courses to help insertion in the labour market (hairdresser) Yes 0.69 0.69 ESF 8 1 8.x 1.10.2015 - 15.06.2016 Hairdresser course (600 hours) 13
21 Vocational training to back up compulsory education (Carpenter) Yes 0.99 0.99 ESF 8 2 8.x 1.10.2015 - 15.06.2016 Carpenter course (1 050 hours) 14
22 Vocational training to back up compulsory education (Agro-food processor) Yes 0.99 0.99 ESF 8 2 8.x 1.10.2015 - 15.6.2016 Agro-food course (1 050 hours) 14
Italy - OP ERDF/ESF Puglia - 2014IT16M2OP002
23 Acquisitions of 10 new buses for passenger transport Yes 4.12 1.41 ERDF 3 3 3a 5.5.2015 - 31.12.2015 none 15
24 Acquisition of equipment for production of gluten-free pasta Yes 2.7 0.8 ERDF 3 3 3a 16.6.2015 - 31.12.2015 - Turnover (2017 o/s)
- Increase 4 work units
15
Italy - OP ERDF/ESF Puglia - 2014IT16M2OP002
25 Restructuring and adaptation (building) of an existing school into a crèche Yes 0.5 0.44 ERDF 9 9 9a 1.1.2016 - 31.12.2016 57 new places created in crèche 16
26 Restructuring and adaptation (building) of an existing school into a crèche Yes 0.35 0.31 ERDF 9 9 9a 1.1.2016 - 31.12.2016 42 new places created in crèche 16
Finland - OP Sustainable growth and work - Finland’s structural funds programme - 2014FI16M2OP001
27 Investment in a powder painting line, acoustic wall element line and steel bending machine. Production reorganized according to LEAN Yes 0.85 0.32 ERDF 3 1 3d 2.1 16.2.2015 - 30.6.2016 Turnover: + €0.3 million, + 10 % of current one
Direct export: + €0.07 million, + 100 % from the current level
Jobs: + 2 (of which 0 are women’s jobs)
Other: Significant energy savings per produced unit, at least 10 % of energy use
17
28 Investment for undertaking concrete production line development, internal logistics re-organisation as well as develop a new range of products Yes 2.45 0.619 ERDF 3 1 3d 2.1 2.3.2015 - 30.9.2016 Right after completion:
Turnover: + €0.41 million
Jobs: + 1
2 years after completion (not known yet):
Turnover: + €2.5 million, + 20 % of current turnover
Jobs: + 7
Project contributes to low carbon economy
17
29 To develop and confirm the service model used in one city’s Labour Service Center for young adults that have a background of substance abuse to encourage them to normalize their life so that they may envisage to start looking for a job Yes 0.24 0.17 ESF 9 5 8i 2.1.2015 -31.12.2016 Beneficiaries: 39 persons (target: 80 persons) 18
Finland - OP Sustainable growth and work - Finland’s structural funds programme - 2014FI16M2OP001
30 To offer training for people employed intermittently to increase their chances of finding jobs and their choice of jobs (social economy) Yes 0.31 0.22 ESF 8 3 9i 1.1.2015 - 31.12.2016 Beneficiaries: 82 persons (target 100 persons) 18
31 Development of a user interface software for medical ultrasound scanner Yes 0.34 0.17 ERDF 3 1 3a 1.1 30.1.2015 - 30.4.2016 At project completion / estimate 2 years after:
Turnover:+ €0.25 million + 100 % / + €10 million + 100 %
Direct exports: + € 0.25 million + 100 % / + €9.8 million + 100 %
Jobs: + 3 / 15 (of which women 0 / 5) (target 35) Jobs in R&D: + 3 / 10 (of which women 0 / 3)
Other intangible rights 0 / 3 (target 4)
17
32 Development of new wooden cladding products coating line Yes 1.4 0.56 ERDF 3 1 3d 2.1 27.1.2015 - 27.1.2017 At project completion / estimated 2 years after completion:
Turnover: + €1 million +12 % / + €2 million + 24 % (target €3 million)
Direct exports: NA / + €0.3 million + 33 % (target €1 million)
Jobs: + 3 / + 6 of which female: + 1 / + 2 (target 2)
17
Finland - OP Sustainable growth and work - Finland’s structural funds programme - 2014FI16M2OP001
33 Extension of a factory for the development of production and logistics operations Yes 1.17 0.35 ERDF 3 1 3d 2.1 1.6.2014 - 31.12.2014 At project completion / estimated 2 years after completion:
Turnover: + €10 million + 150 % (target €11 million) / + €35 million + 350 %
Direct exports: + €9 million + 643 % (target € 0.9 million)/ + €26 million + 999 %)
Jobs: + 10 / + 50 of which women + 3 / + 5 (target 50)
R&D jobs: + 2 /+ 2 of which women + 0/ + 1 (target 4)
Other intangible rights 1 / 3
Project contributes to low carbon economy
17
34 Extension of the machinery park for modern technology for piping and steel structure production and re-organise the lay-out of the production site Yes 1.43 0.5 ERDF 3 1 3d 2.1 13.2.2015 – 31.12.2016 At project completion / estimated 2 years after completion:
Turnover: 0 / + €1.4 million + 34 %
Direct exports: 0 / + €0.46 million + 136 % (target €0.75 million)
Jobs: + 5 / + 8 of which female 0 / 0
Other : energy savings NA / + 1 MWh/a
17

Notes: (a) TO: thematic objective. (b) PA: priority axis. (c) IP: investment priority and SO: specific objective.

Annex II

Thematic objectives

Number Description
1 Strengthening research, technological development and innovation
2 Enhancing access to, and use of quality of, information and communication technologies
3 Enhancing the competitiveness of SMEs
4 Supporting the shift towards a low-carbon economy in all sectors
5 Promoting climate change adaptation, risk prevention and management
6 Preserving and protecting the environment and promoting resource efficiency
7 Promoting sustainable transport and removing bottlenecks in key network infrastructures
8 Promoting sustainable and quality employment and supporting labour mobility
9 Promoting social inclusion, combating poverty and any discrimination
10 Investing in education, training and vocational training for skills and lifelong learning
11 Enhancing institutional capacity of public authorities and stakeholders and an efficient public administration

Source: Article 9 and Annex XI Part I of Regulation (EU) No 1303/2013.

Annex III

List of selection procedures examined in this audit

No Name / identification Remarks Fund TO (a) PA/IP/SO (b) Type of selection procedure (c) Competitive 1st in 1st serve Projects audited (see Annex II)
Czech Republic OPE
1 Instruments of Active labour market policy
(No 03_15_001)
Selection of actions to increase employment of supported persons, especially older, low-qualified and disadvantaged ones
The beneficiary (CZ’s Labour office) set in the Partnership Agreement
ESF 8 1.1.1 Direct award
29.7.2015 - 30.4.2016
  NA 1
2 Childcare services outside the City of Prague
(No 03_15_035)
To decrease the differences in the situation of women and men
Support of creation and operation of day care services for children of pre-school age for companies and public outside the City of Prague (“Childcare services”)
ESF 8 1.2.1 Temporary call for projects: 11.11.2015 - 8.1.2016   YES 2, 3, 4
3 “Social services”
(No 03_15_005)
2.1.1 - To increase employability of persons in danger of social exclusion or socially excluded persons in the labour market and
2.1.2 - Development of social economy sector
Support of selected social services linked to the regional mid-term plans of development of social services (“Social services”)
The beneficiaries (all CZ’s regions) as well as the envelope per region, were set in the Partnership Agreement
ESF 9 2.1.1
2.1.2
Direct award launched: 26.6.2015
Applications accepted between: 20.7.2015 - 13.12.2019
  NA 5
Czech Republic OPEIC
4 Marketing I Objective SO 2.2: To increase internationalization of SMEs
- Services for SMEs focused on international competitiveness enabling entry to foreign markets;
- Sophisticated consultancy of experts in international markets and consultancy for strategic management and innovation management, incl. mentoring and coaching;
- Services to support SME networking in international research cooperation (Horizon 2020, COSME).
ERDF 3 PA2, SO 2.2 Temporary call
2.6.2015 - 30.11.2015
  YES provided that the applications score a minimum of 60 points out of 100 6, 7
5 Real Estate I Objective SO 2.3: To increase the use of entrepreneurial infrastructure
- Modernization of production facilities and reconstruction of existing obsolete infrastructure, and
- Reconstruction of brownfields (excluding costs for the removal of ecological burdens) and their transformation into modern business premises and newly reconstructed areas.
ERDF 3 PA2, SO 2.3 Temporary call in two phases:
1.6. till 31.8.2015
Eligible applications only:
1.12.2015 till 31.1.2016
  YES provided that the applications score a minimum of 60 points out of 100 8, 9
France OP Lorraine et Vosges
6 Entreprenariat et entreprises Objective: To select projects with a view to support the investment in SMEs, at all stages of their development, to generate sustainable jobs and added-value ERDF 3 PA 2 - SO 2.3.A Permanent call
1 year revolving
  YES provided that the criteria are met 10, 11, 13
7 Développement économique et touristique du massif Objective: to increase the number of tourists in the Vosges area ERDF 3 PA 9 - SO 9.3.A Permanent call,
1 year revolving
  YES provided that the criteria are met 12
France PON FSE
8 Call for project Call launched by the Intermediate Body ESF 9 PA3 - 3.9.1.1 Temporary call 16.12.2014 - 31.3.2015   YES provided that the criteria are met 14
9 Call for projects - AAP interne 2014 - 2017 Call launched by the Intermediate Body ESF 9 PA3 - 3.9.1.1 Temporary call
9.6.2015 - 30.7.2015
  YES provided that the criteria are met 15
10 Call for project - 2014IDF-AXE2-01 Call launched by the Delegated Managing Authority ESF 8 PA2 - 2.8.5.3 Temporary call
11.8.2014 - 17.11.2014
  YES provided that the criteria are met 16
11 Call for project Call launched by the Intermediate Body ESF 8 PA1 - 1.8.3.2 Permanent call revolving
1.1.2014 - 31.12.2016
  YES provided that the criteria are met 17
12 Call for projects - CSP 2014/15 Call launched by the Managing Authority ESF 8 PA2 - 2.8.5.4 Direct award revolving
1.1.2014 - 31.12.2017
  YES provided that the criteria are met 18
Italy OP Piemonte  
13 “Mercato del Lavoro” in the region Piedmont (except Turin province) To select trainings to shorten the access to the labour market of mainly unemployed youths and adults (19-25 years sometimes older) with low levels of education and exposed to a range of factors that put them at a greater risk of long-term unemployment (Formazione professionale finalizzata alla lotta contro la disoccupazione)
Call managed by the Managing Authority
ESF 8 PA1
SO: 1.8.i.1.3
Temporary call
10.7.2015 - 29.7.2015
YES scoring and ranking   19, 20
14 “Obbligo d’Istruzione” CMT 2015/2016 in Turin province To select courses to allow young people that have difficulties to integrate and have been excluded from school mainstream education, or even dropped out of school /compulsory education/instruction to fulfil their right/duty to 10 years education and training ESF 8 PA1
SO: 1.8ii.2.4
NA This call was not a selection procedure as such but consisted in the prolongation of the contracts for beneficiaries selected previously via a similar call 2011/2012. For simplicity and time saving, the MA decided to reiterate the same training activity for the period 2015-2018. 21, 22
Italy OP Puglia
15 D.D. No 2487 of 22/12/2014 Objective: Facilitate the creation of new production units; the extension of existing production units; the diversification of the production to new, additional products; fundamental change in the overall production process of an existing unit ERDF 3 PA3 - 3.6 Permanent call   YES provided that the criteria are met. 23, 24
16 DD 367 of 6.8.2015 To select projects aiming at improving and upgrading the network of social welfare, educational and healthcare facilities across Puglia region. Its purpose is to fill in the gaps in the supply of social-care services to people, families and communities in the Region, by co-financing social investment plans or structural measures in the social, healthcare or educational fields. Eligible beneficiaries are e.g. public institutions, mainly municipalities, and private bodies offering social care services, which were formerly public welfare and charity institutions or other public utilities (e.g. local health services, chamber of commerce) ERDF 9 9.10 and 9.11 Permanent call   YES as long as applications score a minimum of 70 points out of 100 25, 26
Finland - OP Structural Funds
17 Enterprise development and support schema Complementary ERDF support schema executed in four Centres for Economic Development, Transport and the Environment ERDF 3 3 / all, discretional focussing on regional targeted actions Permanent, valid for the whole MFF 2014-20. Annual national budgetary appropriation   Yes, as long as applications score a minimum threshold 27, 28, 31, 32, 33, 34
18 Centre for Economic Development, Transport and the Environment South Savo Finland, Mikkeli ERDF: focus on Intelligent specialization
ESF: focus on Support for youth actions and Integration of actors and enhanced cooperation
ESF ERDF 3
8
9
ERDF - PA2
ESF - PA3
ESF - PA4
ESF - PA5
Temporary call:
8.12.2014-16.2.2015
  Yes, as long as applications score a minimum threshold 29, 30
19 Centre for Economic Development, Transport and the Environment, North Ostrobothnia, Oulu   ESF ERDF all All PA/IP/SO Temporary call:
5.5.2014-29.8.2014
  Yes, as long as applications score a minimum threshold N/A
The projects selected were not advanced enough, thus not audited
Finland - OP Structural Funds
20 Centre for Economic Development, Transport and the Environment, North Ostrobothnia, Oulu Under ESF: special focus on the integration of the immigrants into society
Under ERDF: focus on environmental and natural resources in particular bio-recycling economy as well as energy and material efficiency and
on operations/projects linked to research-and development
ESF ERDF 3
8
9
ERDF - PA1
ERDF - PA2
ESF - PA3
ESF - PA4
ESF - PA5
Temporary call:
ESF: 12.12.2015 -5.2.2016 ERDF: 12.12.2015 -12.2.2016
  Yes, as long as applications score a minimum threshold  

Notes:

(a) TO: thematic objective.

(b) PA: priority axis, IP: investment priority, SO: specific objective.

(c) Type of selection procedures:
Temporary Calls: Calls that have a duration inferior of 12 months, usually few weeks to a few months.
Permanent Calls: Calls open with a duration superior of 12 months, sometimes for the whole duration of the programme period.
Direct awards: Involve the earmarking of funding to an organisation, usually a public body, who then redistributes the money to external beneficiaries.

Source: Documentation of the selection procedures.

Annex IV

List of OPs examined in this audit

ERDF ESF Total EU funds Other funds Total EU + other funds TOs covered Partnership Agreement approval date EC approval date of the OP Latest version and approval date Designation of National Authorities by EC/ MS Managing authority (MA)
If applicable: Intermediate Body (IB) or Delegated Managing Authority (DMA)
Amounts in million euros except for CZ
Czech Republic - OP Employment (OPE) - 2014CZ05M9OP001 (amounts in million CZK)
NA 2 119 2 119 450 2 569 8,9,11 13.4.2014 6.5.2015 21.7.2017 EC: 16.9.2016
MS: 13.9.2016
MA: Ministry of Labour and Social Affairs
NO IB
Czech Republic - OP Entrepreneurship and Innovation for Competitiveness (OPEIC) - 2014CZ16RFOP001 (amounts in million CZK)
4 331 NA 4 331 3 611 7 942 1,2,3,4,7 13.4.2014 29.4.2015 1.9.2017 EC: 13.3.2017
MS: 13.3.2017
MA: Ministry of Industry and Trade
IB: Agency for Entrepreneurship and Innovation
France - OP ERDF/ESF Lorraine et Vosges - 2014FR16M0OP015
337 72 409 279 688 1,2,3,4,5,6,9,10 8.8.2014 11.12.2014 NA EC: 13.1.2017
MS: 21.12.2016
MA (since 1 January 2016): L’autorité régionale pour la nouvelle région Grand Est
France - National OP ESF for employment and social inclusion (PON FSE) - 2014FR05SFOP001
NA 2 894 2 894 2 577 5 471 8,9,10,11 8.8.2014 10.10.2014 NA EC : 12.1.2017
MS :21.10.2016
MA: Ministère du travail, de l’emploi, de la formation professionnelle et du dialogue social
13 DMA
Several IB: of which Conseil départemental de la Seine-Saint-Denis, Conseil départemental du Val-de-Marne, Association France active, Fonds paritaire de la sécurisation des parcours professionels
Italy - OP ESF Piemonte - 2014IT05SFOP013
NA 436 436 436 872 8,9,10,11 29.10.2014 11.7.2017 12.12.2014 EC:8.9.2017
MS: 24.10.2016
MA: Region Piemonte (unit “Social Cohesion”)
IB: Citta Metropolitana Torino
Italy - OP Puglia - 2014IT16M2OP002
1 394 386 1 780 1 780 3 560 1,2,3,4,5,6,7,8,9,10,11,12 29.10.2014 17.7.2014 13.8.2015 EC: 21.2.2018
MS:18.7.2017
MA: Region Puglia (unit “Servizio Attuazione del programma”)
IB: Puglia Sviluppo
Finland - OP Sustainable Growth and work - 2014FI16M2OP001
766 513 1 279 1 279 2 558 1,3,4,8,9,10 7.10.2014 11.12.2014 24.5.2016 EC: 20.12.2016
MS: 24.11.2016
MA: The Department of Enterprise and Regional Development at the Ministry of Economic Affairs and Employment
IB: ELY-Centers in Mikkeli and Oulu

Annex V

Priority axis, investment priorities and specific objectives to which the selected projects in Annex I correspond

Priority Axis Thematic Objective Investment priority Specific Objective
Czech Republic - OPE
PA1 8 8i - Access to employment for job-seekers and inactive people, including the long-term unemployed and people far from the labour market, also through local employment initiatives and support for labour mobility of workers 1.1 - Increase employment of supported persons, especially older, low-qualified and disadvantaged ones
    8iv - Equality of women and men in all areas, including access to employment and career promotion, harmonisation of working and private life and promotion of equal pay the same work 1.2 - Reduce the differences in the situation of women and men at the labour market
PA2 9 9i - Active inclusion, including with a view to promoting equal opportunities and active participation, and improving employability 2.1 - Increase the assertion of persons in danger of social exclusion and socially excluded persons in the society and at the labour market
Czech Republic - OPEIC
PA2 3 3 (b) - Developing and implementing new business models for SMEs, in particular with regard to internationalisation 2.2 - Increase the internalisation of small and medium sized enterprises
    3 (c) - Supporting the creation and the extension of advanced capacities for product and service development 2.3 - Increase business infrastructure utilization
France - OP Lorraine et Vosges
PA2 3 A - Promotion of entrepreneurship spirit To support investment in SMEs at all stages of their development
PA9 3 A - Promotion of entrepreneurship spirit To increase tourism in the Vosges mountains
France - PON FSE
PA1 8 8iii - To support self- employment, entrepreneurship and creation of businesses, including innovating micro- and SMEs 2 -0 1. Reinforcement and sharing of the service offer within networks and or between the different actors supporting the creation or take over of activities and the strengthening of activities
      1 - The professionalisation of employees and volunteers from the support networks to the creation of businesses and supporting organisations for strengthening the activities
PA2 8 8v - Employees’, businesses’ and entrepreneurs’ adaptation to change 3 - Actions allowing to meet the conditions and prerequisites for an effective access to training
PA3 9 9i - Active inclusion, including the promotion of equal rights, the active participation and a better ability to work 1 - a) Implementation of personalised paths, strengthening employability, with regard to the different obstacles to overcome, in a global approach
Italy - OP Piemonte
PA1 8 8i - Access to employment for job-seekers and inactive people, including the long-term unemployed and people far from the labour market, also through local employment initiatives and support for labour mobility ER 8.5 - Promote the entry onto the labour market and the employment of the long-term unemployed and those with the greatest difficulties in finding work, and support for people at risk of long-term unemployment
    8.ii - Sustainable integration into the labour market of young people, in particular those not in employment, education or training, including young people at risk of social exclusion and young people from marginalised communities, including through the implementation of the Youth Guarantee ER 8.1 - Increase youth employment
Italy - OP Puglia
PA3 3 3.a - Promoting entrepreneurship, in particular facilitating the economic exploitation of new ideas and fostering the creation of new businesses ER 3.5 - Promoting the set-up and strengthening of micro-enterprises and SMEs
PA9 9 9.a - Investing in health and social infrastructure, which contributes towards development, reduced health inequalities and better service accessibility ER 9.3 - Increasing/ strengthening/ improving the quality of socio-educational services and infrastructure for children and welfare services and infrastructure for people of restricted autonomy, and upgrading the infrastructure network and local healthcare and welfare services
Finland - OP Sustainable growth and work
PA1 3 3a - Promoting entrepreneurship, in particular by facilitating the economic exploitation of new ideas and fostering the creation of new firms, including through business incubators 1.1 - Creating new business activities
PA1 3 3d - Supporting SMEs so that they can be part of the growth in the local, national and international markets and innovative processes 2.1 - Promoting the growth and expansion abroad of SMEs
PA3 8 8i - Access to employment for job-seekers and inactive people, including the long-term unemployed and people far from the labour market, also through local employment initiatives and support for labour mobility of workers 6.1 - Promoting the employment of young people and those in a weak position in the job market
PA5 9 9i - Active inclusion, including with a view to promoting equal opportunities and active participation, and improving employability 10.1 - Improving the ability of those outside employment to work and function

Annex VI

OP’s output and result indicators for the priority axis described under Annex V funded with ERDF

Regions
PA/IP/SO Output indicators Less developed Transition Developed Total Result indicators and source Transition Developed Call Project
Baseline Target 2023 Baseline Target
CZECH REPUBLIC - OP Entreprise and Innovation for competitiveness (OPEIC) 2014CZ16RFOP001
PA 2 CO1: Number of entreprises receiving support 4 600     4 600 SR: percentage of exports in the total turnover of companies (source: MPO) 2011: 21.3 %
22.3 % - 23.8 %
    4 6, 7
3b.2.2 CO2: Number of enterprises receiving grants 2 100     2 100        
  CO4: Number of entreprises receiving a non financial support 3 000     3 000        
  CO6: Private investment matching public support to enterprises (grants) (in million CZK) 84     84            
PA 2 CO1: Number of entreprises receiving support 695     695 SR: total area of regenerated territory in ha (source: national data base on regenerated borwnfields) 2011: 25 875
25 900 – 26 050
    5 8, 9
3c.2.2 CO2: Number of enterprises receiving grants 695     695        
  CO6: Private investment matching public support to enterprises (grants) (in million CZK) 335     335          
  CO22: Total area of regenerated territory in hectares 150     150            
France - OP GRAND EST FR16M0OP015
PA 2 CO1: Number of entreprises receiving support   1 100   1 100 SR4: evolution of the investment of the region's industrial SMEs (source: national statistical office) 2012: - 8.0 %
3.0 %
    6
10,11 and 13
3.A CO2: Number of enterprises receiving grants   805   805    
  CO3: Number of entreprises receiving a support other than grants   530   530    
  CO4: Number of entreprises receiving a non financial support   520   520        
  CO8: Employment increase in supported enterprises   960   960        
PA 9
3.A
CO1: Number of entreprises receiving support   46 33 79 SR22: Number of expected visits and of skiers days
in millions (sources: Observatoires régionaux du Tourisme and Domaine skiable de France)
2011: 1 692
1 861
2011: 1 468
1 614
7
12
CO2: Number of enterprises receiving grants   34 24 58
CO4: Number of entreprises receiving a non financial support   12 9 21        
  CO9: Increase of number of expected visits   11 760 8 240 20 000        
ITALY - OP PUGLIA 2014IT16M2OP002
PA 3
3.a
CO1: Number of entreprises receiving support       2 000 SR3002: company net turnover rate (source: ISTAT) 2012: - 0.6 %
0.4 %
    15 & 16
23, 24, 25 and 26
  CO5: Number of start ups benefiting from support       500    
  CO8: Employment increase in supported enterprises       50         1 2
FINLAND - OP Sustainable growth and work (2014FI16M2OP001)
PA 1 CO1: Number of enterprises receiving support     2 465 2 465 Business dynamics index (source: Statistics Finland)     2010: 1 1.1 17 31
3a CO2: Number of enterprises receiving grants     2 455 2 455        
  CO3: Number of enterprises receiving financial support other than grants     10 10 SME locations     2010: 16 761 18 437    
  CO5: Number of new enterprises supported     1 200 1 200              
  CO6: Private investment matching public support to enterprises (grants in million euros)     51.697 51.697              
  CO7: Private investment matching public support to enterprises (non-grants in million euros)     10 10              
  CO8: Employment increase in supported enterprises     7 800 7 800              
  6: Number of companies that start a new business operation after receiving funding     850 850              
  7: Number of companies with a significant increase in turnover or staff     1 010 1 010              
  8: Number of companies promoting low-carbon solutions     615 615              
  10: Number of SMEs with improved accessibility as a result of the project (Outermost or northern sparsely populated regions)     220 220              
  9: Other investments promoting entrepreneurship as a result of the project (Outermost or northern sparsely populated regions - in million euros)     13.7 13.7              
PA 1 3d CO2: Number of enterprises receiving grants
CO3: Number of enterprises receiving financial support other than grants
    3 175
20
3 175
20
Business dynamics index (source: Statistics Finland)     1 1.10 17 27, 28, 32, 33, 34
PA 1 3d CO6: Private investment matching public support to enterprises (grants) in million euros     337.15 337.15 Growing companies (employs min. 3 people and average growth of staff exceeds 10 %) (source: Statistics Finland)     4 326 4 759  
  CO7: Private investment matching public support to enterprises (non-grants) in million euros     10 10          
  CO8: Employment increase in supported enterprises     4 900 4 900            
  Specific indicators:                    
  Number of companies in projects where the main objective is to promote growth and international business operations     5 170 5 170            
  Number of companies that are starting to export or that expand their exports to a new market area     1 330 1 330            
  Energy saved (MWh)     460 435 460 435            
  Companies with a significant increase in turnover or staff     1 340 1 340            
  Companies promoting low-carbon solutions     650 650            

Legend: CO: common output indicator; CR: common result indicator, SR: specific result indicator (under the ERDF, all result indicators are specific to the Member State / OP).

Annex VII

Output and result indicators for the priority axis described under Annex V funded with ESF

PA/IP Output indicators M F Total Result indicators and source Baseline Target (2023) Call Project
M F Total M F Total
CZECH REPUBLIC - OP Employment 2014CZ05M9OP001
PA1
1.1
CO06: Participants below 25 years of age  
38 571 CR01: Inactive participants who started seeking a job again after termination of their participation (IS ESF 2014+)     2013: 1 210     3 500 1 1
       
  CO07: Participants above 54 years of age     51 429 CR03: Participants gaining a qualification after termination of their participation     13 268     67 000
  CO20: Number of projects fully or partially implemented by social partners or NGOS     100 CR04: Participants employed, including the self-employed, after termination of their participation     2 256     58 740
          CR05: Disadvantaged participants seeking a job after termination of their participation, involved in education/ training, improving their qualification or already employed, including the self-employed     4 089     42 000
  Specific indicators:
CESF0: Total number of participants
    178 000 CR06: Participants employed 6 months after termination of their participation, including the self-employed     1 805     82 000
  Unemployed participants, including the long-term unemployed persons     172 493 CR007: Participants whose position on the labour market has improved 6 months after termination of their participation     903     1 700
  Participants with completed primary ISCED) or lower-secondary (ISCED 2) education     8 900 CR008: Participants at the age of 54+ employed 6 months, including self-employed, after termination of their participation     1 292     12 300
  Inactive participants     5 687 CR09: Disadvantaged participants employed 6 months after termination of their participation, including the self-employed     268     41 000
                           
PA1 1.2 CO20: Number of projects fully or partially implemented by social partners or NGOS     90 CR01: Inactive participants who started seeking a job again after termination of their participation (IS ESF 2014+) 134 1 220 1 354 100 1 400 1 500 2 2, 3, 4
  CO21: Number of projects targeting sustainable employment of women and sustainable career progression of women     410 CR03: Participants gaining a qualification after termination of their participation 49 440 489 50 450 500
  CO22: Number of projects targeting public authorities or public services at national, regional and local levels     60 CR04: Participants employed, including the self-employed, after termination of their participation 26 245 271 48 432 480
  CO23: Number of micro, small and medium sized enterprises supported (incl. cooperatives and social businesses)     130 CR05: Disadvantaged participants seeking a job after termination of their participation, involved in education/training, improving their qualification or already employed, including the self-employed 69 622 691 131 1 184 1 315
  CO35: Capacity of supported child-care facilities or educational facilities     4 000 CR06: Participants employed 6 months after termination of their participation, including the self-employed 48 432 480 55 495 550
Specific output indicators:
CESF0: Total number of participants

940

9 460

10 400
CR007: Participants whose position on the labour market has improved 6 months after termination of their participation 50 243 105 105 495 600
CR008: Participants at the age of 54+ employed 6 months, including self-employed, after termination of their participation 4 34 38 5 45 50
  80500: The number of analytical and strategic documents written and published (incl. evaluation ones)     35 CR09: Disadvantaged participants employed 6 months after termination of their participation, including the self-employed     336     150
          Specific indicators:            
  50100: Number of supported child-care facilities or educational facilities     333 50110: Number of persons using a childcare facility for pre-school children     400     6 000
  50105: Number of employers supporting flexible forms of work     70 50130: Number of persons using a flexible form of work     50     500
                       
PA2
2.1
CO20: Number of projects fully or partially implemented by social partners or NGOS     415 CR01: Inactive participants who started seeking a job again after termination of their participation (IS ESF 2014+)     2 527     3 326 3 5
  CO22: Number of projects targeting public authorities or public services at national, regional and local levels     14 CR04: Participants employed, including the self-employed, after termination of their participation     1 010     1 386
  CO23: Number of micro, small and medium sized enterprises supported (incl. cooperatives and social businesses)     231 CR05: Disadvantaged participants seeking a job after termination of their participation, involved in education/training, improving their qualification or already employed, including the self-employed     934     5 636
          CR06: Participants employed 6 months after termination of their participation, including the self-employed     665     860
  Specific indicators:
CESF0: Total number of participants
    167 244 CR007: Participants whose position on the labour market has improved 6 months after termination of their participation     3 326     16 724
  67001: Capacity of supported services     32 063 CR008: Participants at the age of 54+ employed 6 months, including self-employed, after termination of their participation     143     138
  67101: Number of supporting institutions supported     5 CR09: Disadvantaged participants employed 6 months after termination of their participation, including the self-employed     455     832
  80500: The number of analytical and strategic documents written and published (incl. evaluation ones)     18 Specific indicators:
67010: Use of supported services
    16 000     74 000
  67401: New or innovated social services related to housing     247 67110: Number of ancillary institutions which operate even after termination of the support     2     5
  10210: Number of social enterprises established thanks to the support     138 62910: Participants employed including self-employed 12 months upon leaving     333     430
  67300: Number of participants provided with consulting on social entrepreneurship     80 67310: Former participants of projects in which intervention by means of social work served its purpose
including the self-employed
    27 914     62 558
          10211: Number of social enterprises established thanks to support, operating even after termination of support     100     92
FRANCE - NATIONAL OP ESF 2014FR05SFOP001
PA3
9i
CO01: Unemployed, including long-term unemployed     1 400 000 CR02: Participants following a qualification after termination of their participation 91 322   180 000 8 & 9 14 & 15
  CO03: Participants inactive     675 000
          CR03: Participants gaining a qualification after termination of their participation     25 961     52 500
  Specific indicators:
9i3: Number of women
    1 000 000 CR04: Participants employed, including the self-employed, after termination of their participation     281 063     575 000
  9i4: Number of participants from priority districts from the City policy     230 000              
  9i5: Number of projects aiming at coordinating and facilitating the offer related to social insertion       Specific indicators :
R91.4: Number of actions for coordinating and facilitating implemented
           
  9i7: Number of projects aiming at mobilising employers from the commercial and non-commercial sectors       R91.6: Number of organisations with social utility and of employers that received support            
PA 1
8iii
Specific indicators :
8iii1: Number of entrepreneurs or buyers that received support
    340 400 Specific indicators (no common indicators for this specific objective) the values for different categories of regions are cumulated:
R83.1: Number of businesses created
         
90 000
11 17
  8iii2: Number of female entrepreneurs that received support     129 200 R83.2: Number of businesses created priority districts from the City policy           9 000    
  8iii3: Number of entrepreneurs from priority districts from the City policy that received support     34 000 R83.3: Number of sharing activities achieved                
          R83.4: Number of businesses created by women           36 000    
ITALY - OP PIEMONTE 2014IT05SFOP013
PA 1 1.8i CO01: Unemployed, including long-term unemployed 9 900 10 600 20 500 CR06: Disadvantaged participants in employment, including self-employment, 6 months after leaving training 34 36 35 45 45 45 13 19 & 20
  CO23: Number of supported micro and SMEs     2 700                
PA 1 1.8i CO01: Unemployed, including long-term unemployed                      
  CO23: Of micro, small and medium sized enterprises supported (incl. cooperatives and social businesses)                      
PA 1 1.8ii CO01: Unemployed, including long-term unemployed 8 500 8 500 17 000 CR06: Disadvantaged participants in employment, including self-employment, 6 months after leaving training 30.5 30.5 30.5 40.5 40.5 40.5 14 21 & 22
  CO03: Participants inactive 13 300 7 700 21 000                
  CO06: Participants below 25 years of age 18 150 12 550 30 700                
FINLAND – OP Sustainable growth and work (2014FI16M2OP001)
PA 3 8i Specific indicators:
Number of participants under 30 years of age who are unemployed or outside the labour force
    26 000 Specific indicator:
Participants under 30 in employment when leaving the measure
(source: statistics Finland)
    2013: 23     30 18 29
  Number of participants over 54 years of age who are unemployed or outside the labour force     8 600 Participants over 54 in employment when leaving the measure     17     23  
PA5
9i
CO02: The long-term unemployed     11 250 Specific indicator:
Improved work and functional capacity of the participants (Separate review)
  2012: 5.5   5.9 18 30
  CO04: Those outside employment and not in training     4 100    
  CO20: Number of projects realised or partly realised by social partners or civic organisations     55    

Legend: CO: common output indicator; CR: common result indicator; M: male; F: female.

Annex VIII

Outputs and results in the selection procedures and applications examined

Are there quantified indicators in the applications? Is there an assessment of the indicators and the values provided?
(a) (b) Is there a selection criteria about result indicators / expected results? Output indicators corresponding to those at OP level and quantified Result indicators corresponding to those at OP level and quantified Output Results
Czech Republic OPE 2014CZ05M9OP001
1 1 Expected results to be outlined, result indicators mentioned and corresponds to the OP indicators but no target / quantification required       NA, no information on results
2 2
3 3   Use of “unit costs” (c)  
4  
5   NA, no information on results
Czech Republic OPEIC 2014CZ16RFOP001
4 6 Expected results to be outlined, result indicators mentioned and correspond to the OP indicators but no target / quantification required   Applications outlining expected results and their contribution to the OP objectives, no quantification of the result indicator related to the OP They are binding and are mentioned in the grant agreement Assessment used for the scoring
7
5 8 Assessment used for the scoring
9
France OP Lorraine et Vosges 2014FR16M0OP015
6 10 Expected results to be outlined (project have to prove how they contribute to the achievement of the indicators)   Applications outlining expected results and their contribution to the OP objectives   Limited assessment of the expected results in the assessment reports
  11
  13
7 12     Limited assessment of the expected results in the assessment reports (result indicator not assessed)
France PON FSE 2014FR05SFOP001
8 14 Not explicitely requested but this is required in the standard format application   But result indicators different from the OP result indicators    
9 15 (Indicators different from the OP’s were also provided)  
10 16 But result indicators different from the OP result indicators
11 17 Output indicators not related to OP indicators and not quantified Only qualitative information on the expected results Only the relevance of the indicators used was assessed Only relevance of the indicators used was assessed
12 18   (Indicators different from the OP’s were also provided) Consistency with amount of grant  
Italy OP Piemonte 2014IT05SFOP013
13 19&20       Use of unit costs (c) NA, no info on results
14 21&22
Italy OP Puglia 2014IT16M2OP002
15 23&24 Request for expected results only through the business plan   Applications outlining expected results and their contribution to the OP objectives   Assessment of the described expected results but with weaknesses
16 25&26
Finland OP Sustainable growth and work 2014FI16M2OP001
17 ERDF 27, 28 31, 32 33, 34 But they do not correspond to the OP indicators No information on output indicators but data may be obtained directly from the IT system   NA, no info on outputs But indicators do not correspond to the OP indicator
18 ESF 29         Assessment used for the scoring
30   NA, no info on results
19 ESF/ERDF But for the ERDF they do not correspond to the OP indicator NA, no projects examined under these selection procedures
20 ESF/ERDF
  YES   Partially   NO   NA: not applicable

Note: (a) See Annex III list of selection procedures. (b) See Annex I list of projects examined. (c) Unit costs is a methodology to determine the amount of eligible costs where this amount is equal to the output to be reached (for instance the number of places to be created in a crèche, here number of places created) multiplied by the unit cost per place.

Source: (1) Documentation for the selection procedures. (2) Applications submitted for the projects examined during this audit. (3) Assessment reports for these applications

The Commission’s replies

Executive summary

IV

The selection of projects is the exclusive responsibility of the Member States (MS). The monitoring arrangements, as well as the achievements reported in the Annual Implementation Reports (AIRs) and progress reports, allow to assess progress towards the objectives quantified by the output and result indicator targets as programmed in the operational programmes (OP) (which include baselines and targets). Evaluations make it possible to assess the contribution to policy objectives.

First bullet: The Commission considers that there is a difference between direct results of projects financed and the results to be achieved at the level of the OP (reflected by the result indicators).

As regards the basis of selecting projects, the Commission considers that in many types of assistance, it is more efficient to set quality thresholds and accept all projects that meet this threshold than to apply a method of direct comparison of applications. The latter approach may lead to a high number of initial project applications which in itself do not necessarily result in better overall project quality and would entail unnecessary administrative burden for the managing authorities and for applicants.

Second bullet: Audits of the monitoring systems can only be carried out in a meaningful way once sufficient data on the projects implemented has been collected, aggregated and reported to the Commission. In the cases of the European Regional Development Fund (ERDF) and the Cohesion Fund (CF), due to longer project duration, this explains a later start of the reporting and, therefore, of the audits on reliability of the reported values. The Commission will continue its own audits during 2018 and undertake further efforts to raise the awareness of audit authorities about the importance of providing assurance on the reliability of performance data monitoring systems ahead of the performance review.

The monitoring systems fall under the responsibility of Member States who should have put everything in place according to the regulatory requirements. The Commission has continuously made every possible effort to provide guidance and practical support (e.g. monitoring guidance document, evaluation partnership meetings, the ESF Data Support Centre) to Member States in setting up robust monitoring systems.

Third bullet: The Commission considers that the established monitoring framework is now more results-oriented.

The 2017 Strategic report is based on MS reports as of end-2016. Given the slow start of programme implementation, it was too early to report on results. Until the next strategic report, the achievement values for the European Structural and Investment Funds (ESI Funds) common indicators are available on the Open Data platform.

V

Second bullet: The definition of result indicators, applicable to all Funds, is included in Article 2 of the Commission’s proposal for a Common Provisions Regulation (CPR) for the post-2020 programming period.

The Commission’s legislative proposal for the post-2020 programming period includes a list of common result indicators for ERDF and the Cohesion Fund in line with the Better Regulation Guidelines and with the concept of ‘results’ used under the European Social Fund (ESF). These result indicators would measure the effects on the beneficiaries, including the short-term ones, of the interventions supported.

The Better Regulation guidelines and toolbox provide definitions of output, result and impact indicators, as well as their links with objectives.

Third bullet: The Commission will continue working with the Member States to improve the reliability of data. The Commission has invited the Member States to submit relevant OP amendments necessary to revise targets and milestones by 30 June 2018 in order to ensure their timely processing.

Observations

16

Fourth bullet: As regards the basis of selecting projects, the Commission considers that in many types of assistance, it is more efficient to set quality thresholds and accept all projects that meet this threshold than to apply a method of direct comparison of applications. The latter approach may lead to a high number of initial project applications which in itself do not necessarily result in better overall project quality and would entail unnecessary administrative burden for the managing authorities and for applicants.

25

The Commission considers that there is a difference between direct results of projects financed and the results to be achieved at the level of the OP (reflected by the result indicators). Whereas direct results are influenced only by the action co-financed, results indicators are affected by different external factors outside the control of the managing authorities. A well- designed intervention logic shall ensure that the outputs and direct results of the selected projects contribute to achieving the expected results of the OP.

26

The selection of projects is the exclusive responsibility of the managing authorities that define the selection criteria and apply them when examining the applications for funding.

27

Selection criteria do not necessarily have to include scoring of expected results. The intervention logic of the programme sets out how the outputs of the individual projects contribute to achieving the expected results of the OP.

See also the Commission reply to paragraph 25.

28

A broad population of applicants is not always necessary, depending on the policy area.

32

The Commission considers that in many types of assistance, it is more efficient to set a quality threshold and accept all projects that meet them than to apply a method of direct comparison of applications. In other types of assistance competitive selection procedure may not contribute to the specific objectives of the OP (e.g. transport infrastructure projects or projects of municipalities where competition between public authorities/municipalities is not necessary).

34

The Commission notes that in the case of Finland the procedure included the allocation of a minimum score to ensure that only quality projects are selected. Comparison between projects would not bring more quality except if there was insufficient funding to select all the projects.

The Commission considers that the quality of projects selected depends more on the selection criteria applied than on the type of selection procedure.

See also Commission reply to paragraph 16.

44

The classification is in accordance with the applicable EU rules on data protection. Indeed, in case of the special category of data (sensitive data), the participant can refuse to declare it. Nevertheless, the managing authority (MA) is required by the ESF monitoring guidance to document the attempt of collection. Moreover, the Directive lays down that subject to the provision of suitable safeguards, MS may, for reasons of substantial public interest, lay down exemptions to the participant's consent.

Having regard to the participants' right to refuse consent, the participation records are considered as complete if all non-sensitive data are recorded.

The minimum requirement for participation records complete with all non-sensitive data has been clarified in the ESF monitoring guidance.

Box 2 — Examples

The French MA has taken measures targeting delegated MAs, IB and beneficiaries in order to foster a timely collection and submission of data, as advised by the Commission and national evaluations. Correction of AIR data allowed to report about more than 925 000 participants at the end of 2016.

Box 3 — Examples of IT systems still not fully functional and related risks

First bullet: The Commission constantly monitors the performance of the Czech single IT monitoring system for all OPs. It has been a regular point of the Partnership Agreement follow-up meetings with the National Coordinating Authority which is in charge of the functioning of the ESI Funds IT monitoring system.

Second bullet: The issue raised by the ECA is in line with the results of the Commission's performance data reliability audit of this programme.

52

The shortcomings in the AIRs do not necessarily put into question the reliability of the data reported. When the Commission has doubts about the indicator values reported, the issue is raised in the observation letters on the AIRs. Any relevant audit findings are followed up as well.

53

Performance data should be verified by the managing authority before it is included in the AIR. On the other hand, system audits (by audit authority or the EC) may be carried out more usefully after submission of the AIR, to allow backtesting of the data submitted to the Commission, in order to provide assurance on the reliability of the entire system for collecting, aggregating and reporting the data including the last step of submitting the data to the EC through SFC2014.

The legal base, however, does not give enough time to audit AIR data before the AIRs and strategic reports are issued. Most recent, corrected and updated data can always be found on the Open Data Platform.

60

To mitigate the risk the ECA refers to in this paragraph, the Commission will continue its own audits during 2018 and undertake further efforts to raise the awareness of audit authorities about the importance of providing assurance on the reliability of performance data monitoring systems ahead of the performance review.

61

A meaningful assessment concerning the underachievement of targets set for 2023 can only be performed once the timeline for meeting the targets is reached.

Linking, coupling or modifying allocations across priority axes, as well as the upward and downward revisions of target values for 2023 can be misleading as, whilst such modifications of allocations between priority axes indeed "naturally" lead to changes in target values, there are other reasons that can justify the change of 2023 target values.

62

The Member States may propose revisions of milestones and targets when changes in allocations for a given priority occur or in duly justified cases provided for in the Regulations (Article 30(1) CPR, CPR Annex II, Article 5(6) of Regulation (EC) No 215/2014).

In order to make sure that such OP amendments are submitted and processed in due time before end 2018 and to avoid changes in the performance framework designed to simply adjust values to actual implementation on the ground, Member States have been invited to officially submit any such request by 30 June 2018. The assessment of these OP amendment requests follows a standard procedure in order to ensure consistent treatment.

65

The logical framework for the ERDF in 2014-2020 period takes account of the fact that achievement of the objectives set for the programmes is due to the actions co-financed by the public intervention, for example by the Funds, as well as other factors. Therefore, while the result indicators allow to make a judgement on whether or not the indicators move in the desired direction, to disentangle the effects of the EU intervention from the contribution of other factors and to understand the functioning of a programme it is necessary to carry out an impact evaluation.

69

See Commission reply to paragraph 25.

71

The performance framework relies on various indicators to ensure that programmes are on track to deliver what was agreed. Result indicators may not be appropriate (for ERDF/CF), because the timing when results can be achieved and captured by the system is not compatible with the compulsory timeframe for a useful performance reserve allocation.

72

All ESI Funds common indicator achievements as reported by the MAs are publicly available in the Open Data Platform.

The 2017 Strategic report is based on MS reports as of end-2016. Given the slow start of programme implementation, it was too early to report on results.

73

The definition of result indicators, applicable to all Funds, is included in Article 2 of the Commission’s proposal for a Common Provisions Regulation for the post-2020 programming period.

The Commission’s legislative proposal for the post-2020 programming period includes a list of common result indicators for ERDF and the Cohesion Fund in line with the Better Regulation Guidelines and with the concept of ‘results’ used under the ESF. These result indicators will measure the effects on the beneficiaries, including the short-term ones, of the interventions supported.

The ESF and ERDF context are fundamentally different. However, the Commission has set up an inter-service group to address the differences of indicators across all ESI Funds. In this way, the highest possible consistency as regards the concepts will be reached, taking into account the different nature of the operations co-funded by the ESI Funds.

The Better Regulation guidelines and toolbox provide definitions of output, result and impact indicators, as well as their links with objectives.

Conclusions and recommendations

74

The Commission considers that the established monitoring framework is now more results-oriented.

Delegated regulation 480/2014 requires the recording and storing of monitoring data, in particular the target value of the output and result indicator (data fields 33 and 38), and the achievement values at operation level.

In the context of ESI Funds, outputs and results are equally measured, monitored and reported on regularly through for instance the Annual Implementation Reports. Member States have the possibility to report either on partially or fully completed operations according to the regulatory provisions. Also, due to the duration of the participations, result indicators are reported with a certain time-lag compared to output indicators. The achievement values of all ESI Funds common indicators are available in the Open Data Platform.

75

The Commission considers that there is a difference between direct results of projects financed and the results to be achieved at the level of the OP (reflected by the result indicators). Whereas direct results are influenced only by the action co-financed, results indicators are affected by different external factors outside the control of the managing authorities. A well- designed intervention logic shall ensure that the outputs and direct results of the selected projects contribute to achieving the expected results of the OP.

The selection of projects is the exclusive responsibility of the managing authorities that define the selection criteria and apply them when examining the applications for funding.

77

The Commission considers that in many types of assistance, it is more efficient to set a quality threshold and accept all projects that meet them than to apply a method of direct comparison of applications. In other types of assistance competitive selection procedure may not contribute to the specific objectives of the OP (e.g. transport infrastructure projects or projects of municipalities where competition between public authorities/municipalities is not necessary).

The Commission considers that the quality of projects selected depends more on the selection criteria applied than on the type of selection procedure.

78

The monitoring systems fall under the responsibility of Member States who were required to put these into place according to the regulatory requirements.

The Commission has followed a proactive approach in relation to monitoring by providing guidance to Member States and its desk officers, by carrying out plausibility tests of the data submitted by Member States, and by introducing additional validation rules in the IT system ‘SFC’ minimizing the possibility to report erroneous data. Updates of data are normal and have been encouraged by the Commission. Data validation takes time, especially when setting up the MIS. Most recent, corrected and updated data can always be found on the Open Data Platform.

79

The Commission will carry out additional audits of monitoring systems under its specific enquiry during 2018 and continue collaboration with the audit authorities to increase assurance on the reliability of performance data monitoring systems ahead of the performance review.

80

The Member States may propose revisions of milestones and targets when changes in allocations for a given priority occur or in duly justified cases provided for in the Regulations (Article 30(1) CPR, CPR Annex II, Article 5(6) of Regulation (EC) No 215/2014).

In order to make sure that such OP amendments are submitted and processed in due time before end- 2018 and to avoid changes in the performance framework designed to simply adjust values to actual implementation on the ground, Member States have been invited to officially submit any such request by 30 June 2018.

REGIO has made an internal analysis of upcoming OP amendments relevant to performance framework changes, prioritised the upcoming performance review in its management plan and has planned resources accordingly.

81

The ESF and ERDF context are fundamentally different. However, the Commission has set up an inter-service group to address the differences in concepts across all ESI Funds. In this way, the highest possible consistency will be reached taking into account the different nature of the operations co-funded by the ESI Funds.

The definition of result indicators, applicable to all Funds, is included in Article 2 of the Commission’s proposal for a Common Provisions Regulation for the post-2020 programming period.

The Commission’s legislative proposal for the post-2020 programming period includes a list of common result indicators for ERDF and the Cohesion Fund in line with the Better Regulation Guidelines and with the concept of ‘results’ used under the ESF. These result indicators would measure the effects on the beneficiaries, including the short-term ones, of the interventions supported.

82

Individual projects deliver outputs and direct results, which are not necessarily the same as the specific objectives of the programmes. It is the totality of the projects financed under a specific objective that shall contribute to achieving the expected results.

The intervention logic of the programmes describes how the outputs of the individual projects shall contribute to achieving the objectives of the programme. It is the task of evaluation to assess ex-post whether the intervention logic was effective to reach the objectives.

Concerning the ESF, the grant agreement does not require quantified targets according to the delegated act (480/2014, see above).

83

In the ESF context, outputs and results are equally measured, monitored and reported on regularly for instance through the Annual Implementation Reports. It is to be noted that Member States have the possibility to report either on partially or fully completed projects according to the regulatory provisions.

The performance framework relies on various indicators to ensure that programmes are on track to deliver what was agreed. Result indicators may not be appropriate (for ERDF/CF), because the timing of when results can be achieved and captured by the system is not compatible with the compulsory timeframe for a useful performance reserve allocation.

All ESI Funds common indicator achievements as reported by the MAs are publicly available in the Open Data Platform.

Recommendation 1 – Results-oriented selection

The Commission notes that this recommendation is addressed to the Member States.

Recommendation 2 – Results-oriented monitoring
  1. The Commission notes that this sub-recommendation is addressed to the Member States.
  2. The Commission accepts the recommendation.

The definition of result indicators, applicable to all Funds, is included in Article 2 of the Commission’s proposal for a Common Provisions Regulation for the post-2020 programming period.

The Commission’s legislative proposal for the post-2020 programming period includes a list of common result indicators for ERDF and the Cohesion Fund in line with the Better Regulation Guidelines and harmonised with ESF. These result indicators would measure the effects on the beneficiaries, including the short-term ones, of the interventions supported.

Recommendation 3 – Reporting on performance and preparation of the performance review in 2019
  1. The Commission accepts this recommendation and considers it as being implemented.

    The audit units in DG EMPL have already carried out a significant number of audits on the reliability of performance data and continue to include this aspect in their Early Preventive Audit missions. Furthermore, a specific performance audit, covering also the monitoring systems, was carried out by an external auditor – under the responsibility of DG EMPL – and any issues of concern have been raised and addressed by the Member States.

    The audit plan of DG REGIO for 2017-2018 includes 20 audit missions under the specific enquiry on performance data reliability. On 28 May 2018, the management of DG REGIO discussed an overview of the results from 13 of such audits of reliability of performance data monitoring systems carried out so far in nine Member States. Further audits are planned for the remainder of 2018, and collaboration with the Audit Authorities (AAs) will continue in order to increase assurance on the reliability of the monitoring systems ahead of the 2019 performance review.

  2. The Commission accepts this recommendation and considers it as being implemented.

    Where Commission or AA audits identify weaknesses, the necessary corrective actions are set out in the respective audit reports with a corresponding deadline.

  3. The Commission partially accepts this recommendation.

    OP amendments are submitted at the initiative of the Member States but there is no legal deadline to submit them. The Commission took the necessary steps to organise itself that the OP amendments related to the performance framework are processed in good time. Furthermore, in order to make sure that such OP amendments are submitted and processed in time, Member States have been invited to submit officially any such request by 30 June 2018. The assessment of these OP amendment requests follows a standard procedure in order to ensure consistent treatment. The Commission will deal with these OP amendments in the regulatory deadline provided by the CPR.

Glossary

Europe 2020 Strategy: The EU’s growth strategy for the 2010–2020 period, aimed at allowing it to recover from the crisis. The strategy is split into five headline targets covering employment; research and development; climate and energy; education; and social inclusion and poverty reduction.

European Code of Conduct on Partnership: A set of principles set out in Commission Delegated Regulation (EU) No 240/2014 to support Member States in organising partnerships for drawing up and implementing partnership agreements and operational programmes.

European Structural and Investment Funds (ESIF or ESI funds): The ESI funds are five separate funds that aim to reduce regional imbalances across the Union, with policy frameworks set for the seven-year MFF budgetary period. The funds include the European Regional Development Fund (ERDF), the European Social Fund (ESF), the Cohesion Fund (CF), the European Agricultural Fund for Rural Development (EAFRD) and the European Maritime and Fisheries Fund (EMFF).

Ex-ante conditionalities: Conditions which Member States must meet before they can receive any funding under the European Structural and Investment Funds. When preparing operational programmes in the 2014-2020 programme period, Member States were required to assess whether these conditions have been fulfilled. If they have not, action plans were required to be drawn up to ensure fulfilment by 31 December 2016.

Financial indicators: Values used to monitor progress in terms of the (annual) commitment and payment of the funds available for any project, measure or programme in relation to its eligible cost.

Indicator: A measurable proxy for an objective, providing information to support an assessment of the degree to which an objective has been met.

Intervention logic: The link between the assessed needs, objectives, inputs (planned and allocated), outputs (targeted and achieved) and results (intended and actual).

Investment priorities: For each priority axis, Member States must set out in their operational programmes the investment priorities and corresponding specific objectives. They are listed in the specific Regulations governing the ERDF, ESF and CF.

Managing authority (MA): A managing authority is a public or private body which has been designated by a Member State to manage an operational programme. Its tasks include selecting projects to be funded, monitoring how projects are implemented, and reporting to the Commission on financial aspects and achieved results.

Monitoring: The regular examination of expenditure, outputs and results, which provides up-to-date information on whether projects/programmes are progressing as intended.

Monitoring committee: A committee which monitors the implementation of an OP. It consists of “representatives of the relevant Member State authorities” (e.g. representatives of the managing, certifying and audit authorities, implementing bodies, employer or employee organisations, and civil society). The Commission also has an observer role.

Operational Programme (OP): A statement of a Member State’s priorities and specific objectives which describes how funding (EU and national public and private co-financing) will be used over a given period (currently seven years) to finance projects. The projects within an OP must contribute to a certain number of objectives specified at the level of the OP’s priority axes. OP funding may come from the ERDF, CF and/or ESF. The OP is drawn up by the Member State and has to be approved by the Commission before any payments can be made from the EU budget. OPs can only be modified during the programme period if both parties agree.

Outputs: Something that is produced or accomplished with the resources allocated to an intervention (e.g. training courses delivered to unemployed young people, number of sewage plants, or a length of road built).

Output indicators: Values used to measure the outputs of the operations supported or the outputs at operational programme level.

Performance framework: A set of milestones and targets defined for each priority axis in an Operational Programme, forming an important pillar of the performance-orientation approach.

Performance reserve: Resources constituting 6 % of the resources allocated to the ERDF, ESF and CF or to the EAFRD and EMFF, to be made available after the 2019 performance review if certain requirements are met or exceeded.

Priority axis: One or more investment priorities associated with one thematic objective. The funding in an operational programme is organised by priority axis.

Results: Immediate changes that arise for beneficiaries at the end of their participation in an intervention (e.g. improved accessibility to an area due to the construction of a road, trainees who have found a job).

Result indicators: Values used to measure the results generated by supported projects, or the results achieved at operational programme level.

Selection/award criteria: Predefined criteria used in selection procedures to assess applicants’ ability to complete the proposed action or work programme.

Specific objective: The envisaged result to which EU action aims to contribute.

List of abbreviations

AA: Audit Authority

AIR: Annual Implementation Report

CA: Certifying authority

CF: Cohesion Fund

CPR: Common Provision Regulation

CSF: Common Strategic Framework

DMA: Delegated Managing Authority

EC: European Commission

ERDF: European Regional Development Fund

ESF: European Social Fund

ESI funds or ESIF: European Structural and Investment Funds

IB: Intermediate body

IP: Investment priority

MA: Managing Authority

MS: Member State

OP: Operational Programme

PA: Priority axis

SMEs: Small and medium-sized enterprises

SO: Specific objective

TO: Thematic objective

Endnotes

1 Investment priorities and specific objectives are established based on the Member State’s partnership agreement, which is an agreement between the Member State and the Commission outlining the country’s strategic goals and investment priorities.

2 COM(2010) 700 final of 19 October 2010 “The EU Budget Review”.

3 All these aspects were dealt with in our Special Report No 2/2017 “The Commission’s negotiation of 2014-2020 Partnership Agreements and programmes in Cohesion: spending more targeted on Europe 2020 priorities, but increasingly complex arrangements to measure performance” and Special Report No 15/2017 “Ex ante conditionalities and performance reserve in Cohesion: innovative but not yet effective instruments”.

4 Article 19 of Regulation (EU) No 1303/2013 of the European Parliament and of the Council of 17 December 2013 laying down common provisions on the European Regional Development Fund, the European Social Fund, the Cohesion Fund, the European Agricultural Fund for Rural Development and the European Maritime and Fisheries Fund and laying down general provisions on the European Regional Development Fund, the European Social Fund, the Cohesion Fund and the European Maritime and Fisheries Fund and repealing Council Regulation (EC) No 1083/2006 (OJ L 347, 20.12.2013, p. 320).

5 General Ex ante conditionality 7, Annex XI Part II of Regulation (EU) No 1303/2013.

6 Articles 20, 22 and 96 and Annex II of Regulation (EU) No 1303/2013.

7 Articles 9 and 96 of Regulation (EU) No 1303/2013.

8 Annex I of Regulations (EU) No 1300/2013, No 1301/2013 and No 1304/2013, Article 5 of Regulation (EU) No 1301/2013, article 3 of Regulation (EU) No 1304/2013 and Article 96(2)(b)(ii) of Regulation (EU) No 1303/2013.

9 Special Report No 2/2017.

10 Article 48(3) of Regulation (EU) No 1303/2013.

11 Article 53(2) of Regulation (EU) No 1303/2013.

12 Article 9 of Regulation (EU) No 1303/2013.

13 See ECA’s SR No 2/2017.

14 Article 96(2)(b) of Regulation (EU) No 1303/2013.

15 Article 125(3)(a)(i) of Regulation (EU) No 1303/2013.

16 Article 125(3)(a) of Regulation (EU) No 1303/2013.

17 Projects Nos 27, 28, 30, 31, 32, 33 and 34.

18 Article 116 and Annex XII(3) of Regulation (EU) No 1303/2013.

19 Article 132 of Financial Regulation (EU, Euratom) No 966/2012, Articles 34 and 125 of Regulation (EU) No 1303/2013.

20 COM guidelines: http://ec.europa.eu/regional_policy/en/information/legislation/guidance/.

21 OP Piemonte, Italy, Call “Obbligo d’istruzione” 2015/2016.

22 Articles 72, 73 and 74 of Regulation (EU) No 1303/2013.

23 Article 125(2)(d) of Regulation (EU) No 1303/2013, this requirement is also part of the Management and control system, Article 72(d) of Regulation (EU) No 1303/2013.

24 Ex ante conditionality 7, Annex XI Part II of Regulation (EU) No 1303/2013.

25 Article 19 of Regulation (EU) No 1303/2013.

26 Article 122 of Regulation (EU) No 1303/2013.

27 Article 122(3) of Regulation (EU) No 1303/2013, their actual use by the beneficiaries is not obligatory.

28 EGESIF_14-0010-final 18/12/2014, Guidance for the Commission and Member States on a common methodology for the assessment of management and control systems in the Member States.

29 For instance information on the participant’s background: migrants, foreign background, minorities including marginalised minorities such as the Roma, participants with disabilities, other disadvantaged.

30 On the basis of 20 Commission’s audits on the reliability of performance data carried out mostly in 2017.

31 The MA managed to settle this later during 2017.

32 Article 50(2) of Regulation (EU) No 1303/2013.

33 Article 50(4) of Regulation (EU) No 1303/2013.

34 As well as on the Member States progress reports for 2017. Article 52 of Regulation (EU) No 1303/2013, to be submitted by 31 August 2017 and 31 August 2019 on the implementation of the Partnership Agreements as at 31 December 2016 and 31 December 2018. Progress reports present, namely, the “progress made towards the achievement of the Union strategy for smart, sustainable and inclusive growth, as well as of the Fund-specific missions referred to in Article 4(1), through the contribution of the ESI Funds to the thematic objectives selected, and in particular with regard to the milestones set out in the performance framework for each programme, and to the support used for climate change objectives”.

35 See Articles 75 and 127(4) of Regulation (EU) No 1303/2013 and Annex VII of Commission Implementing Regulation (EU) 2015/207 of 20 January 2015 laying down detailed rules implementing Regulation (EU) No 1303/2013 of the European Parliament and of the Council as regards the models for the progress report, submission of the information on a major project, the joint action plan, the implementation reports for the Investment for growth and jobs goal, the management declaration, the audit strategy, the audit opinion and the annual control report and the methodology for carrying out the cost-benefit analysis and pursuant to Regulation (EU) No 1299/2013 of the European Parliament and of the Council as regards the model for the implementation reports for the European territorial cooperation goal (OJ L 38, 13.2.2015, p. 1) and EGESIF_14-0010-final 18.12.2014, Guidance for the Commission and Member States on a common methodology for the assessment of management and control systems in the Member States - key requirement 6 and 15.

36 Articles 123 and 124 of Regulation (EU) No 1303/2013.

37 Article 124(2) and Annex XIII of Regulation (EU) No 1303/2013.

38 Performance indicators are a set of indicators defined for each priority axis in an OP, which values at the end of 2018 will be compared by the Commission with that of set milestones in order to decide upon the release of the performance reserve in 2019.

39 Milestones are the values set for the indicators of the performance framework to be reached by the end of 2018.

40 See Article 30(1) CPR, CPR Annex II and Article 5(6) of Regulation (EC) No 215/2014.

41 Article 56 of Regulation (EU) No 1303/2013.

42 Projects 27, 28, 31, 32, 33 and 34.

43 Projects 6, 7, 8 and 9.

44 E.g. AR 2015 - Chapter 6 paragraph 6.86 “Result indicators were not defined for 38 % of the projects”. Or AR 2016 - Chapter 6 paragraph 6.56 “For 42 % of the projects it was not possible to identify and measure a specific contribution to the overall programme objectives, since no result indicators or targets were defined at project level”.

45 SR No 15/2017.

46 COM(2017) 755 final of 13 December 2017 “Strategic report 2017 on the implementation of the European Structural and Investment Funds”.

47 See SR No 2/2017, paragraph 150.

48 See SR No 2/2017, paragraph 131.

49 See SR No 2/17, paragraph 150.

50 See SR No 2/17, recommendation 3.

Event Date
Adoption of Audit Planning Memorandum (APM)/Start of audit 18.1.2017
Official sending of draft report to Commission (or other auditee) 3.5.2018
Adoption of the final report after the adversarial procedure 27.6.2018
Commission’s (or other auditee’s) official replies received in all languages 23.7.2018

Audit team

The ECA’s special reports set out the results of its audits of EU policies and programmes, or of management-related topics from specific budgetary areas. The ECA selects and designs these audit tasks to be of maximum impact by considering the risks to performance or compliance, the level of income or spending involved, forthcoming developments and political and public interest.

This performance audit was carried out by Audit Chamber II Investment for cohesion, growth and inclusion spending areas, headed by ECA Member Iliana Ivanova. The audit was led by ECA Member Ladislav Balko, supported by Branislav Urbanič, Head of Private Office and Zuzana Frankova, Private Office Attaché; Myriam Cazzaniga, Principal Manager; Pekka Ulander Head of Task; Michaela Binder, Seconded National Expert.

Contact

EUROPEAN COURT OF AUDITORS
12, rue Alcide De Gasperi
1615 Luxembourg
LUXEMBOURG

Tel. +352 4398-1
Enquiries: eca.europa.eu/en/Pages/ContactForm.aspx
Website: eca.europa.eu
Twitter: @EUAuditors

More information on the European Union is available on the internet (http://europa.eu).

Luxembourg: Publications Office of the European Union, 2018

PDF ISBN 978-92-847-0548-1 ISSN 1977-5679 doi:10.2865/390387 QJ-AB-18-018-EN-N
HTML ISBN 978-92-847-0523-8 ISSN 1977-5679 doi:10.2865/439457 QJ-AB-18-018-EN-Q

© European Union, 2018.

For any use or reproduction of photos or other material that is not under the European Union copyright, permission must be sought directly from the copyright holders.

GETTING IN TOUCH WITH THE EU

In person
All over the European Union there are hundreds of Europe Direct Information Centres. You can find the address of the centre nearest you at: https://europa.eu/european-union/contact_en

On the phone or by email
Europe Direct is a service that answers your questions about the European Union. You can contact this service:

FINDING INFORMATION ABOUT THE EU

Online
Information about the European Union in all the official languages of the EU is available on the Europa website at: https://europa.eu/european-union/index_en

EU Publications
You can download or order free and priced EU publications: https://publications.europa.eu/en/publications. Multiple copies of free publications may be obtained by contacting Europe Direct or your local information centre (see https://europa.eu/european-union/contact_en)

EU law and related documents
For access to legal information from the EU, including all EU law since 1951 in all the official language versions, go to EUR-Lex at: http://eur-lex.europa.eu

Open data from the EU
The EU Open Data Portal (http://data.europa.eu/euodp/en/home?) provides access to datasets from the EU. Data can be downloaded and reused for free, both for commercial and non-commercial purposes.