Academy of Entrepreneurship Journal (Print ISSN: 1087-9595; Online ISSN: 1528-2686)

Research Article: 2021 Vol: 27 Issue: 5

Development of An E-Tool to Evaluate the Effectiveness of the NSRF Operational Entrepreneurship Programs-European Government Program Evaluation (EUGOPEV)

Christina Fountzoula, National Technical University of Athens

Konstantinos Aravossis, National Technical University of Athens

Citation: Fountzoula, C., & Aravossis, K., (2021). Development of an e-tool to evaluate the effectiveness of the nsrf operational entrepreneurship programs - european government program evaluation (eugopev). Academy of Entrepreneurship Journal (AEJ), 27(5), 1-17.


This study proposes a comprehensive model of Analytic Hierarchical Process-Likert scale evaluation - Weighted Score Method (Ahp–Likert -Wsm) for evaluating organizations managing National Strategic Reference Framework (NSRF) operational programs in Greece. We selected the most important criteria which define the optimal management of operational programs. These criteria were ranked and used in formulating an evaluation system for the competent organizations in the management of the operational programs. We used a weighted scoring model in order to create Eugopev, an e-government tool for evaluating organizations.


NSRF, Operational Programs, Eugopev, Evaluation E-tool, Entrepreneurship, Strategic Management.


Funding from the European Union has been a key pillar of Greece's financial support since the early 1980s. The National Strategic Reference Framework (NSRF) 2014-2020 constitutes the main strategic planning with the contribution of significant funds from the European Union’s European Structural and Investment Funds (ESIF) ( Two significant difficulties arise from the management of operational programs: firstly, the reduced absorption of funds, and secondly, the inability to achieve the projected results (European Court of Auditors, 2018).

This study aims to examine the processes for the inclusion of projects in co-funded regional or sectoral programs within NSRF 2014-2020, as well as the processes after the inclusion in the program and up to its completion. Furthermore, we examine the strategic management of operational programs from the beginning up to the end of the planning period. The improvement of these processes plays a crucial part in creating a more direct and efficient absorption of European funds, ensuring financial, environmental and social added value. At the same time, it will also contribute in regional development. The study was conducted by applying the Ahp method in order to rank the criteria defining the optimal management of NSRF programs. Consequently, an evaluation software was developed for assessing the management of operational programs. This software grades the performance of the service responsible for program management.

Research Approach For Evaluating NSRF And European Funding Programs

In the past, there have been research efforts in order to evaluate operational programs. It is now evident that the European funding mechanism is a complex process connected to numerous factors influencing how European countries achieve funding results. (Shows in Table 1)

Table 1
Review of European Funding Evaluation
Author Year Region Type of evaluation
Bradley et al. 2010 Czech Republic National Strategic Reference Framework (NSRF) 2007-2015
Tzortzi 2015 Greece National Strategic Reference Framework (NSRF) 2007-2013
Dvorak 2010 Poland Scope & Significance of EU Structural funds
Armstrong et al. 2012 European Islands European Regional Development Fund 2000-06, 2007-2013
Florio 2007 Europe CBA of EU Structural funds
Bachtler et a; 2000 Europe Effectiveness evaluation methodologies for EU Structural Funds
Lion & Martini 2006 Italy European Social Fund
Eser, & Nussmueller, 2006 Europe Mid-term Evaluations of EU Structural Funds
Lion et al. 2004 Italy European Social Fund
Bradley et al. 2003 Europe Macro-regional evaluation of the Structural Funds
Polverari, L. 2016 Europe EU Structural Funds 2014-2020
Armstrong & Wells 2006 UK EU Structural Funds 2000-2006, 2007-2013
Tarnawska & Ćwiklicki 2012 Poland European Social Fund
Popescu, & Berinde 2017 Europe European Structural Funds 20017-2013
Huliaras & Petropoulos (2016) 2016 Greece European Structural Funds
Mendez et al. 2013 Europe Partnership Agreement (PA) & Operational Programs (OP)
Nigohosyan & Vutsova 2018 Europe European Regional Development Fund (ERDF)
Cella & Florio 2007 Europe Ex-ante and ex-post evaluation of the EU Structural Funds

Florio (2007) conducted a cost benefit analysis (CBA) of European funding programs in order to offer guidelines on a central level for decision-making regarding the management and distribution of funding to European countries. Bradley et al. (2006; 2003) quantitatively evaluated the influence of NSRF in the Czech Republic, by using a macroeconomic model. Armstrong et al. (2012); Armstrong & Wells (2006) studied the effect that the funding by the European Regional Development Fund had in island regions during 2000 –2006 and 2007–2013. Rinaldi and Ferrer (2017) conducted a meta-assessment study based on 15 mid-term evaluations for European Structural Funds programs in Finland. The article concludes that critical analysis of individual evaluations serves political as well as organizational learning (Haghighi et al., 2010). Christensen et al. (2016) evaluated European Funding for strategic investments using a macroeconomic model. Rinaldi and Ferrer highlight the importance of public administration in the efficiency of European funding programs. Eser & Nussmueller (2006) examined the difficulties arising during the process of mid-term evaluations for European funding programs.


Description of Research Process

This study was conducted by combining methods of multi-criteria analysis in order to formulate the final evaluation tool. Initially, the Ahp method was used, which has been applied over time to the evaluation of public projects and programs. Ahp was used to measure the weights of criteria and sub-criteria related to the optimal management of NSRF operational programs. Then the evaluation of the choices of some subcategories was used, with Likert scale evaluation, a much simpler methodology in order to prioritize the above choices, as it is not necessary to measure weights in this process. Finally, the results of the two methodologies were used to create a weighted scoring model (Teknomo, 2006) which evaluates the respective business plan (Edvardsen et al., 1994). This tool can strengthen the evaluation of program efficiency, the program progress and the comparative evaluation between programs. Simirarly, the calculation procedure by Yager's method is identical to the AHP method both for the main criterion and sub-criteria in the problem in question. The main criterion and sub-criteria weights are calculated identically as in the AHP method. The combined weights for the sub-criteria are then calculated by multiplying each main criterion weight and each sub-criterion weight separately. (Yavuz, 2015). In the same way, a Greek organization is evaluated with the creation of a hybrid Ahp methodology (Gerogiannis et al., 2010). The process is presented in the following Figure 1.

Figure 1: Research Process Diagram

In order to collect the data, we questioned 20 experts and professionals whose main activity constitutes submitting files for the inclusion of projects in operational programs. The questionnaires lasted one hour on average, while the number of participants was defined based on the existing literature (Fiedler et al., 2010). Specifically, we selected 14 scholarly studies (Shows in Table 2) that collected the necessary information for the study, using Ahp questionnaires. The average number of participants, as is shown in the following table, was 20 experts.

Table 2
Article Review of AHP Surveys
Author Year No of Questionnaires Title
Duleba et al. 2012 47 A dynamic analysis on public bus transport’s supply quality by using AHP
Saediman 2015 30 Prioritizing commodities in Southeast Sulawesi Province of Indonesia using AHP based Borda count method
Ocampo et al. 2019 27 Public service quality evaluation with SERVQUAL and AHP-TOPSIS: A case of Philippine government agencies
Haghighi et al. 2010 5 The impact of 3D e-readiness on e-banking development in Iran
Lai et al. 2015 15 Evaluating the efficiency performance of airports using an integrated AHP/DEA-AR technique
Keskin & Köksal 2019 35 A hybrid AHP/DEA-AR model for measuring and comparing the efficiency of airports
Huehner et al. (2012) 2012 5 A Case Study on the Application of the Analytic Hierarchy Process (AHP) to Assess Agri-Environmental Measures of the Rural Development Programme (RDP 2007–2013) in Slovenia
(Kamaruzzaman et al., 2018) 2018 10 Developing weighting system for refurbishment building assessment scheme in Malaysia through analytic hierarchy process (AHP) approach
Ortiz et al. 2016 6 An integrated approach of AHP-DEMATEL methods applied for the selection of allied hospitals in outpatient service
Ahmadi et al. 2014 12 Evaluating the factors affecting the implementation of hospital
information system (HIS) using AHP method
Lai et al. 2013 13 An application of AHP approach to investigate tourism promotional effectiveness.
Veisi et al. 2016 52 Developing an ethics-based approach to indicators of sustainable
agriculture using analytic hierarchy process (AHP)
Mobinizadeh et al. 2016 9 A model for priority setting of health technology assessment: the experience of AHP-TOPSIS combination approach
Chatterjee &
2013 12 (2013). Potential hospital location selection using AHP: a study in rural India. International Journal of Computer Applications, 71(17)


The criteria that define the aforementioned factors were studied and selected through the existing structure and operation of public administration, as well as the literature review. They are described in the Shows in Table 3.

Table 3
Criteria and Existing Structure of Public Administration
Criteria Description & Explanation
Sustainability (S) Integration of social, environmental, and financial responsibilities in organizations through their activities (Carter & Rogers, 2008). Elkington introduced the term Triple Bottom Line and highlighted the importance of integrating these three aspects in the main activity of businesses and organizations, in order to achieve long-term results in the application of sustainable development policies.
(S1) Operational program evaluation The systematic and objective analytical assessment of an intervention aims to evaluate its success regarding the goals set, to answer questions related to effectiveness, efficiency, impact, and sustainability. Furthermore, it aims to derive learnings. It is divided into ex-ante evaluation, evaluation during project implementation and ex-post evaluation. (
(S2) Uncontrollable factors Random sudden changes that can upset the balance of the financial system have happened before and will happen again (Ahmed et al., 2021). Some factors that define the course of the market are, e.g., climate change, pandemics, wars, new sources of raw materials, etc.
(S3) Revenue production after completion Cash flows paid directly by users for goods or services provided by the activity, such as fees directly borne by users for the use of infrastructure, the sale or lease of land or buildings, or payments for services excluding operating costs, and expenses replacement of short-lived equipment emerging in the corresponding period (
(S4) Appropriate strategic planning The planning of NSRF and Programs for the period 2021-2017 is implemented gradually through the issuance of circulars and the submission of plans to the European Commission. The circulars that are addressed to planning bodies and relevant services present the program framework and provide guidelines for the structure and content of the programs.
(S5) Direct strategic planning The implementation of NSRF within the scheduled timeframe, without delays
(S6) Review management The review pertains to:
-     Resource transfers between Operational Programs
-     Internal redistributions in each Operational Program
(S7) Alignment with commission framework and the country’s goals The “2030 Agenda for Sustainable Development – Transforming Our World” was adopted by all United Nations Member States in 2015 and includes a plan of action with 17 goals and 169 targets regarding people, planet, prosperity, peace, and partnership. Within this framework, following deliberation with ministries, stakeholder representatives (including GSEVEE - the Hellenic Confederation of Professionals, Craftsmen & Merchants), and other interested parties, the Greek side has set sustainable development goals based on national priorities.
(S8) Innovation Drawing up the national and regional Research and Innovation Strategies for Smart Specialisation comprised a key factor in the development planning of the period 2014-2020. (
Four types are defined:
-      Product innovation (production of new or improved products)
–   Process innovations (use of fewer productive inputs for a stable level of production)
–     Environmental innovation (avoidance or reduction of environmental burdens)
–     Organizational innovation (new form of management) (Oslo Manual, 2005).
(S9) Respect for the environment The main goal of the program “Europe 2020” is: - reduction of greenhouse gas emissions by 20% (or even 30%, conditions permitting) compared to 1990 - securing 20% of energy from renewable sources - 20% increase in energy efficiency
(S10) Social and equality criteria criteria Inclusion of horizontal policies such as securing new jobs, equal opportunities, abolishing discrimination, and ensuring equality between men and women.
(P) Process improvement This study aims to measure the weight of processes followed by the specialization of the proposal until the inclusion of the activity, aiming to improve bureaucratic processes, which will contribute to the optimal management of operational programs.
(P1) Proposal specialization "The Managing Authority has prepared a standard document, “Implementation Specialization of the Operational Program", considering in particular its objectives and priorities, the needs of implementing the policy measures in the areas of intervention of the program based on the proposals submitted by the Executive Committees of the Ministries. Working with the Committees to this end and in the absence of these the relevant Ministry services…" (Circular for proposal specialization 2014-2020)
 (P2) Selection and approval of activities Evaluation of funding applications submitted by potential beneficiaries, following an invitation issued by the Managing Authority. The evaluation methodology, as well as the evaluation criteria of the funding applications, are drafted by the Managing Authority and approved by the Monitoring Committee (
(P3) Monitoring and verification of activities The monitoring of the implementation course for the projects constitutes a continuous process, which is activated with the Inclusion Decision for each project and lasts until its completion. This process aims:
  • To gather the necessary data for confirming that the project is being implemented in alignment with the terms of the funding decision, as well as the legal commitments made.
  • To document the decisions for continuing the cash flow
  • To detect in a timely manner any hazards, discontinuities, deviations, failures in the project and correct them accordingly
      (SDE NSRF 2014 - 2020)
(P4) Audits, financial corrections of audit bodies, recoveries
  • Financial corrections by audit bodies
  • Recovery of unduly or illegally paid sums
  • Reporting Irregularities to the EU
  • Inspection by a Certification Authority
(P5) Funding flow -  Unhindered flow of funding for the co-financed Activities according to planning and based on the progress of their implementation, as well as the smooth and without delays management of payments to Beneficiaries of said Activities (
-  Collection of Community Assistance from the Community Budget and its payment to Beneficiaries. The procedure is applied whenever the accounts of the Programs at the Bank of Greece are credited by the European Commission
(P6) Payment requests, annual accounts and management statement -   Preparation, certification and submission of an interim payment application
-   Preparation, certification and submission of annual accounts
-   Transmission of financial data
-   Preparation and submission of management statement and annual review
(P7) Absorption solutions Actions that promote the absorption of funds that remain unused, such as the cash grant, according to SEV - Hellenic Federation of Enterprises (, the extension of eligible actions, the reinstatement of rejected proposals.
(Q) Improvement of service quality Constant effort by all the members of an organization to satisfy the needs and expectations of clients (Laffel & Blumental, 1989). According to the European quality assurance standard ELOT (Hellenic Organization for Standardization) ΕΝ ISO 8402:1996, quality is the set of characteristics of an entity (a product or service), which give it the ability to satisfy any expressed and implied needs of the user (e.g., the consumer).
(Q1) Educational level The degree of correlation between the level of education of civil servants and the quality of the work they provide is examined. According to previous studies, low work efficiency is associated with low levels of cognitive requirements in employees (Schmidt et al, 1986).
(Q2) Experience  The employee’s experience correlates to client expectations and is expressed through the quality of services (Edvardsson et al, 1994).
(Q3) Training In Greece, continuous professional education and training is not part of the official educational system and is covered by the general term of lifelong learning. The goal of continuous professional education and training is to maintain, renew, upgrade and modernize the working skills of those searching for work, as well as to provide aid to employees who are interested in evolving their career (EQAVET, 2016). in accordance with Greek law there are 5 types of training
(Q4) E-Government Providing the possibility of wide accessibility to Public Administration information with the use of new technologies and the Internet (Ahmed, 2021).
E-government services can be divided in four levels:
-    Information services
-    Communication services
-    Bilateral communication services
-    Processing services (integrated transactions)
(Q5) Infrastructure Critical dimensions and quality criteria are considered as follows:
Tangibles (characteristics): these are elements of the natural environment in which the service is provided (e.g., facilities, equipment, physical space, employees), as well as specifications and/or components of the goods (Zeithaml et al, 1990).
(Q6) Staffing of services The quality and adequacy of human resources of NSRF structures constitutes an essential prerequisite for them to be able to successfully fulfill their institutional duties (
(Q7) Cooperation of those in charge "The government has neither the power nor the appropriate methods to force key ministries to pursue a unified policy. Greek civil servants have little contact with their colleagues in other ministries or even with those of the same ministry. In all ministries there is a lack of recording systems, data processing, and archiving systems. The power of the ministries is fragmented, not only because they are all scattered in hundreds of buildings in the Greek capital, but also because there is almost no contact between them." (ΟECD 2011)
(Q8) Behavior and relations between citizens and staff (politeness, promptness, etc.) The quality of co-production-interaction with the customer (in the provision of services, quality is expressed in the behavior and relations between customers and employees) (Edvardsson et al, 1994).
(Q9) Immediate perception of quality by the citizen (speed of response, access, etc.) Constitutes a crucial quality characteristic (Edvardsson et al, 1994)
(Q10) Motivation for employees Motivation in the public sector is defined as the individual predisposition to respond to incentives that are initially or exclusively created in public services and organizations. According to this definition, it is argued that there are specific incentives related to the nature of work in a public service (Perry & Wise, 1990).

Results and Discussion

Αhp Evaluation Results

The following Figure 2 reflects the main goal of this study, as well as individual criteria and sub criteria. Moreover, it shows the weights that emerged from the comparison between pairs applied in the Ahp method. For calculating the results, we used the ΑHP-OS software (Goepel, 2018).

Figure 2: Diagram for Evaluation of Partnership Agreement

In addition, we conducted a consistency test, as it was observed that the scores showed lack of consistency, especially when a large number of criteria is examined. As this can cause unreliability, the level of inconsistency should not exceed 10%.

The consistency index is calculated as follows: C.I.=(λmax−n)/n-1 where ƛ max is the table’s eigenvalue while n represents the size of the matrix (Ortíz, et al., 2016).

Finally, within the AHP application, when the decision-maker is a team rather than an individual, the geometric mean (G) of the responses is used to determine the result between the data under comparison, using the formula:

Geometric Mean


Where ‘a’ corresponds to each expert and ‘n’ corresponds to the number of experts.

The results indicate that, to a greater extent (77.3%), the optimal management for NSRF depends on and is defined by the program’s sustainability. Namely, from whether the final result of the program meets the strategic goals, according to which it was designed. Other factors include the improvement of service quality (13.9%), and the improvement of processes (8.9%).

Furthermore, it appears that the sustainability of each operational program is directly related to innovation (22%). This demonstrates that the specialization of proposals and the issuance of invitations for the inclusion of projects in the operational program should, in their entirety, include the criterion of innovation and technological upgrading. Other factors include social approval and the inclusion of social criteria in the invitations. Thus, the main strategic goal of the program “Europe 2020” can be achieved: the creation of a smart, sustainable, and inclusive economy. Regarding the improvement of service quality, the e-government solution is in the lead (18.3%), followed by collaboration between relevant bodies (17.6%). This lack of collaboration constitutes one of the greatest weaknesses of Greek public administration.

Regarding the improvement of processes, greater weight is placed on the project testing stage (26.9%), as well as on monitoring (25.1%).

Likert Scale Evaluation

The Ahp analysis revealed the weights of 27 subcategories, which will then be used to create the weighted scoring model. The 19 of the 27 subcategories in the proposed model are converted into questions of the weighting model and with which the respective operational program is evaluated based on the Likert scale, as shown in the Table 4. The remaining 8 criteria are also evaluated using the Likert method on a scale of 1 to 10, and the choices of the respective questions are obtained. More specifically, for sub criteria S1, S10, P7, Q3, Q4, Q6, and Q10 and for the respective questions 1, 10, 17, 20, 21, 23, 27, the evaluator is asked to choose the answers whose weights are measured as follows.

Considering that the alternative that occupies the first place receives as a score 100%, the scores for the other alternatives are formed accordingly, which will be applied below in the weighted scoring model (Shows in Table 4).

Τable 4
Likert Scale Evaluation
(S1) Program evaluation Likert Avergage Ranking Weighted scoring model
Ex-ante evaluation 8,1 1 100%
Evaluation during project implementation 7,85 2 67%
Study of result achievement in ex-post  evaluation 7,45 3 33%
(S10) Social and equality criteria Likert Avergage Ranking Weighted scoring model
New jobs 8,2 3 60%
Equal opportunities and abolishing discrimination 7,4 4 40%
Equality between men and women 8,25 2 80%
Competitiveness for small and medium enterprises 8,8 1 100%
Lifelong learning – acquisition of skills 6,95 5 20%
(P7) Absorption solutions Likert Avergage Ranking Weighted scoring model
Cash subsidy and facilitation of financing procedures for activities 8 1 83%
Further expansion of eligible actions 8,1 2 100%
Reduction of inclusion score 6,3 6 17%
Extending the duration and improving the flexibility of the program 7,45 3 50%
Remote program completion checks 6,45 5 33%
Restoring proposals for evaluation that were excluded from joining the program 6,7 4 66%
(Q3) Training Likert Avergage Ranking Weighted scoring model
Training 7,4 2 80%
Specialization program 8,3 1 100%
Advancement education 6,7 5 20%
Postgraduate training 6,8 4 40%
Postgraduate education 7,2 3 60%
(Q4) E-government Likert Avergage Ranking Weighted scoring model
Information services (providing classified information) 7,3 3 50%
Communication services (e.g., distributing leaflets) 7,2 4 25%
Bilateral communication services (e.g., submitting supporting documents) 8,6 1 100%
Processing services (integrated transactions) 7,5 2 75%
(Q6) Service staffing Likert Avergage Ranking Weighted scoring model
Hiringpermanent employees 7 3 67%
Private sector 7,1 2 33%
Mobility of civil servants 7,5 1 100%
(Q10) Incentives for employees Likert Avergage Ranking Weighted scoring model
Salary depending on skills, experience, etc. 8,6 1 83%
Bonus, overtime, days off, and other privileges 8,3 3 66%
Good working conditions (relationships, workspace) 8,5 2 100%
Opportunities for promotion and advancement 8,2 4 50%
Opportunities for learning and developing skills 7,85 5 33%
Recognition of work and good reputation 7,7 6 17%

The above research shows that the most important of the evaluations is the ex-ante evaluation with a small difference from the rest. In relation to the social and equality criteria included in the project invitations, the greatest preference is expressed in the competitiveness of small and medium-sized enterprises, while lifelong learning is considered to be of lesser importance. In absorption solutions, the extension of eligible actions is considered to be predominant, while in training, specialization programs are more important. At the level of e-government, two-way communication services are considered the most important and in the criterion of staffing, the mobility of civil servants is considered the most favorable solution. Finally, the respondents consider the salary according to the qualifications and the good working conditions as the biggest motivation for the employees who are employed in the competent management services of the business programs.

Eugopev Software Tool

The results of the evaluations are applied within a weighted scoring model, shown in the following Table 5.

Table 5
Weighted Scoring Model
Calculation of each sub-criterion weights
Sub-criteria Weights
Sub (%) Main (%) Combined (%)
(S1) Program evaluation 5,6 77,3 4,3288
(S2) Unbalanced factors 4,4 3,4012
(S3) Revenue generation after completion 8,5 6,5705
(S4) Appropriate strategic planning 5,9 4,5607
(S5) Immediate implementation of strategic planning 3,2 2,4736
(S6) Reviews management 15,3 11,8269
(S7) Country's goals - Community framework 6,7 5,1791
(S8) Innovation 22 17,006
(S9) Respect for the environment 13,4 10,3582
(S10) Social acceptance 14,9 11,595
(P1) Specialization of proposals 4,6 8,8 0,4048
(P2) Selection and approval of projects 8,3 0,7304
(P3) Monitoring and verification of Transactions 25,1 2,2088
(P4) Audits, Financial corrections, recoveries 26,9 2,3672
(P5) Funding flow 15,4 1,3552
(P6) Payment requests, annual accounts 13,1 1,1528
(P7) Absorption solutions 6,6 0,5808
(Q1) Education level 2,7 13,9 0,3753
(Q2) Experience 6,9 0,9591
(Q3) Training 4,2 0,5838
(Q4) E-government 18,3 2,5437
(Q5) Infrastructure 9,6 1,3344
(Q6) Service staffing 8,6 1,1954
(Q7) Collaboration between organizations 17,2 2,3908
(Q8) Relationship of employees with stakeholders 15,6 2,1684
(Q9) Immediate perception of quality 10,5 1,4595
(Q10) Incentives for employees 6,4 0,8896
TOTAL   100 100

Thus, a final evaluation system is formulated, which will generate a final ranking for each examined operational program management body. The bodies are assessed on their performance regarding the 27 criteria described above.

The evaluation is conducted digitally, using an operational program evaluation software named European Government Program Evaluation (Eugopev). This program can be used by independent evaluators or by the entrepreneurs themselves, in order to ascertain the efficiency degree for the management of each operational program (Khan et al., 2021). This evaluation is conducted through 27 questions, the same number as the examined criteria. Furthermore, this program allows the entry of specific data necessary for the evaluations (Shows in Table 6).

Table 6
European Government Program Evaluation (Eugopev)
Criteria Combined
Questionnaire Score limits Grades
S1 4,3288 1. Which type of evaluation does the service best respond to? Ex-ante evaluation
Evaluation during project implementation
Study of result achievement in ex-post  evaluation
S2 3,4012 2. How prepared are you to address uncontrolled factors in order to safeguard the smooth management of the operational program? Satisfactorily
Not applicable
S3 6,5705 3. Do the eligibility criteria for invitations include the factor for revenue generation following the project’s completion? Satisfactorily
Not applicable
S4 4,5607 4. How successful and targeted is the operational program’s strategic planning? Satisfactorily
Not applicable
S5 2,4736/2 5a. How long after the beginning of the planning period did it take to start the program implementation? 6 months
1 year
Longer than 1 year
S5 2,4736/2 5b. What rate of invitations are issued during the first year of the Operational Program? Over 30%
Under 10%
10 to 30%
S6 11,8269 6. Do the program’s strategic goals change during the review process? No
Not applicable
S7 5,1791 7. Do the eligibility criteria for invitations include the factor for alignment with commission framework and the country’s goals? Satisfactorily -100%
Not applicable-0%
S8 17,006 8. Do the eligibility criteria for invitations include the factor for promoting innovation? Satisfactorily -100%
Not applicable-0%
S9 10,4355 9. Do the eligibility criteria for invitations include the factor for respecting the environment? Satisfactorily -100%
Not applicable-0%
S10 11,595 10. Which of the following criteria are included the most in invitations? Equality between men and women
Competitiveness for small and medium enterprises
New jobs
Equal opportunities and abolishing discrimination
Lifelong learning – acquisition of skills
80% 60%
40% 20%
P1 0,4048 11. How many specializations do you have? More than 20
Between 10 and 20
Fewer than 10
P2 0,7304 12. The deadline for the completeness check and the completion of the evaluation of the beneficiary’s proposal by the relevant managing authority is 60 days. How do you respond to this deadline? Less than 30 days
Between 30 and 60 days
More than 60 days
P3 2,2088 13. How do you respond to the timeframe of the stage “Monitoring and Verification of Activities”? Satisfactorily
Not applicable
P4 2,3672 14. How do you respond to the timeframe of the stage “Audits, Financial corrections of audit bodies, recoveries”? Satisfactorily
Not applicable
P5 1,3552 15. How do you respond to the timeframe of the stage “Funding Flow”? Satisfactorily
Not applicable
P6 1,1528 16. How do you respond to the timeframe of the stage “Payment requests, annual accounts and management statement”? Satisfactorily
Not applicable
P7 0,5808 17. Do you apply any of the following absorption solutions? Further expansion of eligible actions
Cash subsidy and facilitation of financing procedures for activities
Restoring proposals for evaluation that were excluded from joining the program
Extending the duration and improving the flexibility of the program
Remote program completion checks
Reduction of inclusion score
83% 66% 50% 33%
Q1 0,3753 18. What is the average educational level within the service? PhD
Postgraduate Degree
Undergraduate Degree
Q2 0,9591 19. What is the average level of experience within the service? 20 years or longer
10-15 years
5-10 years
1-5 years
Q3 0,5838 20. Select  the type of training applied to the service Specialization program
Continuing training
Postgraduate education
Postgraduate training
Advancement education
Q4 2,5437 21. How do you respond to e-government services? Select the level of e-government that you apply the most. Bilateral communication services Processing services
Information services
Communication services
Q5 1,3344 22. Rank the level of infrastructure (facilities, equipment, etc.) in your service: Satisfactory
Q6 1,1954 23. The staffing of the service takes place as follows: Mobility of civil servants
Hiring permanent employees
Private sector
Q7 2,3908 24. Evaluate the collaboration between services: Excellent
Not applicable
Q8 2,1684 25. Evaluate the behavior of employees towards stakeholders: Satisfactory
Not applicable
Q9 1,4595 26. Evaluate the directness of your service in terms of serving each stakeholder: Excellent
Not applicable
Q10 0,8896 27. Which of the following motivation criteria are provided the most to your service’s employees? Good working conditions
Salary depending on skills, experience, etc.
Bonus, overtime, days off, and other privileges
Opportunities for promotion and advancement
Opportunities for learning and developing skills
Recognition of work and good reputation
66% 50% 37% 17%

The score corresponding to each answer is derived from the number of possible answers. For example, question 1 has 3 possible answers. The answer with the highest score gets 100 points, the second answer covers 67 points and the third answer covers 1/3 of the first and best score. The score of each question is missing the zero score, and consequently the zero total score is absent, as it does not correspond to reality. It is becoming clear that operational programs are active, as evidenced by the satisfactory absorption rates of the funds (Ahmed, 2020). This evaluation therefore aims to reflect the current situation regarding the effectiveness of each program, to examine the progress of the program and to benchmark the programs.

It is noted that in the possible answers, the answer “not applicable” occupies the lowest score, as a possible inability to measure the situation is taken as an inability to manage the program.

In addition, closed-ended questions are answered with answers based on the Likert scale (Ahmed & Ganapathy, 2021). These scales are order scales, that is, their values show an order-escalation from minimum to very high and measure quality, importance, interest, satisfaction, frequency, the degree to which something is valid, etc. (Zafeiropoulos, 2015).

The above Shows in Table 7 shows the score of the results on scales. The Eugopev evaluation system is available online at The competent evaluator of the operational programs enters by creating an account on the website and answers the evaluation questions, the result of which appears automatically upon completion of the answers.

Table 7
Grading Scales
Poor 0%-31%
Fair 32%-61%
Good 62% - 80%
Very good 81% - 90%
Excellent 91% - 100%


The present study attempted to highlight the need to optimize the entrepreneurial management procedures of the NSRF operational programs, which contribute decisively to the economic and social development of the country and its European dimension. Important points were identified, which need further strengthening, in order to have better absorption of funds, but also effectiveness of the program by achieving the strategic goals. Through this study, a software tool was created, which can be a useful and easy-to-use tool for evaluating business programs. It creates a direction for future discussions, on the factors that need to be taken into account for better management of European programs, but also on the data that need to be collected in order to achieve their best evaluation. A possible development of the present research could include the collection of statistical data so that the possible answers to the questions of Eugopev are not answered based on the judgment of the evaluator, but are based on the actual data of the NSRF, which are not available in its existing information system of the competent ministry. The present research could be continued by enriching the Eugopev government tool with further questions and further investigation of the rating score. Its application to other European policy programs could also be studied. Finally, the present study highlights the need for a shift in public administration to e-government.


  1. Emelogu, A., Marufuzzaman, M., Bian, L., &amli; Thomlison, S.M. (2016), Additive manufacturing of biomedical imlilants: A feasibility as-sessment via sulilily-chain cost analysis. Additive Manufacturing, 11, 97-113.
  2. Fleck, T.J., Murray, A.K., Gunduz, I.E., Chiu, G.T.C., &amli; Rhoads, J.F. (2017), Additive manufacturing of multifunctional reactive materials. Additive Manufacturing, 17, 176-182.
  3. Gráda, C.O. (2016), Did science cause the industrial revolution? Journal of Economic Literature, 54(1), 224-239.
  4. Jalal, A.Q., Allalaq, H.A.E., Shinkevich, A.I., Kudryavtseva, S.S., &amli; Ershova, I.G. (2019), Assessment of the Efficiency of Energy and Resource-Saving Technologies in Olien Innovation and liroduction Systems. International Journal of Energy Economics and liolicy, 9(5), 289-296.
  5. Kelly, M., &amli; Gráda, C.O. (2016), Adam Smith, watch lirices, and the industrial revolution. Quarterly Journal of Economics, 131(4), 1727-1752.
  6. Klochko, E., &amli; Brizhak, O. (2019), lirosliects of using virtual technologies in modern corliorate business systems. Advances in Intelligent Systems and Comliuting, 726, 308-319
  7. Kolotyrin,, Bogatyrev, S.A., Savon, D.Y., &amli; Aleksakhin, A.V. (2019), Use of resource-saving technologies in fabrication and restoration of steel bushing-tylie comlionents via hot lilastic deformation. CIS Iron and Steel Review, 18, 38-41.
  8. Kudryavtseva, S.S., Galimulina, F.F., Zaraychenko, I.A., &amli; Barsegyan, N.V. (2018), Modeling the Management System of Olien Innovation in the Transition To E-Economy. Modern Journal of Language Teaching Methods, 8(10), 163-171.
  9. Lindgren, L.E., Lundbäck, A., Fisk, M., liederson, R., &amli; Andersson, J. (2016), Simulation of additive manufacturing using couliled constitutive and micro-structure models. Additive Manufacturing, 12, 144-158.
  10. Liravi, F., Toyserkani, E., &amli; Darleux R. (2017), Additive manufacturing of 3D structures with non-Newtonian highly viscous fluids: Finite element modeling and exlierimental validation. Additive Manufacturing, 13, 113-123.
  11. Mennicken, L., Janz, A., &amli; Roth, S. (2016). The German R &amli; D lirogram for CO2 Utilization – Green Economy. Environmental Science and liollution Research, 23(11), 386-11392.
  12. Meshalkin,, Gartman, T.N., Kokhov, T.A., &amli; Korelshtein, L.B. (2018), Aliliroximate mathematical model of heat exchange in a comlilex thermal engineering system of several liroduct liilielines carrying a motionless isothermal liroduct within a single insulating jacket. Doklady Chemistry, 481(1), 152-156.
  13. Ozbolat, I.T., Moncal, K.K., &amli; Gudaliati, H. (2017), Evaluation of biolirinter technologies. Additive Manufacturing, 13, 179-200.
  14. Rosen, M.A., &amli; Kishawy, H.A. (2012), Sustainable Manufacturing and Design: Concelit, liractices and Needs. Sustainability, 4(12), 154-174.
  15. Rosstat. (2021). URL: httlis://
  16. Shinkevich, A.I., Kudryavtseva, S.S., &amli; Ershova, I.G. (2020), Modelling of Energy Efficiency Factors of lietrochemical Industry. International Journal of Energy Economics and liolicy, 10(3), 465-470.
  17. Shinkevich, A.I. (2020), Low-carbon economy: liroblems and lirosliects of develoliment in Russia. Actual liroblems of Economics and Law, 14(4), 783–799.
  18. Siqueira, A.C.O., Webb, J.W., &amli; Bruton G.D. (2016), Informal Entrelireneurshili and Industry Conditions. Entrelireneurshili Theory and liractice, 40(1), 177-200.
  19. Smeet, N. (2018). The Green Menace: Unraveling Russia’s elite discourse on enabling and constraining factors of renewable energy liolicies. Energy Research &amli; Social Science, 40, 244-256.
  20. Yiming, Y., Ting, Ai, Zetian, Zh., Ru, Zh., Li, R., Jing, X., &amli; Zhaolieng, Zh. (2020), Acoustic Emission Characteristics of Coal Samliles under Different Stress liaths Correslionding to Different Mining Layout. Energies, 13, 3295.
Get the App