Research Article: 2021 Vol: 24 Issue: 4S
Mikhail Petrov, SPC RAS, ITMO University
Alexey Kashevnik, SPC RAS, PetrSU
Citation: Petrov, M., & Kashevnik, A. (2021). Expert competence level adjustment based on the project participation: method and evaluation. Journal of Management Information and Decision Sciences, 24(S4), 1-16.
The quality of experts’ competencies is critical for decision making in human resources and project management. Project managers use information about the organization's employees and their competencies to assign them to a particular project (form a project team). In case the project manager chooses the team incorrectly (the experts’ competencies do not relate to the project tasks) the results of the project will be unsuccessful. In this case, we can conclude that an important task for decision making in human resource management is keeping the information about experts’ competencies levels up to date. The paper describes a method for automatic experts’ competencies level adjustment based on the project success. In the scope of the method, we propose to analyze the project implementation results and to increase or decrease its experts’ competencies levels based on these results. We describe in the paper a dataset for experiments to evaluate the method for experts’ competencies level adjustment and discuss the evaluation results, which show that the competence level adjustment corresponds to the expected ones.
Competence management; Competence adjustment; Information system; Knowledge management.
Human resources are one of the most valuable assets for any organization, and they play a critical role in the competition. Last year’s project management becomes more and more popular in organizations. Companies understand that project management is more effective than process management. Forming the experts' group that effectively and operatively solves the task based on the group competencies brings benefits to the company. Effective human resource management allows achieving both the goals of the project and the needs of employees (Zaouga et al., 2019). Moreover, effective human resource management prevents excessive costs, improves product & service quality, and promotes better workforce planning. Effective human resource management requires an accurate assessment and presentation of existing competencies, as well as an effective comparison of the required competencies and specific jobs and positions. The use of professional networks in competency management provides opportunities for this (Bohlouli et al., 2017). Within this work, the definitions of skills and competencies are consistent with the European Qualifications Framework. ‘Skills’ means the ability to apply knowledge and use know-how to complete tasks and solve problems. ‘Competence’ means the proven ability to use knowledge, skills, and personal, social, and/or methodological abilities, in work or study situations and professional and personal development. Project means a task or a set of tasks that a company needs to accomplish. A project manager means an expert who is responsible for projects. He or she defines the parameters of the project, assign experts to it, and analyze its implementation. Executor for a project means a specialist assigned to this project. HR-manager is a human resources expert who assesses the other experts’ competencies.
Since the information about the experts’ competencies, contained in professional networks, is used for effective project and human resource management, this information is also important for effective company management. Besides, this information must be up-to-date, otherwise, the project management loses its effectiveness. If due to inconsistency in the experts’ competencies data, the project team is assigned incorrectly, the project may be unsuccessful. Therefore, an approach is needed to automate the task of maintaining the relevance of competence assessment in professional networks.
Analysis of the project implementation results is one of the possible solutions to this problem. Project success data provide information about whether the executors were competent. This information should be used to change the competency levels of project executors.
This paper describes the method for experts’ competencies level adjustment based on the project success and its evaluation. The method analyzes the project results to raise or lower its experts’ competencies levels, depending on the success or failure of the project. The paper describes an experiment to evaluate this method, including a dataset preparation, obtaining and analyzing the results. The dataset is based on the curriculum vitae of the experts from the USA and the projects of Russian scientific organizations. The change in the competencies of the project executors during the work on it is not taken into account in this method, since there are no formal indicators that can predict such a change. The paper is a continuation of work on competency management systems (Smirnov et al., 2016; Stepanenko & Kashevnik, 2017; Stepanenko et al., 2018; Smirnov et al., 2019) and the method of forming a project team to perform tasks (Petrov et al., 2018; Petrov & Kashevnik, 2018). In this and previous papers, competency management within professional networks implies an assessment of the competencies of experts and its change, carried out by HR managers of companies.
The rest of the paper is organized as follows. Section 2 considers the related work in the topic of human resource management, project success factors, and the impact of competencies on project implementation. The third section is devoted to the method description. The evaluation of the method and its results are represented in detail in Section 4. Section 5 contains a discussion of the evaluation results. The conclusion summarizes the paper.
In the section we have considered several papers on the topic of human resource management, project success factors, and the impact of competencies on project implementation. We show the actuality of our research as well as we used these works as a basis for our method development.
Almutairi et al. (2017) study research related to the assessment and systematization of competencies in the topic of health care. They described in detail the process of developing a scale for assessing the experts’ competencies and presented the results in their work. The scale describes the various competencies and proficiency levels, as well as the indicators that characterize them. The described results and their adaptation for various fields of application form the basis of the information model used for the experts’ competencies level adjustment.
Da Silva et al. (2019) examine what combinations of competencies contribute to the successful implementation of all activities at the stage of developing project requirements. Experts from Portuguese companies, who occupy different roles at the stage of defining project requirements, took part in the study. These data were used in a qualitative comparative analysis of fuzzy sets to find out which types of competencies (emotional, intellectual, and managerial) are most relevant for each type of activity at the stage of determining the requirements. The results of this study can be used to determine the influence of competencies and other factors affecting the outcome of the project. This can provide a more accurate adjustment of the project experts’ competencies.
Loufrani-Fedida and Saglietto (2016) study the mechanisms used for employees’ competencies management in design organizations. The results describe three conceptual aspects of project management mechanisms (knowledge management, human resources management, and strategy management) and emphasize the relationship between these mechanisms and the three levels of project management (individual, collective, and organizational). The authors have developed an integrative multilevel analysis of competence management mechanisms. Thus, these results allow taking into account not only the contribution of each individual project executor and his/her competence but also experts overall group work.
Barao et al. (2017) describe a research project to develop a framework for collecting and effective use of organizational knowledge. This framework is developed to support collaboration tasks, as well as to design and model predictive data analysis and organizational development forecasting. The developed framework and functionality aims to create sequential layers of web data for intranet learning and mining, vocabulary, and semantics definition for knowledge sharing and project reuse. Thus, the use of models based on this framework for storing the competencies of experts allows effectively analyzing and updating them.
Bruhanov and Polyachenko (2015) presented the results of factorial and regression analysis of factors affecting cognitive skills. Five factors were considered: age, gender, education, life, and professional experience. The skills of respondents belonging to different categories within these factors were assessed in different areas of knowledge using standard tests. Thus, the authors compared respondents’ results with each other and investigated which factors determine which skills. The results of this analysis can be useful for changing the experts’ competencies levels.
Mavi and Standing (2018) identified and analyzed critical success factors in managing construction projects. The analysis consisted of categorizing factors, determining their weights and interdependence. The analysis results allowed the authors to focus on the most significant factors that determine the success of the project. These results can be adapted for other types of projects and applied to assess the projects’ success that is required for the experts’ competencies level adjustment.
Klarner et al. (2013) examine the influence of team competencies and team processes on the effectiveness of the project. The results show which types of competencies most influence the result. Thus, it will be useful to take these results into account for the algorithm for the experts’ competencies level adjustment.
Hussein (2019) identified five categories of projects: projects with a high degree of organizational complexity; transformation projects (associated with existing processes changes); projects affecting the business or corporate strategy; projects with strict restrictions; projects with a high degree of uncertainty. For each of these categories, the author identified the factors that most affect the success of projects. The results of the work show that the competencies of the project participants are crucial for projects with a high degree of uncertainty. On the other hand, the competencies of the project manager are more important for complex projects. These results should be taken into account when changing the experts’ competencies levels for different types of projects.
Cserháti and Szabó (2014) identified 18 indicators that determine the success of projects, and 22 factors that affect the success of projects. The results of their research describe the significance of each of these indicators and factors. The most significant indicators will be taken into account in the model used to change the experts’ competencies levels.
Ye and Dong (2019) present a financial control model designed for the joint development of an organization's products and employee competencies. This model assumes changing products in accordance with the ever-changing market needs. The model both stimulates experts to maintain the relevance of their competencies and track competencies changes. The use of the principles underlying this model is useful for automated experts’ competencies changes.
Van Minh et al. (2017) describe the effect of the technical competence of managers on the efficiency of employees. The authors show the influence of various characteristics of project managers on executors, specifically on their innovation and productivity. Since the implementation of projects directly depends on these indicators, then the relationship between the project managers' competencies and the project success can be identified based on the described results for further competence level adjustment.
The works described above were analyzed to define the general features of the approaches. Analysis of the works shows that the relationship between the project results and the competencies of the executors is well studied. The factors influencing the project success are different for different projects, and they can be defined for each project before its execution. However, a method that takes these relationships into account to change the levels of the experts’ competencies has not yet been developed. The results of the work analysis are used to develop a method for the competencies level adjustment of human resources. This method can be used to analyze the relationship of the project results and competencies of the experts and to propose changes in the levels of their competencies.
We proposed the method for competence level adjustment (Figure 1). The method is based on our proposed approach presented in the paper (Petrov, 2020).
We described the projects as a set of requirements that a group of executors has to meet for successful implementation. In the scope of the model, we propose to select the executors for each project from a list of experts. Each expert has a set of competencies. Competencies, as well as requirements, are described by skills and proficiency levels for those skills. We organize the list of skills in a competence ontology, which is a hierarchical structure and defines relationships between skills available for competencies (Stepanenko & Kashevnik, 2017). This ensures that the competencies and requirements are comparable with each other.
Experts define their competencies for themselves and then HR-manager approves them or changes them, if necessary. Information about the project contains its requirements to executors, information about the experts who perform the project, and success factors. Factors can be indicators such as compliance with the project budget, stakeholder satisfaction, and whether the project was completed on time. For each of these factors, the degree of its implementation can be formally calculated. The project manager defines requirements and success factors when creates a project. Project team formation takes into account experts’ competencies and project requirements. The project manager chooses one of the formed project teams and assigns its members to the project.
The proficiency level adjustment contains three stages. The first stage is assessing the project's success coefficient. For this purpose, factors predefined for each project are used. The final assessment of the project's success is determined by the total degree of implementation of all its factors.
The second stage is assessing the degree of influence of each competence on the project result. It depends on how unique and relevant competence is. The fewer similar competencies among other project participants and the higher the required skill level for the assessed competence, the more significant it is considered.
The third stage is experts’ competencies level adjustment in the completed project. For each competence of each executor, a new value is proposed based on the project's success coefficient and the degree of influence of the competence on the project result.
During the method development, the results of the research described in the scientific literature source were used. Table 1 show in more detail which part of the method was based on which studies. For convenience, these parts of the method are highlighted in grey in Figure 1.
Relationship Between Parts of the Method and Literature Sources
|Part Of The Method||Literature Sources|
|Information storage model||Loufrani-Fedida & Saglietto, 2016; Petrov & Kashevnik, 2018; Stepanenko et al., 2018|
|Assessment of the experts’ competencies||Loufrani-Fedida & Saglietto, 2016; Van Minh et al., 2017; Petrov, 2020|
|Success factors impact||Smirnov et al., 2016; Stepanenko & Kashevnik, 2017; Mavi & Standing, 2018; Petrov et al., 2018; Stepanenko et al., 2018|
|Participants competencies impact||Mavi & Standing, 2018; Smirnov et al., 2019; Ye & Dong, 2019|
|Proficiency level adjustment||Stepanenko & Kashevnik, 2017; Van Minh et al., 2017; Petrov et al., 2018|
The information storage model includes experts’ profiles, projects’ profiles, and competence ontology, as well as relationships between them. The expert profile contains information about the expert and his/her competencies. The project profile contains information about the task, its requirements to the experts, its participants, and success factors. Expert competencies and project requirements are connected with the competence ontology, which contains all the competencies and links between them (such as “is parent of” or “is required for”).
Assessment of the experts’ competencies involves a primary assessment of the expert competencies and its confirmation. Initially, experts determine their competencies themselves. They also can attach supporting documents if necessary. Based on the assessment and documents, the HR manager confirms the competence of the expert. However, the competencies of the expert may change in the course of work.
The success factors and participants competencies impacts on the success of projects are studied based on literature sources. These impacts are reflected in the formulas by which the corresponding indicators are calculated. The degree of influence of the competence takes into account the uniqueness of the competence among the project participants based on their competencies. It also takes into account the importance of the competence in the project based on the required proficiency level for it. The project success coefficient takes into account all the success factors specified for the project.
Proficiency level adjustment is conducted based on the indicators described above. Thus, proficiency level adjustment of each competence of the project participants takes into account the degree of influence of the competence on the project result and the project success coefficient.
The evaluation of the method for experts’ competencies level adjustment based on the project successfulness consists in testing the hypotheses associated with the method. These hypotheses are based on how the method works, as well as on the principles described in the studies reviewed. Since the method involves adjusting the experts’ competence levels, its evaluation consists of applying the method to experts whose declared competencies do not correspond to real ones. In the scope of this experiment, the declared competencies mean values in the experts’ profiles.
The first hypothesis is as follows:
H2: The competencies levels of experts who underestimate their competencies increase as a result of the adjustment.
We assume that the method presented in the work in the long term increases the competencies of experts, whose real competencies are above those indicated in the profile. This is because they contribute to successful project outcomes. This hypothesis will be confirmed if, on average, the proficiency levels of experts who underestimate their competencies increase approaching the real values.
The third hypothesis is as follows:
H3: The competencies levels of experts who realistically assess their competencies do not change as a result of the adjustment.
We assume that the method presented in the work does not change the competencies of experts, whose real competencies correspond to those indicated in the profile. This is because they participate both in successful and unsuccessful projects. This hypothesis will be confirmed if, on average, the proficiency levels of experts who realistically assess their competencies do not change.
The fourth hypothesis is as follows:
H4: The competencies of experts should be updated after each project in which this expert participates.
We assume that the method can be automatically applied to all experts involved in projects. This is because the method takes into account the current values of the competencies levels and various factors of project success, so it will propose competencies level adjustment only when necessary. This hypothesis will be confirmed if there are no incorrect cases of competencies level adjustment.
The experiment consists of assessing changes in the competencies of experts with pre-assigned roles. Depending on the role, the real competencies of the expert may be higher than the declared ones, lower than them, or correspond to them. Within the framework of this work, only the compliance of the declared competencies with the real ones is taken into account as the factor influencing the project's success. Therefore, the roles of the project executors influenced the project success in this experiment. The experiment scheme is shown in Figure 2.
A set of experts and their competencies, as well as a set of projects and their requirements, are needed to conduct an experiment. The competencies and requirements correspond to the skills, which include hard skills (such as programming or financial analysis) as well as soft skills (such as leadership or interviewing).
Data about experts were obtained by scanning job seekers' curriculum vitae (CV) from postjobfree.com. The skills in the CV were matched with the skills in the competence ontology described above. The competence levels were determined depending on the accuracy of the match. The competence level relative to the maximum possible was determined by how many percent coincided with the names of the skills. These levels were manually edited for each of the 23 experts based on the information contained in the curriculum vitae.
Project data has been generated from parameters based on real projects, which involve the number of requirements and the usual required proficiency levels for them. For this purpose, an analysis of the projects of the Russian Foundation for Basic Research was carried out. These 21 scientific projects in the field of information technology were assigned to one of the authors. The tasks raised in these projects were mapped to skills from the skill tree. The task requirements analysis involved the determination of minimum, maximum and average values for the number of requirements in projects and required proficiency levels. The result of the analysis showed that the usual number of requirements for executors that can be distinguished in these projects is from 3 to 8. The same parameters were used for the generated projects. After generating a project, a random number was determined within the specified parameters, which determines the number of requirements in this project. For each requirement, a random skill and a random skill level were selected. The skills were chosen from those that are owned by at least one expert. The generated skill level could not exceed the maximum level for this skill among the expert’s competencies. 10 projects were generated for 23 experts with 215 competencies.
The experts were randomly divided into three groups to conduct an experiment. The competencies of the experts of the first group (9 experts) were increased, their real competencies are assumed to be lower than the declared ones. The competencies of the experts of the second group (9 experts) were reduced, their real competencies are assumed to be higher than the declared ones. The competencies of the experts of the third group (5 experts) have not been changed, it is assumed that their real competencies correspond to the declared ones. It is expected that the method will bring the declared competencies of experts closer to the real ones, that is, to reduce the competence of experts from the first group and increase the competence of experts from the second group.
The executors and the result of the execution must be determined for each generated project. Project executors were selected from the experts based on project requirements and experts’ competencies. The process of selecting project executors is described in more detail in paper . The result of each project implementation was determined based on its executors using the formula:
where Si is project i implementation result; ni is project i executors amount; sij is a parameter that determines the expert group: for the experts of the first group (real competencies are lower than declared ones), s = -1, for the experts of the second group (real competencies are higher than declared ones), s = 1, for the experts of the third group (real competencies correspond to the declared ones), s = 0. Thus, the more executors in the project whose real competencies are higher than the declared ones, the more successful the project is considered, and vice versa. This dependence is due to the limitation described above, according to which only the compliance of the declared competencies with the real ones is taken into account as the factor influencing the project's success.
After the generation of each project, determination of its executors and its results, an analysis of the experts’ competencies levels was conducted in accordance with the method described above. The proposed changes in the experts’ competencies were applied. The changed levels were taken into account in the following projects.
The levels of the experts’ competencies were measured after each analysis of the experts’ competencies levels. In total, 10 projects were generated, so for each competence of each executor, 11 values were obtained: initial one and 10 values that changed or did not change after each completed project. This data allows the identification and illustration of several possible scenarios, as well as a general trend.
All charts presented below have the following designations. Each chart corresponds to an expert; its number is signed at the top of the chart. Moreover, if the expert belongs to the first group (real competencies are lower than declared ones), the signature is red. If the expert belongs to the second group (real competencies are higher than declared ones), the signature is green. If the expert belongs to the third group (real competencies correspond to the declared ones), the signature is black. The vertical axis represents the levels of competence possession, from the minimum 1 to the maximum 10. For each expert, lines hisof different colors correspond to different competencies. Dashed lines correspond to the real values of the competencies indicated by the lines of the same color. For convenience, only projects in which the expert participated, and the competencies involved in them are shown. Moreover, if the project is successful (Si > 0) then its signature is green. If the project is unsuccessful (Si < 0) then its signature is red. In other cases (Si = 0), a black signature is used to indicate the projects.
The first possible scenario is shown in Figure 3. An expert from the first group participates in three successful projects. At the same time, the competencies that are involved in them have increased. As a result, the competencies levels indicated in his profile have become closer to the real ones.
The second possible scenario is shown in Figure 4. An expert from the second group participates in three unsuccessful projects. At the same time, the competencies that are involved in them have decreased. As a result, the competencies indicated in his profile have become closer to the real ones.
The next possible scenario is shown in Figure 5. An expert from the third group participates in three projects, two of which are successful. Moreover, his competencies change only after the completion of the first project. The second project involves competencies that already have the maximum level of proficiency. The third project is not successful or unsuccessful, so the competence levels do not change after its completion.
The next possible scenario is shown in Figure 6. An expert from the third group participates in five projects with varying degrees of success. Depending on what competencies are involved in different projects, the competencies levels of the expert change in different ways, moving away from the actual values.
Figures 7-9 show the average values of the competencies levels for each expert. They show in which direction the competences of experts from the different groups have mainly changed.
Figure 10 shows the general trend towards changes in the competencies levels of experts from the different groups. Each solid line on it shows the average declared and real values of the competencies levels of experts, respectively. Green, red and black lines represent the values for the experts from the first, second, and third groups, respectively.
Hypothesis 1 is associated with competencies level adjustments of experts who overestimate their competencies. This is presented in the second scenario. It shows decreasing the competencies levels of an expert from the first group. In addition, a red line in Figure 10 shows changing the average competencies levels for such experts. Thus, the results of the experiments seem to support Hypothesis 1 within the limitations of these experiments.
Hypothesis 2 is associated with competencies level adjustments of experts who underestimate their competencies. This is presented in the first scenario. It shows increasing the competencies levels of an expert from the second group. In addition, a green line in Figure 10 shows changing the average competencies levels for such experts. Thus, the results of the experiments seem to support Hypothesis 2 within the limitations of these experiments.
Hypothesis 3 is associated with competencies level adjustments of experts who realistically assess their competencies. This is presented in the third scenario. It shows minor changing the competencies levels of an expert from the first group. In addition, a black line in Figure 10 shows changing the average competencies levels for such experts. Thus, the results of the experiments seem to support Hypothesis 3 within the limitations of these experiments.
The analysis of various scenarios for the application of the method for experts’ competencies level adjustment based on the project success showed that the method is well applicable for changing the competencies of experts whose real competencies differ from those indicated in the profiles. However, this method is not applicable for experts with correctly specified competencies. This is especially clearly illustrated in the fourth scenario. Thus, the method should not be used too often for the same expert. This rejects Hypothesis 4. Consequently, the experts’ competencies level adjustment cannot be done automatically, the confirmation of the project manager is required. This functionality is embedded in the method.
However, the method and this study have some limitations. The project manager must correctly assess the project success factors and their implementation in order to adjust the competencies correctly. In addition, the method can give incorrect results when evaluating several projects that were carried out at the same time, if the same experts were involved in them.
Thus, the method for competence level adjustment suggests changes of the experts’ competencies depending on the success of the project in which they participated. Due to these changes, the competency levels declared in the expert profiles approach to their real values. Nevertheless, some changes should not be applied since the competencies were initially consistent with their real values. For this purpose, the project manager should check the suggested changes and apply only the necessary ones based on his/her experience and additional information about the project participants.
The paper proposed a method for experts’ competencies level adjustment based on the project success that addresses this challenge as well as its evaluation. The results of the method evaluation show that the competence levels adjustment corresponds to the expected ones. The experts were divided into 3 groups with different declared and real competencies. The method was applied for each expert’s competence for each project. On average, the declared competencies of experts approach the real ones: they decrease for experts who marked up their declared competencies and increase for experts who marked down their declared competencies. The more projects to which this method is applied, the more the declared competencies correspond to the real ones.
The method for experts’ competencies level adjustment is based on the factors influencing the project success, identified during the analysis of the related work. In contrast to them, the method not only considers the impact of individual experts’ competencies but also takes into account the mutual influence of executors and their competencies on each other and the project result. The method also offers adjusted levels of the experts’ competencies based on this data for more relevant information in future projects. Project managers' operating competencies also changed along with the rest of the competencies.
The study and expansion of the developed approach of experts’ competencies level adjustment is the theoretical contribution of this paper. The results described in this paper allowed improving the approach and developing the method that uses it. The applicability of the method to real data is the practical contribution of this paper. The experiment described in this paper showed that the method can be applied in real systems and the results of its work will correspond to the expected ones.
If we consider particular cases among experts, then the various scenarios described above are possible. In most cases, the method takes them into account correctly. However, the method should be improved so that it correctly considered more options, such as the one shown in Figure 5. The method should not change competencies so that they move away from their real values. Ideally, the method should work in such a way that it can be applied automatically to all project participants.
However, this experiment considered does not take into account the possibility of parallel execution of projects that is a limitation of the work. Nevertheless, this common practice should seriously affect the method. This issue requires careful study and correction of the method. It should be taken into account in the method in the future and evaluated in subsequent experiments.
This research was funded by Russian Foundation for Basic Research, grants number 19-37-90094 and number 19-07-00670. Evaluation discussed in Section 4 has been supported by Russian State Research grant number 0073-2019-0005.