Research Article: 2019 Vol: 22 Issue: 3

**Do Anh Duc, Vietnam National University**

**Pham Ngoc Thach, Hanoi University**

**Bui Hong Phuong, Vietnam National University**

**Canh Chi Dung*, Ministry of Education and Training**

**Luu Huu Van, Vietnam National University**

**Pham Thi Hong Diep, Vietnam National University**

**Citation Information:** Duc, D. A., Thach, P. N., Phuong, B. H., Dung, C. C., Van, L. H. & Diep, P. T. H. (2019). A dynamic fuzzy multiple criteria decision-making approach for lecturer performance evaluation. *Journal of Management Information and Decision Sciences, 22*(3), 250-261

Performance evaluation is an important activity of any organization in improving the quality of work input, inspiring staffs making them more engaged. The purpose of the paper is to develop a technique to support the process of evaluating lecturer’s performance through the use of a new Dynamic Fuzzy Order Preference by Similarity to Ideal Solution (DFTOPSIS). In the proposed approach, the ratings of alternatives and importance weights of criteria for lecturer’s performance evaluation are represented by triangular fuzzy numbers in time sequence. A closeness coefficient determines the ranking order of alternatives by calculating the distances of alternatives to both the ideal and negative-ideal solutions. An application for lecturer’s performance evaluation is given to demonstrate the applicability and efficiency of the proposed approach.

Dynamic TOPSIS, MCDM, lecturer performance evaluation, fuzzy sets theory.

Performance evaluation is the process of determining, measuring and developing human performance in an organization or company (Rouyendegh et al., 2018). Several studies in literature have investigated performance evaluation in educational institutions or organizations. EricJacquet-Lagreze & Siskos (2001) presented a panorama of preference disaggregation methods and summarized the most important results and applications over the last 20 years. They found that the preference disaggregation methods have been applied in several real world decision-making problems from the fields of financial management, marketing, environmental management, as well as human resources management. Grigoroudis & Siskos (2002) proposed the multi-criteria satisfaction analysis (MUSA) method for measuring and analyzing customer satisfaction. They also presented the reliability analysis of the provided results, along with a simple numerical example that demonstrated the implementation process of the MUSA method. Siskos et al. (2005) presented a set of utility functions (UTA method) and its variants and summarized the progress made in this field. The detailed presentation of the basic UTA algorithm was presented, including discussion on the stability and sensitivity analyses.

Wiswall (2013) evaluated the importance of various modeling restrictions to the key findings of teacher quality dynamics. Using data covering all 5^{th} grade public school teachers from the state of North Carolina and an unrestricted experience model, the author found that for mathematics achievement there were high returns to later career teaching experience, about twice as much dispersion in initial teacher quality as previously estimated and a pattern of negative selection where high quality teachers are more likely to exit. Skordoulis et al. (2014) determined the factors that affect the student’s satisfaction in the case of a technological education institute. The factors of satisfaction derived from DREEM and LOT-R questionnaires are the process of learning, the environment of the institute, the social life within it, the student’s opinion about their teachers, the student’s opinion about their fellow students, the student’s positive thoughts, their negative thoughts and, the use of their time. Skordoulis et al. (2016) examined the satisfaction of students from a higher education institute using a multi-criteria satisfaction analysis method. Their results indicated that student’s satisfaction is correlated with gender, semester, department of studies and their performance. Rahman (2016) facilitated the perception of the students in regard to the pedagogical action of a good lecturer. Based on the findings, the subject’s concepts of pedagogical competence are (1) The lecturer should have responsibility with the material they hold, (2) The learning objectives should be prepare well and it had to be achieved, (3) Accommodating the students need, (4) Controlling the classroom situation. Pedagogical factor becomes the most important factor to be considered in teaching performance, in the students’ perception, pedagogical competence of good lecturer should cover the readiness in delivering the lesson, how they enliven the classroom condition, the application of media and technology in the classroom, giving feedback and evaluation skill. Findawati et al. (2018) applied a Naive Bayes Classifier method to classify the teacher performance feasibility.

In educational institutions, a lecturer’s performance evaluation plays a vital role in research and teaching activities. Evaluating a lecturer’s performance is often conducted periodically at educational institutions. Universities and/or research organizations often use on the results of the evaluation in decision making process such as promoting lecturers or funding research. Without reliable performance evaluation systems, good performers may not receive adequate positive feedback and become frustrated and leave, causing the organization to incur high recruitment costs (Anisseh et al., 2009; Falsafi, 2011). Hirsch (2005) indicated that for the few scientists who earn a Nobel Prize, the impact and relevance of their research is unquestionable. However, evaluation of other scientists including lecturers, assessing their performance is a difficult and sensitive issue which has quantitative and qualitative aspects, complexity and imprecision (Hirsch 2005, Ate? et al., 2006). This is a complicated process because an individual's performance is of multi-dimensional characteristics and has many criteria.

Recognizing the role and importance of the lecturer’s performance evaluation, several studies have investigated in literature. Some studies have proposed the quantitative and qualitative criteria in order to evaluate a lecturer’s performance such as numbers and quality of publications, personality characteristics, participation in professional society, teaching experience, fluency of foreign language, student evaluation, etc. (Ramli et al., 2010, Rama 2011, Costa & Oliveria, 2012; Phiakoksong et al., 2013; Ghonji et al., 2015; Dat et al., 2019; Thong et al., 2019). There have been many studies that investigated the method of evaluation. Traditional methods included collecting feedback from students, online surveys, group discussion, gathering data as part of personnel decisions,etc (Jauch & Glueck, 1975; Ellington & Ross, 1994; Sproule, 2002; Paulsen, 2002). Many researchers have concerted that evaluating a lecturer’s performance is a multi-faceted exercise that need a high level of precision. In addition, since the judgments from experts are usually vague rather than crisp, a judgment should be expressed by using fuzzy sets which has the capability of representing vague data. Therefore, a lecturer’s performance evaluation is the fuzzy multi-criteria decision-making (MCDM) problem. Although several studies have proposed different models to determine the importance weights of criteria and lecturer’s performance (Phiakoksong et al., 2013; Hassna & Raza, 2011; Huang et al., 2013; Kusurkar et al., 2013; Zahir & Fakhri, 2011), few have used fuzzy sets and a multiattribute method together in the process of evaluating a lecturer’s performance.

Due to the complexity of the issue in real-life, the input data for a lecturer’s performance can be from different time or periods. Therefore, it is necessary to develop a dynamic decision making process from such situations (Li et al., 2015). There is a limited number of studies that utilized the dynamic MCDM model for the evaluation. In this paper, a dynamic fuzzy Technique for Order Preference by Similarity to Ideal Solution (DFTOPSIS) is proposed for the work. In the proposed approach, the ratings of alternatives and importance weights of criteria are represented by triangular fuzzy numbers in time sequence. A closeness coefficient determines the ranking order of alternatives by calculating the distances of alternatives to both the ideal and negative-ideal solutions. An application for a lecturer’s performance evaluation is provided to demonstrate the applicability and efficiency of the proposed approach.

In this paper, Section 2 reviews the basic concepts of fuzzy numbers. Section 3 proposes a DFTOPSIS approach. The proposed DFTOPSIS approach is applied to solve the problem in the evaluation of a lecturer’s performance Section 4. Finally, conclusions are drawn in Section 5.

There are various ways of defining fuzzy numbers. This paper defines the concept of fuzzy numbers as follows (Dubois & Prade, 1978):

**Definition 1**

A real fuzzy number A is described as any fuzzy subset of the real line R with membership function A(x) that can be generally be defined as:

(a) *f _{A}* is a continuous mapping from R to the closed interval [0,ω];

(b) *f _{A}* (x) = 0 for all x ε (-∞,a]

(c) *f _{A}* is strictly increasing on [a,b];

(d) *f _{A}* (x)=ω, for all x ε [b,c];

(e) *f _{A}* is strictly decreasing on [c,d];

(f) *f _{A}* (x) = 0 for all x ε (d, ∞],

Where a, b, c and d are real numbers. Unless elsewhere specified, this research assumes that A is convex and bounded (i.e., -∞ < a,d < ∞).

The membership functions *f _{A}* of trapezoidal fuzzy number A can be expressed as:

(1)

Where and are the left and right membership functions of A, respectively. When b = c, the trapezoidal fuzzy number is reduced to a triangular fuzzy number and can be denoted by A = (a,b,d). Thus, triangular fuzzy numbers are special cases of trapezoidal fuzzy numbers.

**Definition 2**

**α-cuts**

The α-cuts of a fuzzy number A can be defined as where Aα is a non-empty bounded closed interval in R and can be denoted by and are its lower and upper bounds, respectively (Kaufmann & Gupta, 1991). For example, the α-cuts of the triangular fuzzy number A = (a,b,d) can be expressed as:

(2)

**Definition 3**

**Arithmetic operations on fuzzy numbers**

Given fuzzy numbers A and B, where A, B ε R+, the α-cuts of A and B are and, respectively. By the interval arithmetic, some main operations of A and B can be defined as follows (Kaufmann & Gupta, 1991):

(3)

(4)

(5)

(6)

(7)

This section develops a dynamic fuzzy TOPSIS approach the lecturer’s performance evaluation process by the following procedure:

**Aggregate Ratings of Alternative versus Criteria**

Assume that a committee of k decision makers (D_{v},v=1,...,k) is responsible for evaluating *m* alternatives (A_{i}, i=1,...m) under *n* selection criteria (Cj, j = 1,...n) in time sequence , t_{h}, h=1,...1 dynamic fuzzy TOPSIS problem can be expressed in matrix format as:

Let be the suitability rating assigned to alternative , A_{i} by decision maker , D_{v} for subjective , C_{j} in time sequence t_{h}. The averaged suitability rating can be evaluated as:

(8)

**Aggregate the Importance Weights**

Let be the weight assigned by decision maker D_{v} to criterion , C_{j} in time sequence t_{h}. The average weight, , of criterion C_{j} assessed by the committee of k decision makers can be evaluated as:

(9)

**Construct the Weighted Fuzzy Decision Matrix**

Considering the different weights of each criterion, the weighted decision matrix can be computed by multiplying the importance weights of evaluation criteria and the values in the normalized fuzzy decision matrix. The weighted decision matrixes G_{i} in time sequence t_{h} are defined as:

(10)

**Calculation of**

The fuzzy positive-ideal solution (FPIS, A^{+}) and fuzzy negative ideal solution (FNIS, A^{-}) are obtained as:

A^{+} = (1,1,1) (11)

A^{-} = (0,0,0) (12)

The distance of each alternative, A_{i}, i=1,...m from A^{+} and A^{-} in time sequence , t_{h} is calculated as:

(13)

(14)

Where represents the shortest distance of alternative , A_{i} and represents the farthest distance of alternative A_{i}.

**Obtain the Closeness Coefficient**

The closeness coefficient of each alternative in time sequence , t_{h}, which is usually defined to determine the ranking order of all alternatives, is calculated as follows:

(15)

A higher value of the closeness coefficient indicates that an alternative is closer to PIS and farther from NIS simultaneously. The closeness coefficient of each alternative is used to determine the ranking order of all alternatives and identify the best one among a set of given feasible alternatives.

This section applies the proposed DFTOPSIS approach in the case of a university in Vietnam, i.e., University of Economics and Business-Vietnam National University (UEB-VNU). The managers of this University have become confused on how to find out the best lecturer at the end of year. The proposed approach was applied to the process of lecturer’s performance evaluation of this university to help it assessing their lecturers and test the efficacy of the proposed approach. The input data of the proposed approach were collected by conducting semistructured interviews with the top managers of the university. After preliminary screening, fixed lecturers L_{1}, L_{2}, L_{3}, L_{4} and L_{5}, are chosen for further evaluation. A committee of three decision makers, D_{1}, D_{2}, and D_{3}, is responsible for the evaluation of the five lecturers and the selection of the best lecturer. Six evaluation criteria are considered including personality characteristics (EC_{1}), students evaluation (EC_{2}), total number of publications (EC_{3}), participation in professional society (EC_{4}), classroom teaching experience (EC_{5}), and fluency in a foreign language (EC_{6}) . The computational procedure is as follows:

**Aggregate Ratings of Alternatives versus Criteria**

In this study, the decision makers use the linguistic rating set in t={t_{1}, t_{2}, t_{3}}, S = {VP, P, F, G, VG}, where VP=Very Poor=(0.0, 0.1, 0.3), P=Poor=(0.1, 0.3, 0.5), F=Fair=(0.3, 0.5, 0.7), G=Good=(0.5, 0.7, 0.9) and VG=Very Good=(0.8, 0.9, 1.0), to evaluate the suitability of the lecturer’s performance under each evaluation criteria. **Table 1** presents the suitability ratings of five candidates (L_{1},…, L_{5}) versus six criteria (EC_{1},…, EC_{6}) from three decision-makers (D_{1}, D_{2}, D_{3}) using the DFTOPSIS. Using Eq. (8), the aggregated ratings of the candidates versus the criteria from the decision-makers are shown in the last column of **Table 1**.

Table 1 Aggregated Ratings of Lecturers Versus Criteria |
|||||||||||

Evaluation Criteria |
Lecturers |
Decision makers |
Aggregated ratings |
||||||||

t _{1} |
t _{2} |
t _{3} |
|||||||||

D1 |
D2 |
D3 |
D1 |
D2 |
D3 |
D1 |
D2 |
D3 |
|||

EC_{1} |
L_{1} |
G | G | VG | VG | G | VG | G | VG | VG | (0,547, 0,730, 0,941) |

L_{2} |
VG | VG | G | G | VG | G | VG | G | G | (0,586, 0,785, 0,981) | |

L_{3} |
VG | G | G | G | G | G | G | G | VG | (0,510, 0,685, 0,915) | |

L_{4} |
VG | G | VG | VG | VG | VG | VG | VG | VG | (0,586, 0,785, 0,981) | |

L_{5} |
G | VG | VG | G | G | VG | G | G | VG | (0,623, 0,830, 1,007) | |

EC_{2} |
L_{1} |
VG | G | G | VG | G | VG | VG | G | G | (0,510, 0,685, 0,915) |

L_{2} |
G | G | G | VG | G | G | G | G | VG | (0,470, 0,630, 0,874) | |

L_{3} |
G | VG | G | G | G | VG | G | G | G | (0,547, 0,730, 0,941) | |

L_{4} |
VG | G | VG | G | G | VG | G | G | VG | (0,586, 0,785, 0,981) | |

L_{5} |
G | VG | G | G | VG | G | VG | VG | G | (0,547, 0,730, 0,941) | |

EC_{3} |
L_{1} |
G | G | F | G | F | F | G | G | G | (0,409, 0,548, 0,788) |

L_{2} |
G | VG | G | G | G | VG | G | VG | VG | (0,547, 0,730, 0,941) | |

L_{3} |
F | G | G | G | G | G | F | G | F | (0,438, 0,585, 0,825) | |

L_{4} |
VG | G | VG | VG | G | VG | G | G | VG | (0,586, 0,785, 0,981) | |

L_{5} |
F | F | G | G | F | G | G | G | F | (0,377, 0,504, 0,738) | |

EC_{4} |
L_{1} |
G | G | G | G | G | G | G | VG | G | (0,470, 0,630, 0,874) |

L_{2} |
G | F | G | G | F | G | G | F | F | (0,409, 0,548, 0,788) | |

L_{3} |
F | G | G | F | G | G | G | F | G | (0,438, 0,585, 0,825) | |

L_{4} |
VG | G | G | VG | G | G | VG | G | VG | (0,510, 0,685, 0,915) | |

L_{5} |
G | F | G | G | F | G | G | G | F | (0,409, 0,548, 0,788) | |

EC_{5} |
L_{1} |
VG | G | G | G | G | G | G | VG | VG | (0,510, 0,685, 0,915) |

L_{2} |
G | G | G | G | VG | G | G | VG | G | (0,470, 0,630, 0,874) | |

L_{3} |
G | G | G | G | G | G | G | F | G | (0,470, 0,630, 0,874) | |

L_{4} |
VG | G | VG | G | G | VG | G | VG | VG | (0,586, 0,785, 0,981) | |

L_{5} |
G | G | G | G | VG | G | G | VG | G | (0,470, 0,630, 0,874) | |

EC_{6} |
L_{1} |
G | G | G | G | G | VG | G | VG | G | (0,470, 0,630, 0,874) |

L_{2} |
VG | VG | G | G | VG | G | G | VG | G | (0,586, 0,785, 0,981) | |

L_{3} |
F | G | G | F | G | F | F | F | G | (0,438, 0,585, 0,825) | |

L_{4} |
VG | G | G | G | G | VG | G | G | G | (0,510, 0,685, 0,915) | |

L_{5} |
G | F | G | G | G | F | G | F | G | (0,409, 0,548, 0,788) |

**Aggregate the Importance Weights**

The decision makers employ a linguistic weighting set in t={t_{1}, t_{2}, t_{3}}, Q = {UI, OI, I, VI, AI}, where UI=Unimportant=(0.0, 0.1, 0.3), OI=Ordinary Important=(0.2, 0.3, 0.4), I=Important=(0.3, 0.5, 0.7), VI=Very Important=(0.7, 0.8, 0.9) and AI=Absolutely Important=(0.8, 0.9, 1.0), to assess the importance of all the criteria. **Table 2** displays the importance weights of the six criteria from the three decision-makers. The aggregated criteria weights obtained by Equation (9) are shown in the last column of **Table 2**.

Table 2 Aggregated Weights of Criteria |
||||||||||

Evaluation Criteria |
Decision makers |
Aggregated weights |
||||||||

t_{2} |
t_{2} |
t_{3} |
||||||||

D_{1} |
D_{2} |
D_{3} |
D_{1} |
D_{2} |
D_{3} |
D_{1} |
D_{2} |
D_{3} |
||

EC_{1} |
AI | VI | VI | VI | VI | VI | VI | AI | VI | (0,720, 0,820, 0,920) |

EC_{2} |
I | I | I | OI | I | I | I | VI | I | (0,300, 0,500, 0,700) |

EC_{3} |
AI | VI | AI | AI | VI | VI | VI | VI | AI | (0,765, 0,865, 0,965) |

EC_{4} |
VI | I | VI | I | I | I | VI | I | I | (0,562, 0,696, 0,831) |

EC_{5} |
I | VI | I | VI | VI | I | VI | I | I | (0,438, 0,604, 0,769) |

EC_{6} |
VI | AI | VI | VI | VI | VI | VI | VI | I | (0,735, 0,835, 0,935) |

**Aggregate the Weighted Ratings of Lecturers Versus Criteria**

**Table 3** shows the weighted ratings of each lecturer using Eq. (10).

**Calculation of **

As shown in **Table 4**, the distance of each candidate from A^{+} and A^{-} can be calculated using Eqs. (11-14).

Table 4 The Distance of Each Lecturer From A^{+} and A^{-} |
||

Lecturers | d^{+} |
d^{-} |

L_{1} |
0,929 | 0,928 |

L_{2} |
0,881 | 0,975 |

L_{3} |
0,947 | 0,907 |

L_{4} |
0,835 | 1,033 |

L_{5} |
0,952 | 0,897 |

**Obtain the closeness coefficient**

The closeness coefficients of each lecturer can be calculated by Equations (15), as shown in **Table 5**. Therefore, the ranking order of the five candidates is A_{4} > A_{2} > A_{1} > A_{3} > A_{5}. Consequently, the best candidate is A_{4} .

Table 5 Closeness Coefficients of Lecturers |
||

Lecturers |
Closeness coefficient |
Ranking |

L1 |
0,500 | 3 |

L2 |
0,525 | 2 |

L3 |
0,489 | 4 |

L4 |
0,553 | 1 |

L5 |
0,485 | 5 |

This paper proposed a new DFTOPSIS approach to support the process of evaluating the performance of one or many lecturers. Using the proposed approach, university managers can evaluate/rank a lecturer based on six aspects, including personality characteristics, students evaluation, total number of publications, participation in professional society, classroom teaching experience and fluency in a foreign language. In the proposed approach, the ratings of alternatives and importance weights of criteria for the lecturer’s performance evaluation were expressed in triangular fuzzy numbers in time sequence. The proposed approach can also help identify the closeness coefficients to determine the ranking order of lecturers in general. The application in this study shows that the computational procedure is efficient and easy to implement. Thus, for practitioners, the proposed approach is a very effective tool to solve MCDM problems. Future research may apply the proposed approach to other MCDM problems with similar settings in various industries.

This research is funded by Vietnam National Foundation for Science and Technology Development (NAFOSTED) under grant number 503.01-2018.03.

Anniseh, M., Yusuff, R., & Shakarami, A. (2009). Aggregating group MCDM problems using a fuzzy Delphi model for personnel performance appraisal. *Scientific Research and Essay*, *4*(5), 381-391.

Ates, N.Y., Çevik, S., Kahraman, C., Gülbay, M., & Erdogan, S.A. (2006). Multi attribute performance evaluation using a hierarchical fuzzy TOPSIS method. *StudFuzz,* *201*(2), 537-572.

Costa, C.A.B., & Oliveira, M.D. (2012). A multicriteria decision analysis model for faculty evaluation, *Omega 40*(4), 424-436

Dat, L.Q., Thong, N.T., Son, L.H., Ali, M., Smarandache, F., Abdel-Basset, M., & Long, H.V. (2019). Linguistic approaches to interval complex neutrosophic sets in decision making. *IEEE Acess, 7*(3), 38902-38917.

Dubois, D., & Prade, H (1978). Operations on fuzzy numbers. *International Journal of Systems Science, 9*(6), 613-626

Ellington, H., & Ross, G. (1994). Evaluating teaching quality throughout a university a practical scheme based on self-assessment. *Quality Assurance in Education, 2*(2), 4-9.

Falsafi, N., Zenouz, R.Y., & Mozaffari, M.M. (2011). Employees’ performance appraisal with TOPSIS under fuzzy environment. *International Journal of Society Systems Science, 3*(3), 272-290.

Findawati, Y., Taurusta, C., Widiaty, I., & Nandiyanto, A.B.D. (2018). Teacher performance assesment application using naive bayes classifier method. *In IOP Conference Series: Materials Science and Engineering, 384*(1), 47-55.

Ghonji, M., Khoshnodifar, Z., Hosseini, S.M., & Mazloumzadeh, S.M. (2015). Analysis of the some effective teaching quality factors within faculty members of agricultural and natural resources colleges in Tehran University. *Journal of the Saudi Society of Agricultural Sciences**, 14*(2), 109-115.

Grigoroudis, E., & Siskos, Y. (2002). Preference disaggregation for measuring and analysing customer satisfaction: The MUSA method. *European Journal of Operational Research, 143*(1), 148-170.

Hassna, L.O., & Raza, S. (2011). An assessment of the relationship between the faculty performance in teaching, scholarly endeavor, and service at Qatar University. *Research in Higher Education Journal, 10*(2), 1-18.

Hirsch, J.E. (2005). An index to quantify an individual's scientific research output. *Proceedings of the National academy of Sciences of the United States of America, 102*(46), 165-169.

Huang, S.Y., Huang, Y.C., Chang, W.H., Chang, L.Y., & Kao, P.H. (2013). Exploring the effects of teacher job satisfaction on teaching effectiveness: Using teaching quality assurance’as the mediator. *International Journal of Modern Education Forum, 2*(1), 17-30.

Jacquet-Lagreze, E., & Siskos, Y. (2001). Preference disaggregation: 20 years of MCDA experience. *European Journal of Operational Research, 130*(2), 233-245.

Jauch, L.R., & Glueck, W.F. (1975). Evaluation of university professors’ research performance. *Management Science, 22*(1), 66-75.

Kaufmann, A., & Gupta, M.M. (1991). *Introduction to Fuzzy Arithmetic: Theory and Application*. VanNostrand Reinhold, New York.

Kusurkar, R.A., Ten, C.T.J., Vos, C.M., Westers, P., & Croiset, G. (2013). How motivation affects academic performance: a structural equation modelling analysis. *Advances in Health Sciences Education, 18*(1), 57-69.

Li, G., Kou, G., & Peng, Y. (2015). Dynamic fuzzy multiple criteria decision making for performance evaluation. *Technological and Economic Development of Economy, 21*(5), 705-719.

Paulsen, M.B. (2002). Evaluating teaching performance. *New Directions for Institutional Research, 114*(2), 5-18.

Phiakoksong, S., Niwattanakul, S., & Angskun, T. (2013). An application of structural equation modeling for developing good teaching characteristics ontology. *Informatics in Education-An International Journal, 12*(2), 253-272.

Rahman, A.W. (2016). Students'perception toward good lecturer pedagogical competence. *English and Literature Journal, 3*(2), 141-155.

Rama, S. (2011). Professor's performance for effective teaching (Kosovo case). *Procedia-Social and Behavioral Sciences, 12*(3), 117-121.

Ramli, N., Mohamad, D., & Sulaiman, N.H. (2010). Evaluation of teaching performance with outliers data using fuzzy approach. *Procedia-Social and Behavioral Sciences, 8*(2), 190-197.

Rouyendegh, B.D., Öztürk, B.N., & Cebeci, K. (2018). Performance Management of Supervisors in Railway Company: A case study. *Management and Business Research Quarterly, 7*(1), 10-20.

Siskos, Y., Grigoroudis, E., & Matsatsinis, N.F. (2005). UTA methods. *In Multiple criteria decision analysis: State of the art surveys*. Springer, New York, NY.

Skordoulis, M., Chalikias, M., & Koniordos, M. (2014). Students’ satisfaction from their educational context through DREEM and LOT-R. *In Joint Conference on Knowledge-Based Software Engineering* (113-122). Springer, Cham.

Skordoulis, M., Drosos, D., & Mandalenaki, M. (2016). An analysis of students’ satisfaction using a multicriteria method. *In Proceedings of 2nd International Conference on Lifelong Education and Leadership for All (ICLEL 2016)*, Liepaja.

Sproule, R. (2002). The under determination of instructor performance by data from the student evaluation of teaching. *Economics of Education Review, 21*(3), 287-294.

Thong, N.T., Dat, L.Q., Son, L.H., Hoa, N.D., Ali, M., & Smarandache, F. (2019). Dynamic interval valued neutrosophic set: Modeling decision making in dynamic environments. *Computers in Industry, 108*(2), 45-52.

Wiswall, M. (2013). The dynamics of teacher quality. *Journal of Public Economics, 100*(2), 61-78.

Zahir, A.T.P., & Fakhri, L.S. (2011). Improve faculty effectiveness by sabbatical leave. *Procedia-Social and Behavioral Sciences, 29*(3), 917-926.