A Hybrid Applied Optimization Algorithm for Training Multi-Layer Neural Networks in the Data Classification

H. Hasan Örkçü, Mustafa İsa Doğan, Mediha Örkçü
1.895 563


Backpropagation algorithm is classical technique used in the training of the artificial neural networks. Since this algorithm has many disadvantages, the training of the neural networks has been implemented with various optimization methods. In this paper, a hybrid intelligent model, i.e., hybridGSA, is developed to training artificial neural networks (ANN) and undertaking data classification problems. The hybrid intelligent system aims to exploit the advantages of genetic and simulated annealing algorithms and, at the same time, alleviate their limitations. To evaluate the effectiveness of the hybridGSA method, three benchmark data sets, i.e., Breast Cancer Wisconsin, Pima Indians Diabetes, and Liver Disorders from the UCI Repository of Machine Learning, and a simulation experiment are used for evaluation. A comparative analysis on the real data sets and simulation data shows that the hybridGSA algorithm may offer efficient alternative to traditional training methods for the classification problem.


Artificial neural networks; data classification; training of neural networks; genetic algorithm; simulated annealing.

Full Text:



Fisher, R.A. “The use of multiple measurements in taxonomy problems” Annals of Eugenics, 7, 179–188 (1936).

Xu, G. & Papageorgiou, L.G. “A mixed integer optimization for data classification” Computers & Industrial Engineering 56(4), 1205-1215 (2009).

Fred, N. & Glover, F. “A linear programming approach to discriminant problem” Decision Sciences, 12, 68–74 (1981).

Fred, N. & Glover, F. “Simple but powerful goal programming

European Journal of Operational Research, 7, 44–60 (1981).

Lam, K.F., Choo, E.U., Moy, J.W. “Minimizing deviations from the group mean: A new linear programming approach for the two-group classification problem” European Journal of Operational Research, 88, 358–367 (1996).

Lam, K.F. & Moy, J.W. “Improved linear programming

discriminant problem” Journal of Operational Research Society, 47, 1526–1529 (1996). for the multi-group of some recently developed

linear [8] Sueyoshi, T. “DEA-discriminant analysis in the view of goal programming” European Journal of Operational Research, 115, 564–582 (1999).

Sueyoshi, T. Extended “DEA-Discriminant analysis” European Journal of Operational Research, 131, 324– 351 (2001).

Sueyoshi, T. “Mixed integer programming approach of extended DEA-Discriminant Analysis”, European Journal of Operational Research, 152, 45–55 (2004). [11]

Methodological comparison among eight discriminant analysis approaches” European Journal of Operational Research, 169, 247–272 (2006).

Analysis: [12] Bal, H., Örkcü, H.H., Çelebioğlu S. “An experimental comparison of the new goal programming and linear programming approaches in the two-group discriminant

Engineering, 50(3), 296–311 (2006).

Computers&Industrial [13] Bal, H. & Örkcü, H.H. “Data envelopment analysis approach to two-group classification problems and an experimental comparison with some classification models” Hacettepe Journal of Mathematics and Statistics, 36 (2), 169–180 (2007).

Glen, J.J. “A comparison of standard and two-stage mathematical

method” European Journal of Operational Research, 171, 496–515 (2006). discriminant

analysis [15] Üney, F. & Türkay, M. “A mixed-integer programming approach to multi-class data classification problem” European Journal of Operational Research, 173, 910–920 (2006).

Bal, H. & Örkcü, H.H. “A new mathematical programming approach to multi-group classification problems” Computers&Operations Research, 38, 105– 111 (2011).

Örkcü, H. H., & Bal, H. “A combining mathematical programming method for multi-group data classification” Gazi University Journal of Science, 24(1), 77–84 (2011). [18] Maskooki, A. “Improving the efficiency of a mixed integer linear programming based approach for multi- class classification problem” Computers and Industrial Engineering, 66, 383–388 (2013).

Denton, J.W., Hung, M.S., Osyk, B.A. “A neural network approach to the classification problem” Expert Systems with Applications, 1(4), 417-424 (1990).

GU J Sci, 28(1):115-132(2015)/ H. Hasan ÖRKÇÜ, Mustafa İsa DOĞAN, Mediha ÖRKÇÜ

Patwo, E., Hu, M.Y., Hung, M.S. “Two-group classification using neural networks” Decision Sciences, 24(4), 825–845 (1993).

Holmstrom, L., Koistinen, P., Laaksonen, J., Oja, “E. Neural and statistical classifiers taxonomy and two case studies” IEEE Trans. Neural Networks, 8, 5–17 (1997).

Mangiameli, P. & West, D. “An improved neural classification network for the two-group problem” Computers and Operations Research, 26, 443–460 (1999).

Pendharkar, P.C. “A threshold varying artificial neural network approach for classification and its application

Computers and Operations Research, 32, 2561–2582 (2005). prediction

problem” [24] Yim, J. & Mitchell, H. “Comparison of country risk models:

discriminant analysis and cluster techniques” Expert Systems with Applications, 26(1), 137-148 (2005).

Örkcü, H. H., & Bal, H. Comparing performances of backpropagation and genetic algorithms in the data classification. Expert Systems with Applications, 38(4), 3703–3709 (2011).

Xu, T., Wei, H., Hu, G. “Study on continuous network design problem using simulated annealing and genetic algorithm” Expert Systems with Applications, 36, 1322-1328 (2009).

Liu, M., Sun, Z.J., Yan, J.W., Kang, J.S. “An adaptive annealing genetic algorithm for the job-shop planning and scheduling problem” Expert Systems with Applications, 38, 9248-9255 (2011).

Örkcü, H. H. “Subset selection in multiple linear regression models: a hybrid of genetic and simulated annealing

Computation, 219, 11018–11028 (2013). Applied Mathematic

and Hosseini, P. & Shayesteh, M.G. “Efficient contrast enhancement of images using hybrid ant colony optimization,

annealing” Digital Signal Processing, 23, 879–893 (2013). algorithm, and

simulated [30] Zameer, A., Mirza, S.M., Mirza, N.M. “Core loading pattern optimization of a typical two-loop 300 MWe PWR using Simulated Annealing (SA), novel crossover Genetic Algorithms (GA) and hybrid GA(SA) schemes” Annals of Nuclear Energy, 65, 122-131 (2014).

Zhang, X.P. “Thresholding neural network for adaptive noise reduction”, IEEE Transactions on Neural Networks, 12 (3), 567-584 (2001).

Huang, C.Y., Chen, L.H., Chen, Y.L., Chang, F.M. “Evaluating the process of a genetic algorithm to improve the back-propagation network: A Monte Carlo study” Expert Systems with Applications, 36, 1459–1465 (2009).

Rumelhart, D.E., Hinton, G., Williams, R. “Learning representation by back-propagation errors” Nature, 32(9): 533-536 (1986).

Holland, J. Adaptation in Natural and Artificial Systems. Michigan Press, Michigan, (1975).

Goldberg, D.E. “Genetic algorithms in search, optimization and machine learning”, Addison Wesley, Readin, MA (1989).

Gen, M. & Cheng, R. Genetic algorithms and engineering design, Wiley, New York, (1996).

Davis, L., Handbook of genetic algorithms. New York: Van Nostrand Reinhold, (1991).

Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E. “Equation of State Calculations by Fast Computing Machines” J. Chem. Phys., 21 (6), 1087-1091 (1953).

Kirkpatrick, S., Gerlatt, C.D., Vecchi, M.P. “Optimization by Simulated Annealing” Science, 220, 671-680 (1983).

Kirkpatrick, S. “Optimization by Simulated Annealing-Quantitative Studies” J. Stat. Phys, 34, 975- 986 (1984).

Salzberg, S.L. “On comparing classifiers: pitfalls to avoid and a recommended approach”, Data Mining and Knowledge Discovery, 1, 317-328 (1997).

Seera, M. & Lim, C.P. “A hybrid intelligent system for medical data classification”, Expert Systems with Applications, 41, 2239–2249 (2014).

Polat, K., Şahan, S., Kodaz, H., & Güneş, S. “Breast cancer and liver disorders classification using artificial immune recognition system (AIRS) with performance evaluation by fuzzy resource allocation mechanism” Expert Systems with Applications, 32(1), 172–183 (2007).

Luukka, P. “Classification based on fuzzy robust PCA algorithms and similarity classifier” Expert Systems with Applications, 36 (4), 7463–7468 (2009).

Luukka, P. “Feature selection using fuzzy entropy measures with similarity classifier” Expert Systems with Applications, 38 (4), 4600–4607 (2011).

Stoean, R., & Stoean, C. “Modeling medical decision making by support vector machines, explaining by rules of evolutionary algorithms with feature selection” Expert Systems with Applications, 40 (7), 2677–2686 (2013).

Çomak, E., Polat, K., Güneş, S., Arslan, A. “A new medical decision making system: Least square support vector machine (LSSVM) with fuzzy weighting pre- processing” Expert Systems with Applications, 32(2), 409–414 (2007).

Özşen, S., & Güneş, S. “Attribute weighting via genetic algorithms for attribute weighted artificial immune system (AWAIS) and its application to heart disease and liver disorders problems” Expert Systems with Applications, 36(1), 386–392 (2009).