Wind farm layout optimization is one of the challenging problems in the field of renewable energy. In the present study, a new nonlinear mathematical model for layout of wind turbines under multiple wake effects is proposed considering two objective functions separately: maximization of total power production and minimization of cost per power. To incorporate multiple wake effects into the proposed model, Jensen's wake decay model is employed. It was proven that the proposed model has totally unimodularity property and according to this property, relaxation of binary decision variables related with the wind turbine locations makes the model relatively simple to solve. Computational study reveals that results of total power production and cost of power obtained from the proposed model outperform that of the previous studies in the literature on a set of example cases and therefore, can be used to layout more productive wind farms. (C) 2018 Elsevier Ltd. All rights reserved.