Commonly applied theories of self-excited vibrations in machine tools are prevailingly based on frequency domain analysis of linear models, and in most cases only focus on evaluation of stability limits of strictly linear systems. In this way the behaviour of more stable or more unstable systems cannot be studied and analyzed. Neither is it possible to study certain important nonlinear effects because of limitations connected with the classical representation of systems only in the frequency domain. Evaluation of stability by analysing nonlinear systems in the time domain has not been applied until recently. The paper shows a new approach based on an interpretation of self-excited vibration systems as nonlinear servo-systems. With this approach linear and nonlinear systems of any degree of stability can be studied using a combination of frequency and time-domain methods. This can considerably contribute to better understanding of complex phenomena in various regimes and specific situations.