Problems of developing subject education and science have become more acute during different social changes. The most problematic is the teaching of exact disciplines, including mathematical disciplines (mathematics, algebra, geometry), which form knowledge for working in high-tech industries. It is worth noting that the methodology of teaching mathematics has accumulated a large unsystematized portfolio of new techniques and methods. Thus, open problems aimed at developing students’ creativity are used more actively in the lessons. Tasks that require students to use non-standard approaches to the solution affect academic success. But a teacher of mathematics needs to have a classification of problems in the educational process to choose tasks of an open type correctly. In this regard, the article deals with substantiating the classification of open problems for teaching mathematical disciplines and developing methodological guidelines for lessons that help increase learning motivation and academic achievement of students. The leading research methods are the following: observation of teachers’ methodological work, talks with teachers, analysis of methodological developments, educational documentation and teachers’ surveys. Statistical processing of the results using Wilcoxon W test. In 2020-2021 the authors carried out an experiment with 358 students from 16 classes. Based on its results, the authors of the article have identified a classification of open problems for teaching mathematical disciplines; developed and implemented methodological guidelines for lessons of mathematics that contribute to developing students’ creativity in the academic progress. The authors evaluated the effectiveness of using the selected classification of open problems for teaching mathematical disciplines. They concluded about the increase of group motivation among students and academic performance in mathematical disciplines. The practical use of the classification of open problems allows the teacher to differentiate educational problems in mathematics, taking into account the relationship between their logical structure and the level of ability achieved by the student to solve problems in mathematics. Methodological guidelines developed by the authors can be used to help increase the learning motivation and academic achievement of students while preparing for mathematics lessons.