It is very important for airlines and passengers to train qualified pilots.In order to meet the requirements of aviation laws and regulations and flight safety, airlines are required to train and evaluate all pilots regularly. It is of great significance for airlines and training centers to study how to better evaluate pilots through tests, and which indicators can more truly and objectively reflect the level of pilots:flight skills of pilots or their evaluation scores. Based on the Rasch model of IRT theory, this paper establishes a set of adaptive theoretical evaluation methods by analyzing and deriving important parameters such as the difficulty and quality of test questions, the discrimination index of test questions, and the pilot ability, etc., and then automatically judges the reliability of an exam. At the same time, the regression analysis was carried out with real airline data, so as to establish a set of unique self-adaptive evaluation management methods. And the important concept of Three-Tier of Core Competency was put forward for the first time in the world, which provides a feasible implementation a feasible and valuable theoretical basis for pilot skill evaluation and test reliability assessment.