Dimethyl sulfoxide, urea, potassium acetate and amino silane were used to modify kaolinite, respectively, which was applied to impart the flame retardancy of unsaturated polyester resin (UPR). Cone calorimeter was used to study their flame retardancy and smoke suppression, SEM and TG were used to study the micro morphology of the char and flame retardant mechanism. The results showed that the flame retardant properties of UPR were improved by the modified kaolin, inside the urea modified kaolin showds the best flame retardant, smoke suppression and thermal stability, which peak of heat release rate, total smoke product, CO yield and CO₂ yield reduced by 14.9%, 59.1%, 30.8% and 29.6% compared with pure UPR, respectively, followed by amino silane modified kaolin, and poor for dimethyl sulfoxide modified kaolin and potassium acetate modified kaolin.