Waste drilling fluids are difficult to reuse because they are strongly alkaline and remain highly stable as dispersed colloidal systems, which hinders solid-liquid separation. Therefore, gel breaking and destabilization are required before reuse. The destabilization efficiency is controlled by multiple operating conditions, among which breaker dosage, reaction temperature, pH regulator dosage, and inorganic salt proportion are critical. In this study, the Zeta potential of the treated drilling fluid was used as the stability indicator because it sensitively reflected changes in particle aggregation and dispersion. Based on experimental data obtained from a Box-Behnken design (BBD), response surface methodology (RSM) combined with analysis of variance (ANOVA) was employed to assess the significance of main effects and interactions among factors. Building on these results, three surrogate models—a backpropagation neural network (BP-ANN), random forest regression (RF), and Gaussian process regression (GPR)—were constructed to predict and compare the performance of Zeta potential modeling. Results indicated that the BP neural network achieved optimal fitting performance, with an overall coefficient of determination (R2) of 0.9629 and an average R2 of 0.8125 under 5-fold cross-validation—significantly higher than those of other surrogate models. Thus, it was selected as the process optimization model. To further identify the optimal solution, a genetic algorithm (GA) was integrated with BP-ANN for global optimization. The GA-BP method yielded optimal parameters: gel breaker dosage of 3.306 g/L, reaction temperature of 57.74℃, pH adjuster dosage of 3.45 g/L, an inorganic salt proportion of 0.80, and a predicted Zeta potential of -10.735 mV. The measured value was -10.85 mV, with a relative error of 1.06%. Overall, the proposed experimental design-based modeling and optimization framework provides a reliable basis for determining operating parameters for gel breaking and destabilization, and supports the subsequent reuse of treated drilling fluids.
