As the pillar of national fiscal revenue, non-tax revenue has a high degree of autonomy and is an important foundation for the realization of national governance. Although the expansion of non-tax revenue has brought some negative effects, the proportion of non-tax revenue has been fluctuating around 40%, which has not broken the global consensus that non-tax revenue occupies an auxiliary position. In addition, it cannot be ignored that local governments using non-tax autonomy to participate in the primary and secondary distribution of income realizes the rational allocation of resource elements among various departments, which in turn promotes the upgrading of department structure. This paper uses the panel data of 292 prefecture-level cities from 2011 to 2019 to analyze the impact of non-tax autonomy on the upgrading of department structure. Further, using the non-tax proportion as the threshold variable, the panel threshold effect model is used to empirically examine the reasonable range of non-tax proportion to realize the structural upgrading effect of non-tax autonomous departments. The empirical results show that: (1) Non-tax autonomy significantly promotes the rationalization and advanced level of department structure, and promotes the upgrading of department structure. (2) There is an inverted “U”-shaped relationship between non-tax autonomy and the rationalization and advancement of department structure, which indicates that only by controlling local non-tax autonomy within a reasonable boundary, can it be improved for a long time. (3) The threshold effect test shows that there is a significant non-tax proportion threshold effect between non-tax autonomy and the upgrading of department structure. Specifically, there are single thresholds and double thresholds for the rationalization and advancement effects of the department structure of non-tax autonomy, which further proves the above conclusion. (4) The mechanism analysis shows that the improvement of non-tax autonomy improves the level of technological innovation and the efficiency of resource allocation, so as to promote the upgrading of department structure. This paper manually sorts out the data of non-tax revenue, government fund revenue and state-owned capital operating revenue in the general public budget, obtains the variable of total non-tax revenue, and examines the linear and nonlinear effects of non-tax autonomy on the upgrading of department structure under the total non-tax revenue. It effectively supplements the research on the theory of non-tax revenue, provides a reference for local governments to reasonably control the scale of non-tax revenue and realize long-term flow, and then provides experience support for the reform of non-tax revenue system and the optimization of department structure.