Time series forecast for everyday for till a distant future

I have time series data for every single day from last 5 years with seasonal variation and a general increase in trend. This is what my data looks like:

And I am trying to predict for every single day for 4-5 years in future. Approaches I have used currently:

• LSTM, GRU (but they are extremely prone to overfitting ad I am trying successive predictions which results in massive error accumulation and often flat over time for smaller lookbacks and sine wave driven patterns for long lookbacks.
• SARIMAX and other seasonal forecast models but they tend to scale very bad for a large period of predictions. Mostly they eventually result in a curve that follows trend(increase/decrease) but values are almost linear or unrealistically close to each other.

What I am currently thinking of:

Separate same day of each year and fit a model to those values and then have successive predictions for the same day for subsequent years. Thus having 365 different models and then concatenating the resultant values to have the whole year and future years predictions. This way I’ll perfectly preserve the seasonality and yet accumulation errors will be minimized. So my question is:

Is this a good idea (and if it is, is there a method to do it feasibly without assessing each of the 365 models) or I should try any other approach?