Why doesn't the banana experiment change world lines in Steins;Gate?

In Steins;Gate, when Okabe finally decides to use one banana instead of the whole bunch to experiment on the PhoneWave (Name subject to change), the single banana returns to the bunch in gel-bana form. With this, they finally manage to observe the time-travelling effect of the PhoneWave (Name subject to change), effectively sending the banana back in time.

Later they figure out that for each second set on the PhoneWave (Name subject to change) they send items or messages one hour back in time.

Since they always set the timer to 120 seconds, wouldn’t that always change the world line in these situations? Because they send the banana 5 days back, it would always default to a world line where Mayuri doesn’t buy a banana bunch with a gel-bana attached to it, making it enough of a change to move between lines.

Is this ever addressed in the anime or visual novel?