Where does it go?

I don’t know but I’d be calling Spectrum and asking. You’re paying for it, why not get the faster speed all the time?
Maybe they think you’re asleep at those times and figure you won’t notice.

I had symptoms similar to yours earlier this year. Spectrum guy came out and fixed some wiring at the outdoors junction box. Worked well for a while, but gradually and sporadically went downhill, until one day I was getting 0.2 Mbps about mid-morning. Called Spectrum, asked tech support (nice, knowledgeable lady) to check my modem. She said the modem was OK, that I had a poor signal-to-noise ratio. Set up an appointment. The guy who came out was really, really good. He re-did some connection at the junction box and checked my signal coming off the telephone pole; good. Then we worked connection-by-connection from the outside box to the modem. Went into the attic where the first connection was; fortunately, that wire from the junction box to that first connection was good. Checked some intermediate cable from that wire to the cable that goes down inside my wall; OK. Checked the connection at the modem; NG. However, that connection came off a wire inside the house that connected at the wall with that cable portion that lay inside the wall and behind a double bookcase. These Spectrum guys expect to sometimes have to stay three or four hours, so we decided to unload that double bookcase (vs. making another appointment), move it, and then check the wire inside the wall; there was the problem! Fortunately, I had already drilled a hole in the upper supporting beams for an OTA antenna cable for my TV, so it was relatively easy for him to thread a new section of cable inside the wall to the wall's junction box. I had an "L-shaped" connector there (because of the bookcase being close to the wall) and he said these things often give problems, so he just made a regular, straight connection with the final portion remaining cable that went to my modem (no connection at all to the wall plate cover). For years I had a signal amp just before the modem. He had made and checked all the adjustment at the outdoors junction box and checked my signal without the amp; good. So we did away with the amp and fed the cable to the modem as it was supposed to be connected. All has been good. Yes, the internet speed does vary depending on time of day, number of users on the feed, etc., but I've been pretty happy; best in years. I don't check my speed very often (supposed to be 300 Mbps nowadays), but the mid-200s is a low as I found it to be.

You have to tell your router the max transmission unit (MTU) you want, rather than accept what it gives you.

Look here for some details on this:

Very informative video; stuff I didn't know. My computer nerd grandson set up my router and modem, so I'll have to see if he did all this. 
(I get nervous when plunged into a command prompt screen with all those things I might mess up)

The guy in the video uses the same speed test I do ... and I get similar readings (480 Mbps) after a reboot. But the issue for me, is where does all that speed go later on? I can conceive of being "throttled" by Spectrum at busy times, but at 4:00 am?
Used to be I could start off the morning with a fresh reboot, with 480 Mbps ........... and at 5:00 pm, do another speed test and still have the same number.
Not any more.
That's why I used the balloon with a slow leak analogy. During the day, the Mbps units just drift away. Fresh reboot and I get them all back again. Until they bleed off again.