There is lots to unpack here. That recognition of scale, both time and space, goes both ways in this argument. Which is where things get interesting.
Fermi paradox says if intelligent life is common and the galaxy (not even considering the universe here) is so immense, why isn't there more evidence of intelligent life? Where are the dyson spheres? Where are the radio transmissions? Would we be even able to recognize or detect them? Does intelligent life have a very limited lifespan until it gets to a point where it snuffs itself out? Maybe it evolves in on itself into virtual worlds where there would be no need to expand beyond their native star systems. Maybe all intelligent life sees other intelligent life as a potential threat, or at least that they might be thinking the same thing so as protocol they would wipe out any other intelligent life as a precaution (check out the dark forest theory).
I mentioned von Neumann probes before which are from a thought experiment that assumes an advanced civilization with technology that can exist in the realm of physics (as we know it) can create self-replicating space probes driven by AI that would travel to the nearest star, explore, gather resources, create another probe(s) and then off to the next star. Assuming 10% speed of light travel and a nominal stay of a few thousand years in each system, the entirety of the galaxy could be "colonized" that way within a few million years. So on the scale of billions of years, why haven't they consumed the galaxy with this method? Mabye a sufficiently advanced civilization that can get to this point of building said probes would recognize this danger and value galactic sustainability and thus limit these probes?
But yea, anything is possible given the scale of the universe, but is it probable? What if there are thousands of interstellar space faring civilizations in our galaxy alone, but what if they don't exist at the same time?
What are the chances they would, could, or should visit us?
Speed of light is the real buzz-killer here. Hard to get around that, no matter how advanced you are. Its not arrogant either to assume at least most of our physics is correct.