In a vacuum San Diego over Norfolk 1000% of the time.
I chose Norfolk due to timing issues regarding taking San Diego orders, and my parents are getting older (declining health issues etc.).
I have family and friends within 3 hours so I'm usually doing something fun in my free time instead of HATING life here. Trips to Northern Va, Eastern Shore, Lake Anna and beaches in North Carolina round out the list in addition or doing things I find fun here locally (beach concerts/festivals, adventure park, soaring etc. there's something here for everyone.)
There are only 3 places you should live here if your boat is based out of NOB. Ghent, Freemasion area downtown, Chicks (chix beach, Chesapeake Beach... Whatever you want to call it).
Ghent/Freemason have year round scenes, where the beach gets a little quieter in the winter. I live in Chix beach and have a 5 minute drive to the dog park, 5 to the water, 20 to NOB gate (regardless of hour of the day... Once you hit gate traffic it gets to be a bit of a dice roll). And top eating and drinking establishments within walking distance.
You'll get similar with Ghent/Freemason, just that they're not the beach... Think gentrified urban.
If your boat is going to be spending 24 months in the yards, and you want to be close but not live in a shitty area, old town and the swimming point areas of Portsmouth are nice, but the rest of the town is a shithole for the most part. Some people live in Williamsburg and Love it as well, but it's a long drive every day, even without traffic.
If you pick Norfolk your level of enjoyment with this place will depend largely on where you live, because you're going to spend every day going to work hoping your ship sinks, and hating your life when you get to it. If you fuck away where you live here (and don't avoid having to take any road that that ends in 64, drawbridges or tunnels to and from work) you're going to HATE it.
Traffic here isn't all that great (but still tons better than DC) and peoples commutes to NOB from the family neighborhoods, that are popular and cheaper often, switch from 30 minutes to 1.5hrs+ because some air department dude high on bath salts crashes his car in the tunnel outside the gate, or some other nonsense.
Pay the premium to live in neighborhoods where you don't have to take the highway to work and your quality of life with increase 10 fold.