Kirsten Arnesen works as a secretary (that of course would be Personal Assistant today) in the Pacific Gas and Electric building at 245 Market Street (map). Here's a recent photo of the building, built in 1924 then subsequently strengthened and renovated following the 1989 earthquake.
Then ... After his embarrassing first meeting with Kirsten Joe visits her office to make amends. He is seen below approaching the entrance lobby from the left. The address 245 is on the door.
Then ... Joe puffs away as he waits for an elevator inside the lobby. Note that elegant brass lampstand ...
Then ... Joe tries his best to make up but Kirsten plays hard-to-get and they end up awkwardly waiting for the elevator outside her office.
Then ... Joe leaves the building but Kirsten chases after him - she relents and accepts his offer of dinner. That's the Ferry Building standing guard at the end of Market Street as best it can given the intrusively ugly two-level Embarcadero Freeway standing in its way.