He did love both. But by the time he wooed Kay in New Hampshire, he was older, colder, more calculating. That scene in NH, with him dressed in a banker's homburg hat, the '47 Cad limo trailing him, is almost chilling. I don't doubt he loved Kay, but he also wanted her WASP credentials and persona to put the veneer of respectability on his life and business.