"If you only love those who love you, what is your reward? Even the tax-gatherers do as much."
You are judging people by their behavior with their friends and colleagues, and that tells little about their conduct with outsiders. You object that physicists are no more or less ethical than investment bankers. But physicists are not looting the global financial system. Why would I share your insider's judgment? I am not an insider; as tnh memorably said, "You're not a member, never will be." So why would I--why would any outsider--buy into bankers views of themselves?
I've no doubt that the bankers you know are mostly pleasant people to hang out with, if you are also a banker, if you are "one of them." But they are still participants in the looting of the global financial system. And memory is selective and malleable. Because you like these people, because you are one of the tribe, you remember less of their flaws than you would of a stranger, or an enemy. Outsiders see simply a tribe of looters, and you a member. If you want to change the world's view of your tribe, reform your tribe.
And, you know, you're a johnny-come-lately. Your erudition is not respected by the old timers, and they'll jettison you in a second if they think you might be a liability. You are not an insider, not really, not where it counts.