The Federal Reserve is usually understood as the bankers' bank. But what if it was the people's bank?

At the Fed's annual Jackson Hole conference last week, an assortment of community organizers, activists, labor organizations, and economists showed up to push America's most important financial institution towards putting the concerns of working and nonwhite Americans at the center of monetary policy. The group, called Fed Up, has met with Federal Reserve officials before, but Thursday's meeting was nonetheless unprecedented and striking — both for being on the record, and for the detailed, impassioned, occasionally heated, and remarkably pointed conversation that resulted.

Fed Up's complaints are several. The Fed is too worried about inflation, the activists say, and not worried enough about pushing the boundaries of maximum employment when it sets interest rates. They also argue the population of Fed officials is not diverse enough along racial, gender or class lines, and that the Fed itself could do with some institutional reform.

To a large extent, Fed officials agreed: "I'd be surprised if anyone in the Federal Reserve thinks we've done well on [diversity]," said New York Fed President William Dudley. "We're going to run this economy hot. Get unemployment down lower," added San Francisco Fed President John Williams. "So I don't think we disagree about that basic view."

And in Fed official's defense, some of this balancing is a judgment call. The Fed's inflation target is 2 percent, but since that is neither a ceiling nor a floor, officials must decide how far to overshoot that target and for how long in the name of spurring job growth. There is also generally a lag time between a change in interest rates and when it's felt in the economy. So Fed officials have to make educated guesses about when to drop rates to firm up a stumbling economy or raise them to get ahead of inflation.

Finally, Fed officials themselves can disagree over the full extent of their powers. That point was illustrated in Thursday's meeting when Boston Fed President Eric Rosengren and Fed Vice Chair Stanley Fischer disagreed over just what tools the Fed has to deal with asset bubbles. (Rosengren argued there were lots of tools while Fischer insisted there were few.)

The first complication here is that, even from a technical perspective, it's hard to see why the Fed is even contemplating another interest rate hike in December. Rod Adams, a neighborhood organizer from Minneapolis, noted in a particularly impassioned moment that there's essentially no indication that inflation is on the rise. Fed officials' own projections show inflation will just barely touch 2 percent through 2018. Josh Bivens, an economist at the Economic Policy Institute who joined Fed Up at Jackson Hole, argued that fully healing the damage from the Great Recession will require a prolonged period of overshooting the Fed's inflation target.

And in truth, none of the Fed officials at the meeting really debated any of these points. What did come up was the threat of asset bubbles and financial instability. "One of the ways that you get maximum employment is that you don't allow excesses to build up to the point that you actually have another recession, which hurts everybody in the room," Rosengren said.

This is where the unspoken moral problem of Fed policy really becomes inescapable. In very broad terms, the effects of lower interest rates and higher inflation tend to fall harder on the more fortunate members of society: retirees with savings portfolios, people with financial assets, those who work in the financial industry, and so on. Meanwhile, the effects of raising interest rates and slowing down jobs and wage growth tend to fall hardest on the least fortunate: Racial minorities, people with only a high school education, or people with prior criminal records.

At any given moment, unemployment for African-Americans is roughly double the national unemployment rate. But that gap tends to close during boom times and widen during downturns. "The economy has recovered for much of white America, but for black and Latino workers it has not," said Adams. "If you decide that we're at maximum employment now and you intentionally slow down the economy, you'll be leaving us behind, pulling up the ladder right after you've climbed it."

The brutal truth is that when the Fed slows down the economy by raising rates, it is throwing people out of work. And the people most likely to be thrown out of work first are those forced to the fringes of the labor market already by discrimination and other circumstances. So raising interest rates to fight financial instability essentially means black, Latino, and other underprivileged workers are the first to be thrown on the sacrificial alter to save us all from Wall Street's irrational exuberance.

This is partly why Fed Up is pushing for more racial and gender diversity among Fed officials, and to remove the financial industry from its privileged position among those officials. The idea is that the voices of those people the Fed will help or harm should all be equally heard in its deliberations. Fed officials may agree on running the economy hot for a while, but the lack of those voices may be hiding just how hot we need to run it.

There are practical policy changes the Fed could make as well. Bivens has released work on alternative tools the Fed could develop to pop bubbles without causing all that collateral damage: Higher capital requirements for banks, more use of its research powers and public relations to alert the markets to bubbles, and other ideas that already lie in the scope of the Fed's powers. Rosengren said the Fed should use all tools at its disposal to fight financial instability. But if Fed officials are looking for practical ways to build Fed Up's concerns more fully into its ways of doing business, it could start by developing those tools and explicitly rejecting interest rate hikes as a way to combat bubbles.

Another would be to adopt a higher inflation rate target like 3 percent or even 4 percent. If the effects of maximum employment take longer to reach marginalized communities, then the 2 percent target is driving the Fed to cut off job growth before those communities can ever heal.

It should be noted, as Fed Up activists did, that within the scope of its tools, the Fed has actually done a far better job maximizing jobs for marginalized Americans than Congress or the state governments. To some extent, the Fed is catching heat because these it's actually willing to listen to reason.

But it's long been said that societies are judged by how they treat their weakest and most vulnerable citizens — that the powerless have a uniquely powerful claim on our responsibilities. And Thursday reminded us that truism applies to the Fed too.