Here's what the U.S. Secret Service doesn't do: the U.S. Secret Service does not drop the President off at events and say, "Hey, we'll see you later!"
If security plans were perfect, and none are, then the most iconic images associated with presidential protection -- agents surrounding the protectee as he shakes hands or gives a speech -- would not be so deeply ingrained in our consciousness.
It is precisely because there are always unknowns and unknowables that agents stay in very close proximity to the president at all times. The head of Obama's detail was steps away from the President as the sign language faker pantomimed his way to infamy.
Now, South Africa was primarily responsible for the event security plan. It was a loose plan; most of the crowd, apparently, did not go through magnetometers; exit and entry routes for VIPs seemed crowded and uncleared; and the security forces in charge of vetting event staff, including our now world famous hand actor, failed to investigate the backgrounds of everyone cleared to be on the podium.
In my experience watching the Secret Service conduct advances, agents in charge of the Secret Service's event plan would have assumed, as their starting point, that some of all of the South African plan would fail.
I have no specific insights into how this all works, but you can be sure that the Central Intelligence Agency and the National Security Agency surged resources, both human and technical, to run the names of known show participants against terrorist watchlists. Since the NSA does not keep records on the mental health of South Africans citizens, there would be no reason for them to have collected any information in advance about a sign language interpreter with no foreign intelligence value. These intelligence efforts run parallel to the Secret Service's own Protective Intelligence Division, which regularly works with foreign counterparts on precisely the type of sensitive details like the names of people who will be allowed into the secure zone. This hidden diplomacy is a 365-day-a-year endeavor.
In places where the Secret Service does not control the entire venue, agents will insist on securing, with its people -- our people -- the President's immediate surroundings and keeping clear an emergency evacuation route. Take, for example, the main platform. The Service would have calculated, based on technical and classified studies of ammunition, explosives and weapons, how much "stand off" is adequate, and it will draw its own perimeter.
A combination of counter-sniper teams and covert counter-surveillance teams, active technical surveillance, physical barriers and post standers would work together to secure the lines of sight to the President. This "box within a box" concept provides a visible deterrent and allows site agents to adjust the plan dynamically. With a limited number of agents, assets and a limited faith in the home country, this is pretty much the best the Service can do.
I haven't mentioned the agents who surround the President physically yet. They're the inner ring. They don't stand there and wait for the President to finish a speech. They aren't adornments. The reason these agents look so morose is because they are watching other people intently, assuming that threats WILL arise from within the inner perimeter -- from the crowd, from staff, from the security details for foreign leaders -- and rehearsing how they will remove the president from danger when those threats pop up. Not if, but when.
Without going into too much detail, you can safely assume that the Presidential Protective Detail frequently practices responding to potential assassins who've made it through other perimeters and wind up right next to the President. If someone close to Obama had drawn a knife, agents around the President would have reacted, based on training and planning, but other Secret Service assets and personnel, using protective methods that aren't disclosed, would kick into gear just as quickly.
The logistics involved in advancing a trip like this are Herculean, and the agents in charge of the trip certainly had to improvise.
There is no way to ensure that every box the Secret Service wants South Africa to check will be checked.
And there is no way to predict what a protectee will do, as agents guarding President Clinton at King Hussein's 1999 funeral in Jordan found when Clinton joined a throng of mourners who had not been screened for weapons.
The knock on the Secret Service today is that something bad could have happened. But that's why the Secret Service exits: because stuff has, and can, always happen.
N.B.: a former State Department official says the Secret Service should have kept the President away from a terrorist named Raul Castro. The former charge is just stupid: world leaders responsible for killing Americans come to the United Nations all the time, and they're afforded Secret Service protection. Castro might be a malevolent head of state, but he's a head of state.