American idols are falling.

In the wake of the terrorist attack at Emanuel AME Church in Charleston, SC, where a 21-year-old white supremacist callously shot and killed nine of the church’s black parishioners during bible study to incite a race war, debates have surged amongst politicians and the public alike to take down the confederate flag. On Saturday, June 27th, activist Bree Newsome took the matter into her own hands and brought it down. As of this past Friday, the flag has finally been taken down indefinitely from the South Carolina capitol grounds.

Companies today are making similar moves after a judge, per the request of the Associated Press, released court documents from a 2005 lawsuit where Cosby admitted to drugging women before sex. The judge took these actions to rectify the cognitive dissonance between Cosby’s public and private life. To the masses, Cosby, through the notoriety of The Cosby Show and his work as an educator, was “America’s dad.” But after the stage lights fell and behind closed doors, Cosby was cunning, time and time again exploiting his power as the country’s moral compass to seduce and rape young women and later silence them. In addition to the ties that have been cut between him and UMass Amherst and Temple University, Walt Disney World Resort has officially removed the bust of the actor from the Hollywood Studios theme park. Cosby reruns have been immediately pulled from Atlanta-based African-American network Bounce TV, and likewise those of The Cosby Show from BET-owned network, Centric.

Oh, how the mighty fall.

Yet why have we not wondered if the mighty still have a place to land in their exile?

Have we forgotten that even isolating Cosby into the position of persona non grata leaves intact the very grounds on which he originally stood? Cosby does not deserve to be absolved of his crimes. Neither, however, does the sociocultural matrix of which Cosby is a part and that he willfully corroborated. The complex workings of rape culture in the United States are no exception, and the labor their dismantling demands continues (as always) to be necessary.

One of the fundamental ways rape culture operates is by creating conditions that enable and require the disregard and denial of consent. Cosby had a history of systematically preying on women, so much so that comedian Hannibal Buress could include it in a set. Thirteen women, Buress told his audience, and yet it was Buress’ commentary, not the testimony of the thirteen different women he referenced, that created Cosby’s PR nightmare. It didn’t matter that each story mirrored those that came before — involving a paternalistic Cosby using his prestige to lure and drug woman after woman before sexually assaulting them. Even the cultural and economic capital of a victim in their own right, including model and actress Beverly Johnson, had little bearing. Each woman brave enough to come forward was a storyline framed with skepticism and indifference. Only after the judge released the 2005 deposition, forcing Cosby to unwittingly undermine himself, was the case definitely closed in the public eye. Despite tracing his activities back as early as the ’60s and ’70s, it was only through a game of “he said, he said” in recent months — between Cosby and Buress and finally between Cosby’s public denials and his courtroom admissions — that doubt ceased to be viable.

However, as much as Cosby’s downfall demonstrates the workings of patriarchy, it too provides a window into its racialized nuances. Cosby was protected by both his maleness and racial respectability politics that mythologized him into an untouchable example of the black community’s figurative “best.” This often emerged around Cosby’s legacy. Phylicia Rashad, his TV wife, told us to “forget these women” by treating the allegations as a deliberate smear campaign. Jill Scott did the same, demanding “actual” evidence from her Twitter followers who criticized the neb-soul singer for taking to Cosby’s defense.

Guarding Cosby was a means of buying time, or protecting the time black people had accrued through Cosby. Scott articulated this as being hurt and feeling betrayed by a man she respected. Time is too often something black people have been robbed of and systematically denied. We have seen this in the way Tamir Rice was shot in seconds and how Aiyana Jones’ murderer faced no retribution after killing her while she was sleeping on the couch next her grandmother. Our futures are often precarious. Cosby, whether we wanted him to be or not, has been an anchor. For this reason, Brittany Cooper noted that Cosby’s fall from grace was “more lament about the kind of loss he is to the community.” The foundation erected with Cosby has caved in on itself, leaving some wanting for the relief we once had.

Our grief for what we’ve lost for Cosby, however, is a major source of strength. Mostly because Cosby, and all of his condescending hypocrisy, highlight the stark ways that respectability politics will not and cannot save us. Standing by Cosby’s side for the sake of how he has been esteemed by a mainstream white gaze, despite his actions, implicitly requires black people’s survival be determined by our willingness to pledge unconditional loyalty to the predation of women. This unnecessarily demands black women, in particular, to sacrifice ourselves for black people to arrive somewhere none of us have any business being in the first place.

If we are not all going to be able to be free unapologetically as we are, none of us ever can be. No number of pudding pop or jello jokes change this.

But the circumstances of Cosby’s denouncement are also worth noting, specifically because he is getting chastised in ways that his white counterparts in Hollywood, such as Woody Allen, are not. In January, Tina Fey and Amy Poehler were praised for roasting Cosby while hosting the Golden Globes. No mention, however, was made of Woody Allen, a filmmaker, who has faced allegations of molesting his adopted daughter, and who later married another one of his adopted daughters. A man who, a year earlier, received a Lifetime Achievement Award from the same organization. None of Allen’s films have been pulled. Instead, he is in the process of creating his own series for Amazon. Where Cosby is getting blacklisted, Allen is thriving.

There might be no clear reason as to why. It is clear, however, that the difference between the two is arbitrary and this unjustified distinction does work to support the world we currently live in, but that we’re trying to dismantle.

Absolving Allen by turning a blind eye while making a spectacle of Cosby propagates the idea that black men, unlike their white counterparts, are especially prone to prey on women. We’ve heard this trope since Reconstruction and have watched it unfold in The Birth of a Nation. More recently, it was evoked during a Wednesday night bible study as justification for murder.

Cosby is not a victim. But he is not the only one who is guilty. A society that enables rape culture and does not unilaterally and uniformly punish rapists must be indicted simultaneously, one that is as much linked to the confederate flag and Allen’s Golden Globe award as it is to a statue of Cosby’s bust.

But visibly taking down both the confederate flag and Cosby isn’t enough. It is a washing of hands, without also washing ourselves of the violent racialized masculinity that haunts us. But that stain won’t come out by pushing it to the shadows. That stain requires work, the kind that scrapes knees and elbows. That guts a foundation in order to rebuild anew.

Want more commentary like this? Subscribe to our weekly digest below.