Ben Carson has finally figured out that living near the projects as a kid doesn’t mean he’s qualified to run the Department of Housing and Urban Development (HUD).

The famed neurosurgeon and former presidential hopeful told The New York Times that his job as President Trump's HUD secretary is harder than his old craft. “There are more complexities here than in brain surgery,” Carson said. “Doing this job is going to be a very intricate process.”

According to The Times article, sources close to Carson say he finds the job “frustrating and puzzling,” and that he considered resigning during a struggle with the budget. Additionally, he has been unable to meet department goals that include establishing community centers so people reliant on HUD’s services can become self-sufficient.

Carson’s HUD has been under scrutiny for ethical concerns. The agency’s inspector general is conducting an investigation after concerns were raised that Carson's family may be profiting from his office, The Washington Post reports, because officials became concerned when Carson’s son, Ben Jr., invited potential business associates to a HUD event last summer. Additionally, Carson’s wife, son and daughter-in-law (all of whom are not administration officials) have attended official HUD meetings.

“I expressed my concern that this gave the appearance that the Secretary may be using his position for his son’s private gain,” Linda M. Cruciani, HUD’s deputy general counsel for operations, wrote in a memo.

Carson seems unconcerned about the allegations and doesn’t see an issue with his family’s involvement.

“I don’t have any problem with ethics,” Carson said. “Here is a rather unique situation, Ben is somebody who is integrally important to me, and wants to help. I’m not going to just say no because it looks this way or that way.”

Carson found himself wrapped in controversy after the public found out someone placed a $31,000 furniture order for his office. He released a statement, also posted on his Facebook page, with an explanation and said he wants the order canceled.

“I was as surprised as anyone to find out that a $31,000 dining set had been ordered," Carson said in the statement. "I have requested that the order be canceled. We will find another solution for the furniture replacement."