For second consecutive year, Roswell Park doesn’t make Top 50 in U.S. News cancer hospital rankings

The 2023-24 rankings, released Tuesday, did not assign Roswell Park a numbered ranking – it only ranks the Top 50 – but the Buffalo institution’s overall score of 51 (out of 100) was the 95th-highest score among cancer hospitals.

That was better than Roswell Park’s score last year of 48.9, though that figure was the 76th-highest mark a year ago.

In an email to employees Tuesday, President and CEO Candace S. Johnson blasted U.S. News & World Report’s methodology, saying the process the publication “uses to build this feature is surrounded by significant and well-founded questions and criticisms.”

“Know this: This list says nothing about the care we provide to our patients,” Johnson said in the message, which Roswell Park shared with The Buffalo News. “You have shown in so many measurable ways that we should take pride in Roswell Park’s care for our patients. U.S. News simply does not do a good job of understanding, measuring or highlighting what is truly exceptional in cancer care today.”

Johnson said U.S. News & World Report bases its conclusions about cancer patients’ outcomes on a “narrow category of inpatient cases that do not reflect modern cancer care.” She said the publication’s methodology favors centers in large cities and those with a high percentage of patients on traditional Medicare plans.

Meanwhile, it “unfairly looks at a tiny fraction of care at centers in smaller markets like Buffalo,” Johnson wrote.

U.S. News & World Report confirmed that its patient outcome data, which is 45% of the overall score, includes traditional Medicare members but not Medicare Advantage patients. 

The methodology – and Johnson’s criticism of it – underscores the love-hate relationship that hospitals, colleges and many other institutions have with numbered rankings. 

For years, Roswell Park was a mainstay in the Top 50, ranking No. 34 in 2021 and peaking as high as No. 14 in 2020 and 2019 – a ranking it still promotes in its advertisements around the area, such as inside Buffalo Niagara International Airport in Cheektowaga. 

Roswell Park, a National Cancer Institute-designated cancer center, also made the Top 50 in 2018, 2017 and six years in a row from 2010 to 2015. 

Experts have long advised patients to use the hospital rankings as a starting point to make health care decisions. While the U.S. News & World Report rankings are just one of many, hospitals zone in on the rankings and – when things go well – use them to recruit physicians and to market to patients. 

Larry Zielinski, a health care administration expert at University at Buffalo and a former Buffalo General Medical Center president, said the reason U.S. News & World Report focuses on traditional Medicare patients is because the data is readily available. It is a limited segment of the population, he said, but health systems have typically been hesitant to share data that can be easily compared and understood.

Still, he advises patients to carefully scrutinize how rankings are computed.

“In my experience, when hospitals are highly ranked, they trumpet the results,” Zielinski said. “When they are ranked poorly, they usually find some blemish in the methodology and downplay them. “

“I would advise patients to dig a little into the rankings when comparing two hospitals and focus on those quantitative measures,” he said. “If used like that, the rankings can be a useful tool.”

On its face, Roswell Park’s rapid drop in ranking from two years ago doesn’t seem to make sense, said Dr. Stephen Edge, the cancer center’s vice president for system quality and outcomes and a professor of oncology.

“There’s been some jumpings in rankings for us and a number of other cancer centers, but nothing’s really changed,” said Edge, a breast cancer surgeon. “People are still getting the same excellent care they got at Roswell Park three years ago, five years ago, 15 years ago. If anything, it’s getting better.”

But because U.S. News bases its patient outcome data on only those with traditional Medicare who are admitted to the hospital, he said the publication is missing a significant segment of Roswell Park’s patients and failing to account for how more cancer care is shifting to outpatient centers. 

“Because of the small numbers of patients, the ups and downs, the jumping in the rankings, are due to slight changes in that, with just a few patients here and there,” Edge said. “So it’s a flawed methodology. They use it because that’s the only information they can get on every hospital in America.”

Ben Harder, managing editor and chief of health analysis at U.S. News & World Report, said in a statement that the publication appreciates the “constructive and ongoing dialogue” it has had with Roswell Park leadership about its methodology. 

The publication said early last month it is working on “numerous methodological refinements” to keep pace with the evolution of health care. Some of those changes were included in the 2023-24 Best Hospitals rankings, while others are in development and may be introduced next year.

“Future U.S. News methodology refinement may include supplementing traditional medical data with data on commercially insured patients, incorporating social risk into our risk adjustment algorithms, and broadening our study of outpatient outcomes for services such as uterine and prostate cancer surgeries,” Harder said.

Roswell Park is not the only hospital that has criticized the rankings. 

The University of Pennsylvania Health System announced in late June that it would discontinue active participation in the annual U.S. News & World Report Best Hospitals rankings, which are now in its 34th year. That system, too, pointed to the need for “greater transparency and access to more comprehensive quality data beyond the narrow information” collected by the publication.

In the 2023-24 rankings, U.S. News & World Report did, however, dub Roswell Park “high performing” in six conditions/procedures: leukemia, lymphoma and myeloma; colon cancer surgery; lung cancer surgery; ovarian cancer surgery; prostate cancer surgery; and uterine cancer surgery.

While Johnson in her message to employees congratulated those services, she said that Roswell Park for years has urged U.S. News & World Report to improve its methodology.

“We will not promote ‘Best Hospitals’ in its current form, as it simply doesn’t reflect the quality of care at Roswell Park,” Johnson said. “It cannot be used as a measure of our success.”