Rating the RatingsPart 1 - Intro

By Vince Ciotti & Elise AmesHIS Professionals, LLC

(with a lot of help from Mr. HIStalk,and many of you readers)

It All Started With…• We must give credit for this idea to several readers of HIStalk

who posted “Readers Write” entries about rating of vendors: – “From Former McKessoner: "You have to also take these

ratings with a grain of salt as well (in reference to the MD Buyline rating comment). Horizon Meds Manager was #1 one year and dead last the next.”

– And a fascinating post by Randall Swearingen, CEO (I swear!) of Swearingen Software Inc. in Houston: “Who audits them? What independent source verifies their data to make sure it is accurate and fairly represented since they seem to have influence over some buying decisions? Answer: nobody.”

• This got us thinking about the many ratings of HIS systems and just how accurate they are...

And Continued With…• We don’t mean to slam these agencies as disreputable; indeed,

Black Book earned our respect for this press release last year:– “Black Book’s survey inspection of the nearly 29,000 EHR user ballots

collected from Q3 2014 to Q1 2015 uncovered 33 hospital resellers casting EHR satisfaction and loyalty responses for 740 physician practices, prompting supplementary audits and an extended resurveying phase to resolve what Black Book deems intentional bias.”

– “It’s not Black Book’s issue that community physicians were pressured into implementing a particular EHR from their leader hospital,” said Doug Brown, Partner of Black Book. “Black Book's issue is that the hospital IT managers responsible for (re)selling and overseeing system implementations for particular EHR suites were also grading the product satisfaction and service delivery perceptions on behalf of providers they actually sold to. That is no different than soliciting a salesman to rate his own merchandise…” (italics ours)

• Odd how the Partner of Black Book quoted is named Doug Brown. You’d think they would have recruited the CEO of Allscripts…

Which HIS Vendor is #1?• There are a number of sources that rate HIS system for hospitals

such as EHRs, all of which rank the leading vendors differently:

And Physician Practice Vendors?• As if the various ratings of the top 10 hospital systems isn’t

confusing enough, consider the plethora of ratings of Physician Practice systems, of which 785 achieved ONC certification:

Who Should You Believe?• This rating disparity is not unique to the HIS industry; take a look

at these varied rankings for the “best” sports car for 2016:

- Car & Driver – one of the oldest sport car magazines rated the Chevy Corvette #1. Frankly, I loved my old ones from the 1960s, but today’s don’t appeal to me as much…

- US News & World report – this rag rates almost everything, including hospitals & law schools (are they related?). They picked the (ugly) Porsche Boxter as #1, while the ‘vette only rated #4

- Consumer Reports – who I trust the most due to no ads by vendors, although do they really know much about hot cars? They ranked the ‘vette as #13!

How to Rate the RatersTo evaluate the various HIS rating agencies, we’ll be covering a lot of ground in several episodes over the next few weeks, including key questions one needs to ask about their data such as:– Number – how many facilities responded: 10, 50, 100?– Job Class – of respondents: IT staff, clinicians, executives?– Date – when were responses received: this year, past years?

– Size – How big or small were the facilities surveyed:• Were hospitals <25 bed CAH, or 500+ bed AMCs?• Were MD practices a single GP, or 50+ specialty group?

– Products – which were rated? Quite a difference among:• Meditech: Magic, Client Server or Release 6.X• Allscripts: Touchworks, Professional or Sunrise

– And the biggest question and easily the hardest to answer:

“Follow The Money”This great advice from Deep Throat in “All The Presidents’ Men” is as germane today as it was the 70s’:– How much to vendor$ pay to

“participate” in $urvey$?– Can they pay $ome more to

get “preferred” treatment?– Are big $penders

notified of any poor review$?

– How much do vendor$ pay for their report$?

– Any correlation between $pending and rating$?

It won’t be easy to get the answers, but we’ll sure try..

HIStalk Survey• Probably the most fascinating result

will be the responses from HIStalk readers about how they “rate the ratings,” using a series of questions in a survey HIStalk just ran last week.

• Over 40 responses have been received from readers already, which we are compiling in a spreadsheet.

• Credit to Mr. HIStalk for refusing our request to charge a fee to the various rating agencies to “participate.”

• And he won’t let us charge you readers to view the results!

• Ah well, some day he’ll learn how capitalism works…

http://histalk.com/machform/view.php?id=33836

D.I.Y.• After analyzing your scores for rating these many ratings, we’ll

give what might be the best advice for your next selection:– How to “Do It Yourself” – maybe the best people to trust to

research how satisfied users of vendor systems really are, is actually your own users who need to chose a new system.

– References – we’ll explain how simple it can be to solicit relevant references from vendors in an RFI that correspond to your size, location, products, apps/modules, location, etc.

– Questionnaire – a simple form for your users to call their counterparts at current clients of system being considered.

– Score-sheet – how to summarize the calls in a simple SS, with some sample scores we have received in recent selections.

• This “DIY” approach not only gives you far more relevant data, but builds user “buy in” for the impending implementation...

Next Week• We’ll start with the HIS-tory of how this whole thing got started

back in the 1970s with pioneers such as SIDA (Sheldon Dorenfest) and Ron Johnson, and the origins of today’s firms.

• Then we’ll give an overview of the many sources out there for:– Hospitals – reviewers of vendor HIS systems, which will

include leaders like KLAS, Black Book, HIMSS Analytics, etc.– Physician Practices – some of the dozens of reviewers who

rate MD systems for this much larger market, with 785 vendors selling to over 250,000 physician practices.

• We’ll include anecdotal stories from CIOs and vendor reps about their real-life experiences with un-named rating agencies, to give readers some idea of the risks vs. rewards of trusting others to recommend systems, and how to put ratings to good use.

• For questions, comments, or legal action, please contact us at:– vciotti@hispros.com, 505.466.4958 or eames@hispros.com, 413.329.6925