Methodology
How vendors are assessed, what a pro or a con has to clear to be published, and how the ratings at the foot of each review decide the awards.
Rubric v0.1-draft · updated · 1 of 1 vendors reviewed in depth
What we publish
Each in-depth review makes a case in prose and lists what the vendor does well and where it falls short. Those pros and cons are the product; they are specific enough to argue with, and a review cannot be published without at least one con. Ratings are secondary and are placed accordingly: at the foot of a review, after the argument, scoring the vendor 1-5 on each of the five criteria below using only evidence a reader can check - published documentation, published pricing, named integrations, and public compliance material. Those five figures exist to decide the awards and are used for nothing else. The directory is alphabetical and carries no score at all. Where a vendor publishes nothing on a criterion, it rates low on that criterion. That is a deliberate choice: this index measures what a buyer can verify before signing, and a capability that exists but is undocumented is one a buyer cannot verify.
There are three kinds of article here, and they do different jobs.
- In-depth reviews assess one vendor. The output is a list of pros and a list of cons, each one a specific claim a reader can disagree with, followed by the argument behind them. A review cannot be published without at least one con — that is a schema rule, not a house style, and it exists because a directory of only-strengths published by one of the listed vendors would be advertising.
- Comparison battles take two vendors and say where each of them wins. Both sides get a wins list. There is no combined score and no declared victor, because which of the two is better depends on what the reader is buying it for.
- Rankings list vendors in order — first, second, third — with a “why” for each placing. The order is an editorial judgement argued in prose, not a score computed from the rubric, and a ranking carries no ratings. The number of slots is fixed by a count field that must match the entries, so a “Top 10” with eight rows cannot publish as a “Top 10”.
Reviews and battles are not ordered by score, and the directory is alphabetical. The awards are the only pages on this site ranked by the rubric; a ranking is an editorial order, argued separately.
Every article of a kind is written to the same outline. A review has the same five sections in the same order, every time — what the product is, where it is strong, where it falls short, who it suits, and what would change our mind. A battle has three. A ranking has no prose outline of its own; its structure is the ranked list itself — one numbered slot per vendor, each with its “why” — and the slot headings are generated from the order rather than typed. The headings are generated from that outline rather than typed by whoever wrote the piece, so two reviews can be read against each other section by section instead of being two essays that happen to share a subject.
Ratings, and the one thing they are for
At the foot of each in-depth review — after the argument, not before it — is a small table rating the vendor from 1 to 5 on each criterion below. Those numbers exist to decide the awards and are used for nothing else. They do not appear on the directory, on a card, in search results, or in a comparison.
The placement is structural rather than a matter of layout taste: ratings are a separate field on the document, not a block inside the article body, so an author cannot move the table up into the middle of the piece.
The criteria are structural too. Every review scores the same 5, in the same order, because they are fixed fields on a form rather than a list a writer adds rows to — there is no way to skip one quietly, score one twice, or invent a sixth. What this publication chooses is how much each one counts, which is the table below. A review that leaves a criterion unrated is published as it stands and its vendor is excluded from every award, rather than being scored zero for the gap.
Each rating is rescaled to a 0–10 range, multiplied by the criterion’s weight, and summed. Weights always sum to 100%; the build fails if they do not, because a weighted total is only comparable between vendors when the weights are a complete distribution. Vendors on the same total share a position — =2 rather than 2 and 3 — and within a shared position the order is alphabetical. There is no hidden tiebreak.
Criteria and weights
| Criterion | Weight |
|---|---|
| Agent capabilityWhat the agent completes end to end without a human in the loop, across voice and digital channels — as distinct from what it drafts, suggests, or hands off. | 25% |
| Evaluation and QAWhether the platform ships tooling to measure its own quality: offline evaluation sets, simulation, regression testing before release, and automated review of live conversations. | 20% |
| Enterprise readinessDeployment options, data residency and retention controls, administrative permissions, audit logging, and named compliance certifications. | 20% |
| Integration depthPrebuilt connectivity to the systems the agent has to sit inside: telephony and CCaaS, CRM and case management, knowledge sources, and the data warehouse. | 20% |
| Published evidenceHow much of the vendor's claim can be verified from public material — documentation, pricing, benchmarks, named customers — without a sales conversation. | 15% |
Evidence standard
This site separates three kinds of claim and holds them to different standards.
- Facts — pricing, deployment model, integrations. Every one carries a source URL and the date it was read. A fact without both is rejected by the build, so an uncited figure cannot be published by accident. Facts live in the vendor record and are the only thing on a profile that is not somebody’s opinion.
- Pros and cons — judgements, attributed to the review that made them and to the author who signed it. They are specific by design: “no SOC 2 report published” is checkable, “enterprise-ready” is not.
- Ratings — the 1–5 scores that feed the awards. Each one carries a written rationale of at least forty characters, published beside it. A rating with no stated reasoning is rejected the same way an uncited fact is.
The facts published for every vendor are:
- Starting price
- Deployment model
- Channels (where published)
- Named integrations (where published)
Who is eligible for an award
Only a vendor whose in-depth review rates every one of the 5 criteria above. Right now that is 1 of the 1 vendors listed.
A vendor with no review, or with a review that rates only some criteria, is excluded from every category rather than being scored zero on what is missing. An unrated criterion is a gap in our coverage, not a finding about the vendor, and treating it as a zero would publish a total nobody earned. A category with fewer than two eligible vendors is not published at all — an award won from a field of one is not a result.
How often this is reviewed
Cited facts are treated as stale after 180 days, and the build warns on any that pass that age. Each vendor profile shows the date its facts were last checked. Assessments change when the evidence changes; when they do, award winners recompute automatically, because no winner is written down anywhere.
Awards
This site does not currently publish awards.
What this is not
These assessments are editorial opinion, not a certification, an audit, or a recommendation to buy. They reflect publicly available evidence as of the dates shown. Nothing here is a statement about how any platform will perform for a particular organisation, and no vendor pays to appear, to be reviewed, or to win.