Build a shortlist of repeated complaints with a local AI model

“I was charged twice” and “Two payments were taken” may describe the same problem without sharing the words you would search for. A small local language model can bring such messages together for review. The useful output is a spreadsheet of candidate pairs, with both original messages and their IDs beside one another.

This walkthrough takes a CSV called tickets.csv and writes the 20 closest pairs to review-pairs.csv, or fewer if there are fewer possible pairs. AI News executed the matching on 12 invented English messages on 18 October, then checked the exact instructions below against the same unchanged input. The example is small enough to read in full and deliberately includes similar-looking messages about different outcomes.

Prepare a small file of short messages

Use a Python environment with Sentence Transformers installed. The observed run used Python 3.12.14, sentence-transformers 6.1.0, transformers 5.20.1 and PyTorch 2.14.1 on a Linux CPU. To install that library version, run python -m pip install sentence-transformers==6.1.0; dependency installation varies with your existing environment. The script selects CPU execution, so a GPU is unnecessary for this small task.[1][2]

In a spreadsheet, create two columns named id and text. Give every row a distinct ID and one short message, then export a UTF-8 CSV named tickets.csv. For a first run, write a dozen invented messages, including two pairs you expect to mean the same thing. Let the spreadsheet quote any text containing commas; Python’s CSV reader uses the first row as the column names.[6]

The model is all-MiniLM-L6-v2, which represents a sentence as 384 numbers. Similarity compares those representations. Its model card says inputs longer than 256 word-piece tokens are truncated by default; keep this exercise to short messages. The first use downloads the model files, then the script performs the matching locally. The model revision in the script fixes the snapshot used for this example.[3]

Make the review sheet

Save a Python file called shortlist.py in the same folder as tickets.csv. Copy the following statements in order, one numbered item per line, without the item numbers. Run python shortlist.py from that folder. The mining function returns scored pairs in descending order, avoiding a duplicate reversed pair; the script keeps the first 20.[4]

1. import csv
2. from sentence_transformers import SentenceTransformer
3. from sentence_transformers.util import paraphrase_mining
4. input_file = open("tickets.csv", encoding="utf-8-sig", newline="")
5. tickets = list(csv.DictReader(input_file))
6. input_file.close()
7. assert len(tickets) >= 2 and all(r.get("id") and r.get("text") and None not in r for r in tickets), "Use id,text headers, nonempty rows and quoted commas."
8. model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", revision="1110a243fdf4706b3f48f1d95db1a4f5529b4d41", device="cpu")
9. pairs = paraphrase_mining(model, [r["text"] for r in tickets], show_progress_bar=False)
10. preview = [[round(float(score), 4), tickets[i]["id"], tickets[j]["id"], tickets[i]["text"], tickets[j]["text"]] for score, i, j in pairs[:20]]
11. output_file = open("review-pairs.csv", "x", encoding="utf-8", newline="")
12. writer = csv.writer(output_file)
13. writer.writerow(["cosine_similarity", "id_a", "id_b", "text_a", "text_b"])
14. writer.writerows(preview)
15. output_file.close()

Open review-pairs.csv. Each row contains the similarity score, the two IDs and both messages. The “x” file mode refuses to replace an existing output: move the old review sheet or choose a new output filename before rerunning. If the input check fails, check the exact id,text headers, empty cells and CSV quoting.

Read what the model actually put first

Our 12 messages produced all 66 possible unordered pairs: 12 × 11 ÷ 2. Four deliberately paraphrased issue pairs appeared at ranks 2, 4, 5 and 7. But the highest-scoring pair was an instructive mismatch. These selected rows reproduce the observed ranking; scores are rounded to four decimal places.

• Rank 1, cosine similarity 0.9285: “The refund has arrived in my bank account.” / “The refund has not arrived in my bank account.”
• Rank 2, cosine similarity 0.6650: “The password reset email has not arrived.” / “I requested a password change, but no reset link reached my inbox.”
• Rank 3, cosine similarity 0.6186: “The mug arrived with a cracked handle.” / “My mug is fine; the cardboard box was crushed.”
• Rank 7, cosine similarity 0.5292: “I was charged twice for the same order.” / “Two payments were taken for my single purchase.”

The first pair differs by “not”, which reverses the outcome. Its 0.9285 score measures closeness between two sentence representations; it is not a 92.85% probability that they describe the same issue. Cosine similarity is the library’s default comparison, not an equivalence verdict. That is why this output is a review shortlist.[5]

Add your own review column and give accepted pairs an issue label, such as “password reset email missing”. Check the original messages before using those labels to count recurring problems. The damaged-mug pair should remain separate because one person’s mug is intact. The two double-charge messages, despite their lower score, belong together in this invented example.

A useful first session ends with a few groups you can explain: these messages concern the same failure, and these merely use similar language. Keep the IDs and the original wording. They let you move from a plausible cluster to a complaint pattern you have actually checked.

Report an error or suggest a correction