Sorting email with AI: classification, assignment, and handling exceptions

A system that reads incoming mail and sorts it works well on normal messages. The problem is that the emails that waste time aren't normal: the ambiguous ones, the ones forwarded three times, the ones where the information sits in the attachment and the body just says "as agreed." It's on those that you find out whether the system removes work or adds to it.

The useful question isn't whether a model can read email, because it can. It's a different one: what do we let it decide on its own, what should it only propose, and where does what it didn't understand end up.

Categories: few, distinct, each with an owner

The categories aren't an ontology of your company: they're the list of paths a message can take. If two paths lead to the same desk and the same work, they're a single path.

• A category exists if it changes who takes on the message or what happens next. Otherwise it's a decorative label.
• Every category has an operational owner: a person or group that responds, not a generic address everyone reads and no one monitors.
• There must be a "needs review" category, and using it shouldn't be seen as a failure of the system: it's the correct behavior in front of an ambiguous message.
• Priority is an attribute separate from category. An urgent complaint and a routine complaint go to the same place, in a different order.

What it can decide and what it must propose

There's a single criterion for splitting up the responsibilities: how much it costs to fix the error after it's been made. If the fix is immediate and visible, the automation can decide. If the error spreads silently into other systems, it can't.

The row about data extraction is the one that gets skipped most often, because it's also the one that saves the most time. A workable compromise is to show the extracted data next to the point in the message it was taken from, so confirming it takes a second instead of a re-read.

The messages you should really test on

A test sample made of tidy emails proves nothing. The useful sample is built by taking, from the real inbox, the cases that an experienced person solves by slowing down.

• The message that contains two different requests: an order and a complaint in the same email.
• The forward with the whole history underneath, where the real request is in the first three lines and the longer text is the old one.
• A person's reply to an automated notification, arriving with a subject line that has nothing to do with it.
• The attachment that contains the information, with the message body empty or reduced to "see attached."
• The attachment that's a photo of a document, crooked and with the shadow of a hand in it.
• The message with the wrong subject line because the sender replied to an old email instead of writing a new one.
• The incomplete request: missing the code, missing the address, missing the quantity.
• The automated messages: out-of-office replies, delivery failures, service notices from vendors.

Confirming before the actions that matter

Human confirmation works if it's fast. If it requires reopening the message, finding the data, and rewriting it, people stop confirming and start approving without looking, which is the worst situation of all.

• Show, next to each extracted piece of data, the point in the message it comes from.
• Distinguish "not found" from "found but uncertain": these are two situations that call for different actions.
• Allow a field to be corrected without restarting the whole process.
• Set a threshold below which the message goes to review regardless, decided together with the people doing the work and revisited after the first few weeks.

The review queue: where the doubtful cases end up

Everything the system doesn't resolve must end up in a single place, with an owner and a maximum sitting time decided by you. A queue with no owner turns the system into a second place where requests get lost: it's the most common way of making the starting situation worse.

• Every correction is logged: what the system proposed, what was right, who corrected it.
• Corrections are counted by type, not one by one: they're used to understand where the system is systematically wrong.
• The queue has an expected size. If it grows, that's not a matter of patience: it's a signal that the categories or thresholds aren't working.

Fixing recurring errors

1. Group the corrections by type, at the end of each week in the early stages.
2. Distinguish between three causes: a badly defined category, information missing from the message, a flaw in the system.
3. Address the right cause. A badly defined category gets redefined together with the people doing the work, not compensated for with ever-longer instructions.
4. Retest on the same sample before putting it back into operation, to check you haven't broken the cases that were working.

When the information isn't in the message, no model can infer it. In that case you act upstream: the site's form asks for the code, the customer's automatic signature includes the reference, the vendor gets instructions on how to write the subject line.

When automating isn't worth it

• If volume is low and one person clears the inbox in a few minutes: the time saved doesn't repay the supervision.
• If the categories are still being debated between departments: the system would automate a disagreement.
• If no one can monitor the review queue every day.
• If the mailbox receives particularly sensitive data and it hasn't yet been decided who can see it, where it resides, and how long it's kept.
• If the downstream process is already disorganized: sorting faster into a mess just produces the mess faster.

In several of these cases, the workable path is narrower and more useful: don't sort everything, just one well-defined, high-volume category, leaving the rest as it is today.

What this guide doesn't cover

Here we cover interpreting email: categories, testing, and the boundaries of automated decisions. How to collect, without losing them, requests arriving from different channels — site, email, messaging, phone — is covered in a dedicated guide. Extracting data from structured documents, with the checks it requires, also has its own guide.