Where Jev worked for us, and where it didn't

Last week TypeSafe released a model called Jev. You ask it a closed question, and it gives back a fixed answer type (yes/no, one of a list of options) plus a number saying how sure it is. We moved our four decisions over to it. Here's what worked, what didn't, and one thing we didn't expect.

Some background: GitLoom reads a team's GitHub activity and sends a daily summary. To do that it makes a few hundred small decisions a day about pull requests. Almost none of them need a written answer.

Review threads

The first decision we moved was about code review threads. GitHub gives you two fields. isResolved means someone clicked the "resolve" button. Some teams never click it, and some click it just to tidy their own screen. isOutdated means the code lines moved, which happens when you fix something and also when you rebase.

Neither field tells you the thing you actually want to know: did the reviewer get what they asked for? We used the fields anyway because they were free, and then ignored them later when they were obviously wrong. In the end, all the summary model saw was "3 unresolved comments." That tells you how many, not whether anyone is stuck.

What we liked about Jev

The confidence number mattered more than accuracy. A normal model asked yes or no sounds equally sure whether it has strong evidence or none. Jev gives a probability, so we can set a rule: only act if it's above 0.8, otherwise change nothing. When the model isn't sure, GitLoom behaves the way it did before. That's what made us comfortable shipping.

Price mattered, but not because of the bill. Sorting pull requests into categories costs about six cents per thousand. The real effect was on how it enabled us to rebuild that, which I'll explain below.

Fixed answers can be checked. We can compare the model's answer to one a person labelled by hand. Our category sorter gets 98.0% right on the 85% of cases where it gives an answer, compared to 88.2% for the pattern-matching code it replaced, tested on sixty pull requests we labelled ourselves. Now when we change a prompt, we get a number telling us if it helped.

Searching relevant PRs

Now that the model is fast and nearly free, we ask Jev to tell us which PRs are relevant to a user query the moment the query comes in.

We first filter in the database by team, project, and date, then score each remaining PR against the words the person actually typed, using the PR summary, branch name, and changed file names. Nothing is decided ahead of time, so it works for any topic.

Previously we had a lot of category fields and tag systems that existed only because running a model used to be slow and expensive. Once that's no longer true, they aren't just extra work. They're the wrong design.

Where Jev didn't work

Ranking. The top result was right for every topic we tested. Results two through five often weren't.

You'd expect to just sort by Jev's score. We tried. The score answers "is this PR about the topic?" It's the model's confidence in a yes. Once a PR clearly is about the topic, the score is 0.97, 0.98, 0.99, and the order inside that band is noise. It also says nothing about importance: a PR that renames a permissions constant and a PR that rewrote permissions both get a high score, because both are clearly about permissions. The model was never asked which one matters more.

So sorting by score gives you a list ordered by how sure the model is that each item is on-topic, which is not the same as a list ordered by usefulness.

This part will need some further experimentation or a different strategy. It might not be a shortcoming of Jev itself, but with the way we used it.

What changed for GitLoom users

Two things you'll notice. First, the daily summary no longer says "3 unresolved comments." It tells you whether a reviewer is actually waiting on someone, and only when it's confident enough to say so. Second, asking GitLoom what happened to a topic ("what changed in permissions last month?") now works for any topic, not just the handful of categories we had guessed at in advance.

Neither of these was possible before. The first needed a judgment that GitHub's fields don't give you. The second needed inference cheap enough to run on every question instead of once on ingest.

If you want to see how it works on your own repos, GitLoom is free for 10 days and takes about two minutes to connect to GitHub. The first summary lands in your inbox the next morning.