Moderation before the credit reservation: checking first

On 20 October the owner decided that every part of genvi refuses certain content. That includes sexual content, anything sexual involving a minor, threats, graphic gore and instructions for self-harm. Battle scenes and horror are still fine. The next question was where to run the check. We run it in exactly one place, before any credits are set aside, so a refusal costs you nothing. That is moderation before the credit reservation.

You can reach generation in many ways: the website, the phone app, chat, the agent, and the developer tools. If we put a check at each door, one door could end up without it. Instead, every one of those ways passes through the same first step of the generation engine. So that is the one place we check.

The order inside that step matters too. We check what the AI model will really receive. That means after any pictures you tag with an @mention have been added, and after extra pictures beyond the limit have been dropped. The check finishes just before credits are set aside. So a refused request returns before anything is charged, and no job is left behind either.

Checking is a call to another service, and generation already waits on several. So we start the check early and let it run alongside the other lookups, such as your project, your keys and the price. We only wait for the answer at the last moment.

One shared piece of code makes the check. It began in our Explore feature and was moved to be shared by everything. The checking model looks at one picture at a time, so your words and each picture are sent as separate checks, all at once.

On top of that sits our policy. It says which things block a request: the categories above, but not plain violence, harassment or hate. It checks up to eight input pictures, with the first and last frames of a video first, and it includes pictures added by an @mention. When it refuses, the message names the reason in plain words. If you started from an image to make a video, and the image was the problem, the message says so.

If the checking service is unavailable, we let requests through, make a note of it, and still charge for the run. We chose this because an outage in one service should not stop every creator from working. There is one exception: a simple local check still refuses an obviously sexual prompt about a child, even when the service is down.

Sharing to Explore works the other way. A share is public, so it uses a stricter list, and if the check is down, the share is blocked. You will see "Sharing is paused for a moment, try again shortly."

When you close a tab, a helper finishes your generation in the background. That job already passed the check when it first started. So the helper skips the check on the second run. Asking again would add a delay and tell us nothing new.

The part of the website that starts a generation expected a refusal to arrive in one form. But the server sent it in another. So when you broke the rules, you saw "Generation failed (422)" instead of the real reason. It now understands both. A refusal with no explanation is almost as bad as no refusal, because people just try again.

The same day, we also switched on the safety filters that the AI providers offer, where they document them. For some providers we turned filters on only where the provider says they exist. Sending a setting that does not exist makes the provider refuse the request after we have already set credits aside.

This also fixed a gap. One video model had its filter off by default, so its clips had gone out unfiltered. Now it is on. One image model's sexual content filter was off by default, so we turned it on, while leaving other filters off because they block battle scenes. One provider has no way to set a filter. So if its own filter blocks a result, we no longer deliver it and you get your credits back.

Automatic tests confirm the whole behaviour. A sexual prompt is refused with a clear reason before credits are touched. A battle scene passes. An outage lets requests through and still charges. A rescued job is not checked again. Shares to Explore are held when the check is down. And a refusal that arrives in the other form is shown properly.

The idea is the same one we use for pricing. Decide once, in the shared path, in an order that cannot charge you for something we refuse. Our safety rules live in two files, instead of in every area of genvi.