A lease that was never renewed, and the double bill

When you start a generation and close the tab, your work does not vanish. It keeps running in the background, and a helper finishes it if something goes wrong. In August we made that helper about thirty times faster. While doing it, we found a mistake: the helper could run the same generation twice and pay the AI provider twice for one result. It came down to a lease that was never renewed.

When you start a generation, we save a record of it first. While the job runs, it holds a lease on that record. A lease is just a note that says "someone is working on this, leave it alone until this time".

If you close the tab, the job keeps going and finishes by itself. If the job crashes, the note runs out. Then a helper that checks every few minutes sees the abandoned job and runs it again.

A second run must never charge you again. Your credits were set aside on the first try. So the helper skips that step, and only gives credits back if the job truly cannot finish.

The helper picked up 4 jobs at a time and finished them one by one. It checked every five minutes. Jobs that get abandoned tend to be long ones. A five second text reply is rarely left behind, but a three minute video often is. So the helper rescued only about twelve long jobs an hour.

We changed four numbers. It now works on 6 jobs at once, picks up 18 each round, gets more time per round, and checks every 2 minutes. That is up to 360 rescues an hour for 4 minute jobs. We also tested it for real. We staged 8 abandoned jobs, and the helper finished all of them in 15.8 seconds, with real results and no extra charges.

Why only 6 at once and not 60? The limit is memory. To save a finished video, the server holds the whole file for a moment, and a video can be around 100 MB. If the server runs out of memory, every job in the group fails together. So we keep the group small on purpose.

We also let each job run longer, up to 800 seconds. That exposed a worse mistake. A running job is supposed to renew its lease every so often, like checking in. But it only checked in when the AI provider reported progress. Many providers say nothing until they are completely done.

Those jobs never renewed. About six minutes in, the helper decided the job had been abandoned, even though it was still running. It picked the job up and started the same generation again.

This had stayed hidden because jobs used to stop on their own before the lease ran out. Once we let them run longer, the double run appeared. You were charged once. We were charged by the provider twice.

A running job now renews its lease on a plain timer, once a minute, for as long as it is alive. It no longer waits for the provider to say anything. The lease lasts six minutes, so a job can miss several check-ins, for example on a slow connection, before the helper gives up on it. A job that really has died is still picked up within minutes.

Speeding things up showed us three more problems.

Cancel did nothing. If you cancelled a job, it could still finish a moment later and overwrite the cancel. You were charged and the result still arrived. Now a finished job can only be saved if it was still running. If you cancelled, you get your credits back. The refund can only happen once, even if two helpers race to do it.

One person could block everyone. The helper took jobs in the order they were made. Someone with fifty abandoned videos went first, and everyone else waited. Now the helper takes at most three jobs from each person per round.

Very long jobs were retried until we gave up. A generation that was truly too long for the time limit was stopped and started again three times. We paid the provider three times and refunded one credit. Now, after two attempts that run out of time, we stop and tell you why.

We added automatic tests for this. They confirm that rescues run side by side, and that one failing rescue does not stop the others. They also confirm that jobs renew their lease on a timer. A job is allowed to miss a few check-ins before it is called dead. If someone changes renewal back to waiting for the provider, a check fails.

The lesson is simple. A lease only works if it ends exactly when nobody is working. If it depends on a signal that may never come, it will end while someone is still working. To see this for yourself, close the tab in the middle of a video generation and come back later.