Resource deduplication, and why syncing is a daily job

X puts it like this:

“All resources are deduplicated within a 24-hour UTC day window. If you request and are charged for a resource, requesting the same resource again within that window will not incur an additional charge.”

The unit is the resource, not the request or the endpoint. Post 1795432198765432198 costs $0.005 the first time it is returned to you today, and nothing on any subsequent response today — whether that response comes from the bookmarks endpoint, a lookup, a timeline, or the same call repeated.

The window is a UTC day, not a rolling 24 hours from your first request. That distinction is the one that catches people out, and it has a scheduling consequence below.

Take a walk through a bookmark list that stops at the first post ID you already have. Suppose seven new bookmarks are waiting and you read in pages of five: the first page is five new posts, the second page is two new posts and three you already stored. Those three are the overlap, and you pay for them because they came back in a response.

# Good: one paid pass per UTC day, and manual syncs after it are near-free.
sync at 06:00 UTC          → pays for genuinely new posts + one overlap page
user taps "sync now" 14:00 → overlap already paid today; only new posts cost

# Bad: a job at 23:55 local time in UTC+2 runs at 21:55 UTC one half of the
# year and 22:55 UTC the other, and can land twice in one UTC day or skip one.

One paid sync per UTC day. Because the overlap re-charges once a day and no more, a single scheduled pass per UTC day is the cheapest schedule that never misses anything. Align it with whatever you do afterwards — generating a digest, running a classifier — so the data is read once and used once.

Manual sync can be generous. A “sync now” button pressed later the same UTC day costs only the resources that are genuinely new, because the overlap has already been paid for. This is the rare case where the user-friendly option is also the cheap one, so there is no reason to rate-limit it aggressively.

Polling multiplies cost by nothing. Checking hourly does not cost twenty-four times a daily check — the repeated resources are free — but the first check of each new UTC day still pays the full overlap, and each poll that finds new posts pays a fresh page’s worth of waste. You get no new information for the trouble, since a one-result probe already tells you whether anything changed.

The practical version: meter every call by resources actually returned and charge it against a per-user budget regardless of whether you expect it to be free. Deduplication then shows up as spend coming in under forecast, which is a pleasant surprise rather than a load-bearing assumption. That metering is also what makes the Developer Console spending limit a backstop rather than your only line of defence.

Reading it as a cache. It is not one. X still serves the request and still returns current data; the only thing that changes is whether the meter ticks. You are not getting a stale copy of the post, and you do not need to invalidate anything.

Assuming it is a rolling 24 hours. It is a UTC day window. A resource first read at 23:00 UTC is not free at 22:00 UTC the next day; it stopped being free at midnight.

Trying to exploit it with bigger pages. Deduplication makes a repeat read free; it does nothing for a first read. Asking for 100 results to find five new posts still costs 100 post reads the first time, and walking in small pages is what bounds that waste.