# Vendor dependency: where your automation ends | Auspex > Bitrix24 documents it: no event retries. Shopify advises against relying on webhooks. Where your automation ends and what to cover first. Source: https://auspex.company/en/blog/automation-vendor-single-point/ [Blog](/en/blog/) / Field Notes # Where your automation ends By: Andrew Maryasov · Aug 27, 2026 · 13 min read Bitrix24 states it plainly in its documentation: there are no event retries. Shopify says you shouldn't rely on webhook delivery at all. GitHub doesn't automatically redeliver failed deliveries. Automation built on top of a SaaS platform inherits these points of failure along with the platform, and your own monitoring doesn't see them, because it measures only your side of the boundary. What costs the most is not what the vendor hides but what it wrote down honestly and nobody read. Below is a list of commitments your vendor never made, and which of them to cover first. ![Automation on top of a SaaS platform and the three points of failure it inherits: no webhook retries, a limit you can't count, a surface that changes without your release](/images/blog/automation-vendor-single-point-en.webp) Three points of failure inherited from the platform On a diagram, an integration looks like two rectangles with an arrow between them: your system and the CRM. The right-hand rectangle is drawn the same way as the left one — with a solid line, as if the same rules applied inside it as on your side. Inside it is someone else's code, someone else's release schedule, someone else's network and someone else's lawyer. You don't see the logs, you don't know the date of the next update, you can't roll back a version, and the only thing you get from that half is an HTTP response. When automation breaks, the first theory is almost always “a bug in the integration,” and the company goes to its contractor. Some of those breakdowns happened on the other side of the boundary, where neither you nor the contractor has permissions or visibility. Finding out which ones is cheaper before the incident. ## Why doesn't a successful response mean the action was carried out? The most expensive class of failures is the one where nothing crashes. The platform accepts the request, returns a success code and doesn't do what you asked. Below is what Auspex measured in its own test runs on a Uspacy portal in August 2026. This is not a complaint about a particular vendor: Uspacy has complete public REST documentation with OpenAPI schemas, which not every platform can claim. It's simply the one we measured, and the boundary looks like this: - list values passed inside a field when that field is created are discarded by the portal — the field is created empty, there is no error, the status is 201 - writing a role's permissions replaces the whole set of pipeline permissions instead of adding to it, so saving one permission wipes the rest — and that succeeds too - when a task is created, three of the five statuses are accepted with a 201 code and silently turned into “planned,” and the substituted value already comes back in the response body - a permission written to a system pipeline returns 200 and isn't saved: the permissions template appears on an entity only at the moment the entity gets its first pipeline of its own - in the error body, the status field equals true — the success flag is set even when the operation didn't go through Each item on its own is a minor thing you can spot in five minutes by looking at the portal. Together they say something else: the response code confirms that the request was accepted, not that the state changed the way you wanted. An integration built on checking the HTTP status treats everything that didn't crash as done. A rule that costs one extra request: after a write with a price attached, re-read the object and compare it with your intent. Not with the response code, but with what you meant to write. Documentation describes the vendor's intent; a live run shows the actual behavior. The difference between the two is yours to carry. ## What does “no retries” mean? A webhook is an automatic call the platform sends to your system when something happens on its side. It looks like a guarantee: an event happened, a request arrived. The vendors that write about this openly say the opposite, and Bitrix24 is the most straightforward of them. In the section on events, its documentation lists two limitations without softening either. > Load cannot be regulated. When mass data changes occur, you will receive many consecutive calls. If a thousand deals are changed simultaneously in Bitrix24, the handler will receive a thousand calls. No retries. If your server does not respond or returns an error, the Bitrix24 queue server will log the failure but will not resend the event. > > — Bitrix24 REST API documentation, Events section In other words: the load is not regulated. If a thousand deals change in Bitrix24 at the same moment, the handler gets a thousand calls. There are no retries: if your server didn't respond or returned an error, the queue server logs the failure, but the event is not sent again. Bitrix24 is no worse than the others here — it simply said it out loud. Shopify's wording is harsher still: > Your app shouldn't rely on receiving data from Shopify webhooks. Webhook delivery isn't always guaranteed, and your app can miss or mishandle events for other reasons, such as handler failures or downtime. For redundancy, use reconciliation jobs to periodically fetch data from Shopify so that your app stays consistent with Shopify's data. > > — Shopify Developer Documentation, Build webhooks In other words: don't count on the data arriving by webhook. Delivery is not guaranteed, and the app can miss an event for reasons of its own — a handler failure or downtime. For reliability, run a reconciliation job that periodically fetches the data from Shopify on its own. That said, Shopify does retry delivery eight times over four hours — and after eight failed attempts in a row it deletes the subscription, with a warning email to the app's technical address. GitHub doesn't retry at all: if your server didn't respond within ten seconds, the delivery is marked as failed, and it can be redelivered only by hand or with your own script, and only for the last three days. The policies differ; the conclusion for you is the same: a process that exists only at the moment of the webhook is not a process but a hope. A periodic reconciliation has to run alongside it: every N minutes, request the list of entities changed within the window and compare it with what you received. The discrepancy for the past week shows your current loss rate, and it's better to know it as a number. A worse case is when you have nothing to reconcile against. In Bitrix24, the method for getting Feed posts doesn't return a post's recipients, so there is no way to check through the API which workgroup a post went to: the only way is to look in the portal. ## Which limit can't you count? Bitrix24 documents its limits in detail, and that's a good thing: the rate limit works on the leaky bucket principle, the counter drains by 2 requests per second, the blocking threshold is 50, on the Enterprise plan the figures are 5 and 250 respectively, and exceeding it returns a 503 with the code QUERY\_LIMIT\_EXCEEDED. The limit is counted per the request's source IP address, so several apps on one server share a common budget. Your second integration silently eats the first one's quota, and in the first one's logs it will look like random platform failures. Next to it lives a second limit — on the accumulated execution time of a method, with the code OPERATION\_TIME\_LIMIT and a 429 in response. Its threshold for the cloud is not given in the documentation: it says only that the value is set by configuration, and the figure of 420 seconds applies to the on-premise version, where the limit is off by default. The same page notes that one and the same request consumes a different amount of this limit in different portals, because it processes a different number of records, which means the resource intensity of a request can't be determined on the app side alone. A limit whose consumption can't be measured from your side can't be designed into the architecture — all you can do is stay under it by guesswork. Uspacy is built the opposite way. We ran 152 calls on its portal, creating and deleting records in different numbers of threads: not one returned a 429, and throughput grew almost linearly — one thread gave about one record per second, four gave 4.9, eight gave 8.1, and after that the curve bent. Reading this as “there are no limits” is the most expensive mistake you could make. What was measured is the behavior of one portal on one day, and Uspacy's public documentation doesn't mention limits or the 429 code, delivery retries, or idempotency — the guarantee that an event delivered a second time won't create a second record. A limit that isn't in the documentation carries no obligation to stay absent either: it can be introduced without an announcement precisely because nothing about it was ever promised. That's why a queue with exponential backoff is needed even where a 429 has never come back. It costs a day of work once and saves you from rewriting the processing pipeline in the week the limit shows up. ## Why does the surface change without your release? On 14 August 2026, a live Bitrix24 webhook on our portal gained three new access permissions that hadn't been there before. A workaround had been built around their absence: the list of workgroups was pulled through the message history. The workaround became unnecessary overnight, and we shipped no release that day. Wider permissions are a pleasant change. The mechanism is the same as for an unpleasant one: the surface your code relies on changed from the outside, and we found out because we were checking, not because we were notified. By our count from the public Bitrix24 changelog (a snapshot as of early September 2026), 355 documented API changes shipped in 91 releases over the first eight months of the year. That is the vendor's pace, and your release cycle has nothing to do with it. How far this can go is visible in the spread of notice periods different platforms have given in recent years. In February 2023, Twitter announced the end of free API access seven days before the cutoff date, and in March of the same year it gave thirty days to migrate off all the old plans. In June 2023, Reddit named its price — $0.24 per thousand calls — about a month before billing began; the developer of the Apollo client publicly asked for three months and didn't get them. At the other end, Shopify commits to supporting each API version for at least twelve months, with at least nine months of overlap between adjacent versions, and Salesforce gave about four and a half years of notice before retiring old versions and voluntarily extended the deadline after consulting the community. The difference between seven days and four years is not a technical characteristic; it is what the vendor wrote about itself. You can read it before signing the contract, not after. The Bitrix24 license agreement reserves the platform's right to restrict access to a portal under excessive load, and the support page explains the procedure: under non-critical load, the first administrator is sent a message asking them to contact support, and if they don't respond, the source of the load is blocked; under high load, it is blocked right away. There is no numeric threshold, no notice period, no appeal procedure. The documented limits show the minimum; the ceiling the vendor keeps for itself. ## Who else is in the chain without a contract with you? The last vendor in the chain is the one you have no contract with. It's the network. In Spain, ISPs block addresses by IP during football broadcasts, and every neighbor on the same edge address goes down along with the pirates: by OONI's count, half a million domains on two thousand addresses, and it is machine calls that take the hit — webhooks, payment callbacks, ingest. Our own production service was unreachable from Spain for several evenings in a row in August 2026 for exactly this reason, while all of its metrics stayed green. A breakdown of that boundary, with numbers and a one-hour check, is in a separate note; the link is below. ## What can you do about it without rewriting everything? Protection against the boundary comes down to four habits and one document. No architectural overhaul is needed here. - keep your own append-only event log on your side, independent of the platform — without it, the question “how much did we lose?” simply has no answer - put a periodic reconciliation next to the webhooks: the webhook brings the event in seconds, the reconciliation picks up what didn't make it - re-read after writing wherever the write has a price attached: compare with your intent, not with the response code - a queue with backoff by default, even when the limit isn't documented anywhere The fifth item is the document, and it is the most important one: a list of what the vendor never promised you. Guaranteed event delivery, the retry policy, the request limit and a way to measure it, the deprecation window, the procedure when you are blocked for load — next to each line there is either a link to the documentation or the word “no.” Everything with a “no” next to it you either cover yourself or deliberately leave uncovered. There is no third option — only one you haven't looked into. ## Where does this advice stop working? The most common overreaction is an abstraction layer “so that switching CRMs is easy later.” It makes every feature more expensive today for the sake of an event whose date and probability are unknown, and in practice it describes exactly the platform it was written against. The second is duplicating the platform with a service of your own. Then you become your own vendor: the same silent failures, only without someone else's support team and without someone else's budget to pay for it. The proportion is simple. Protection should cost less than the expected losses: for a process with twenty inquiries a month, your own event log is overkill; for a processing pipeline that carries every inbound call, it is the minimum. ## Three checks worth starting with - take one integration and write out two lists: what in it you can fix yourself and what you can't. Read the second list carefully: everything on it is no longer yours - open your platform's documentation and find the answers to five questions: are failed event deliveries retried, what is the request limit, how do you measure it from your side, how long does an API version live, and what happens when the load is exceeded — no answer is an answer too - compare the number of events received by webhook over the past week with the number in the platform for the same period If the third item shows a discrepancy, you can skip the rest: the silent loss is already there, and it keeps accumulating for as long as nobody counts it. One more thing that people notice by the second incident: the event log must not live where the integration lives. If your log goes down together with what it logs, you learn about the loss at the same time as everything else — that is, from the customer. Work out that discrepancy for the past week and bring it to a CRM and automation audit. In one meeting, Auspex works through which parts of your automation sit on the other side of the boundary and which of them are worth covering first. The service page is in the links below. Read next - [When the boundary of your automation runs below the API](/en/blog/when-the-network-is-the-limit/) - [Why some automations stick and others quietly die](/en/blog/why-automations-get-abandoned/) - [Why your CRM stopped working two months after go-live](/en/blog/crm-decay-after-launch/) - [What to write into an AI agent development contract](/en/blog/ai-agent-contract-what-to-write/) - [Where to start automating in a small business: 5 boring processes](/en/blog/where-to-start-automation-smb/) - [CRM and automation audit](/en/services/crm-audit/) ## Sources - [Bitrix24 REST API — Events: overview](https://apidocs.bitrix24.com/api-reference/events/index.html) - [Bitrix24 REST API — Incoming event queue](https://apidocs.bitrix24.com/settings/performance/queue.html) - [Uspacy — API Reference: Introduction (authentication)](https://uspacy.readme.io/reference/introduction) - [Bitrix24 REST API — rate limits and method execution time limits](https://apidocs.bitrix24.com/limits.html) - [Shopify Developer Documentation — Build webhooks: reconciliation jobs (primary source of the quote)](https://shopify.dev/docs/apps/build/webhooks) - [Shopify — Verify webhook deliveries: 8 attempts over 4 hours, subscription removal](https://shopify.dev/docs/apps/build/webhooks/verify-deliveries) - [Shopify — API versioning: 12 months of support, 9 months of overlap](https://shopify.dev/docs/api/usage/versioning) - [GitHub Docs — Handling failed webhook deliveries](https://docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries) - [Uspacy — public REST API documentation: index of all methods](https://uspacy.readme.io/reference) Andrew Maryasov Founder, Auspex Founder of Auspex. Implementing CRM and automating business processes since 2015 — 1,500+ projects for 1,200+ companies. [amaryasov.expert →](https://amaryasov.expert) ## Frequently asked What does “vendor dependency” mean in automation? It is the part of your workflow that runs inside someone else's platform and over which you have neither permissions nor visibility. It doesn't come down to the subscription price or the difficulty of migrating: in day-to-day work it is event delivery, request limits, the set of permissions the integration has and the stability of the API contract. Each of these can change without your release and without warning, unless the vendor has promised otherwise in writing. Is it true that Bitrix24 doesn't retry failed webhooks? Yes, and it says so in its own documentation, in the section on events: if your server didn't respond or returned an error, the failure is logged, but the event is not sent again. The same section warns that the load is not regulated — a thousand simultaneous deal changes produce a thousand calls in a row. The documented workaround is offline events, which the app pulls from a queue itself, but the queue holds the object's current state, not its change history, so a thousand edits to one deal produce a single record. Why does monitoring show “all good” when the automation isn't working? Because it measures your side of the boundary: your services are up, the metrics are flat, there are no errors in your logs. An event that never arrived leaves no trace in your system at all — there is nothing to show as an error. A loss like that is found only by reconciliation: comparing what you received with what the platform holds for the same period. Webhooks or polling — which is more reliable? They are not alternatives. A webhook brings the event in seconds and may not bring it at all; polling lags by the polling window but gets everything. In production you need both. Shopify recommends exactly this pairing explicitly; GitHub suggests writing the re-check script yourself. If you have to pick one, go by the cost of a lost event: for notifications a webhook is enough, for payments and calls it isn't. Is it worth building an abstraction layer so you can switch CRMs later? In most cases, no. A layer like that makes every feature more expensive now for the sake of a migration whose date and probability are unknown, and in practice it describes exactly the platform it was written against. It's cheaper to invest in making your documented processes and data portable: they survive a change of vendor, while the integration code will have to be rewritten anyway. How do we find out how many events we're losing right now? Take any window, say the past week, count the events your system received by webhook, and ask the platform for the list of entities changed in the same period. The difference is what you didn't get. If you have no event log of your own, there is nothing to count with — in that case start with the log and run the reconciliation against it. ## Business automation digest 2–3 emails a month — what actually works in CRM and automation. No spam. Unsubscribe in one click. [Free CRM audit →](/en/contact/) ↳ FREE AUDIT 30 min · free · ↓ ## Ready to Get Your Sales Under Control? Free process audit — 30 minutes, and you'll receive: 1,200+ companies since 2015 · 30-day median go-live 01 A map of where you're losing leads 02 Priorities: what to automate first 03 An implementation plan for 2-4 weeks We'll call within 24 hours. The plan is yours, even if we don't work together.