[Status Alert] Delay in Product Events

Happening Announcements
,
1

We are noticing delays in product events lately. Respective teams are aware of the situation and are working on the solution. Updates on the situation will be posted in the same thread. This time the delay has been more than an hour, however, no events are expected to be dropped.

Update: 11pm IST
The issue has been identified and addressed so that events are now being pushed out to apps at a fast clip again. A large number of events have piled up however, which means delays will be expected for another couple of hours while we scale up and drain the backlog.

Update: 1:40AM IST
The piled-up backlog of events has been drained. Delays are no longer expected for events served to serverless apps. We apologize for any delays caused to important app executions. On to the next task now of root-causing this problem so it does not occur again.

Update: 6-May-2020, 9.40 PM IST

The delays noticed in the Freshdesk product events have been fixed around 11 AM IST today. So, no event delays are noticed at our end after the mentioned time.

Note that this recent incident today has only affected Freshdesk product events. All other product’s events were delivered without any noticeable delay.

4 Likes
2

Hi Hemchander, Today again the app functioning is getting lagged by almost 2 hours. Is this issue completely fixed? Client had asked for an update on the same and the reason for the issue. Could you please help?

3

@Arvind_Shanmugasunda

We are witnessing delays in US & EUC events due to some issues in the upstream.

US - The delay has subsided.
EUC - Still we are having delays by a huge margin.

We will keep you posted on further updates.

4

Is there a channel we can subscribe to where we can get alerted when there are issues in production?
@Hem @Saif

3 Likes
5

Update: 6-May-2020, 9.40 PM IST

The delays noticed in the Freshdesk product events have been fixed around 11 AM IST today. So, no event delays are noticed at our end after the mentioned time.

Note that this recent incident today has only affected Freshdesk product events. All other product’s events were delivered without any noticeable delay.