ChocolateMoo53 - Invidious, Invidious API outage – Incident details

Invidious, Invidious API outage

ResolvedMajor outage5 hours 30 minutes

Affected

Invidious Companion #1

Degraded performance from 11:45 PM to 5:16 AM

Invidious Companion #2

Degraded performance from 11:45 PM to 5:16 AM

Invidious Companion #3

Degraded performance from 11:45 PM to 5:16 AM

Invidious Companion #4

Degraded performance from 11:45 PM to 5:16 AM

Invidious Companion #5

Degraded performance from 11:45 PM to 5:16 AM

Invidious Companion #6

Degraded performance from 11:45 PM to 5:16 AM

  • PostmortemUTC

    What happened: From 23:35 UTC on Oct 29, videos started failing. This is a normal response, as the same issue happened not too long ago. However, this was an exception, as it had continued on for longer than normal and the working time of the software would fluctuate too often.
    Cause: New connections hung for a long time as YouTube blocked more requests or made them wait extra long periods. Idle per-video connections had been piling up.
    Fix: At first, the version was downgraded from a recent update. Instead, the actual fix was in the backend software. Backends now closes idle connections sooner and no longer opens warm sessions. The updated image was published at 04:54 UTC.
    Follow-up: An updated message about the actual status is to be made. If the problem lasts for too long, identify the issue automatically because YouTube is not the only factor in the companion's failure.

  • ResolvedUTC

    This incident has been resolved. Thanks for the patience, investigation work was about 2 hours. The playback rate has gone much higher since this patch. The status page will be updated once again in case of more playback faults.

  • MonitoringUTC

    A fixed was published on the backend, this will not fix any YouTube errors, but it has not provided any problems since deploying the fix.

  • IdentifiedUTC

    The problem is not with just slow YouTube responses since the incident started, as it would have properly resolved itself by now, after investigations, it seems some of the problems are coming from a bug in the internal system as well, which is causing some extra issues with this degraded response. Other instances are not affected as they do not run the same code in the backend. The issue identified is being fixed.

  • InvestigatingUTC

    (Initially created as an automatic response, this is an updated description from the initial incident)

    An initial slowdown of requests has been identified, but usually this automatic warning is just for YouTube problems itself, as this checks for degradation of the source, which is YouTube. If this response is created, you may see an outage of the service as a whole.