CircleCI Outages: Have They Kept Their Promises in 2022?

CircleCI promise of reliability analysis by StatusGator

At the beginning of April 2022, a massive disruption in CircleCI caused large portions of their cloud offering to be unavailable for users worldwide. It occurred after CircleCI deployed a change to its front end and an auto-vacuum job on one of its core databases.

Due to this outage, CircleCI users were unable to run tests and deploy code.

After the incident, CircleCI promised to prevent these kinds of disruptions in the future

We decided to check CircleCI’s reliability and see if they kept their promises in 2022.

Did they? Read on to find out.

The Data Behind This Article

To evaluate if CircleCI kept its promises of reliability, we analyzed the outage data between January 1 and December 31, 2022. Covering the whole year, including before and after the April disruption, provides a reliable comparison.

We used historical data from our product, StatusGator, which aggregates data from official status pages to create team status pages. Our unique dataset allowed us to understand the reliability of all CircleCI features. 

CircleCI features include Docker Jobs, macOS Job, Windows Job, CircleCI UI, Artifacts, and others.

On its status page, CircleCI also shows the history of CircleCI Dependencies and  Upstream Services. They define Dependencies as third-party services that their infrastructure depends on and Upstream Services as third-party services that impact CircleCI’s jobs. 

CircleCI reports on scheduled maintenance, minor incidents, and major outages. However, for reliability analysis, we only considered major outages.

Every Cloud Vendor's Status in One Place

CircleCI Outages Overview in 2022

Let’s take a look at the history of CircleCI outages across both its own features and third-party services (CircleCI Dependencies and Upstream Services) to analyze reliability in 2022.

CircleCI outages 2022, analysis by StatusGator
CircleCI's outage duration in 2022, analysis by StatusGator
CircleCI outage acroos features, stats by StatusGator

After an initial update on reliability on April 13, 2022, we can see that the number of outages increased in May and June. Yet, no spikes in duration were observed. 

July and August 2022 exhibited a drop in outages and duration. Yet, in September and October, the number of outages and duration increased — looking similar to April 2022. 

October exhibited another increase in outages, yet at the end of the year, the outages and duration dropped significantly.

To understand the difference properly, we should look at outage data before and after the reliability “promise”. But, it is important to note that we compared January–April and May–December. This was because the reliability promise came in April.

Average outage stats on CircleCI's features and 3rd-party services, 2022
Before promiseAfter promiseChange
Outage count4.55.63⬆+0.8 (~18%)
Outage duration4:37:302:47:30⬇-1:50:00 (~41.4%)

While the average number of outages went up by ⬆18%, the average duration for the period after the promise dropped by ⬇41.4%. So overall, the number of outages increased, but they’re not lasting as long. This interesting finding points to an increase in transparency by publishing more incidents and a renewed focus on faster resolution time.

CircleCI outages and duration before and after the promise, analysis by StatusGator 2022

It’s clear from this data that CircleCI had a few tough months of outages. Still, the fact that they brought the average duration down indicates improvement. 

With that in mind, it’s important to remember that CircleCI depends on third-party services. Therefore, we decided to go deeper to understand how much influence these third parties had on reliability. 

HELPFUL TIP – This data is a clear reminder that you should monitor your cloud dependencies – you never really know who causes downtime without doing so. 

Don’t fall behind! StatusGator shows on your status page aggregated data from all your cloud dependencies (over 2,660+ popular cloud providers).

CircleCI Features Outages (without the third-party services)

The graph below demonstrates CircleCI outages across its own features without third-party services.

CircleCI's outage and reliability, analysis by StatusGator
CircleCI's features  outage duration in 2022, analysis by StatusGator

A few interesting points to note: 

  • The total outage duration in 2022 adds up to nearly 31 hours.
  • The longest outage was in January, lasting 9 hours and 25 minutes
  • The highest number of outages in one month occurred in October.

As we can see from the chart, October was notorious for the number of features affected during an outage. Two out of three outages affected 6 features and 5 features accordingly – a significant statistic. 

We decided to look further and calculate their downtime separately.

Outages across CircleCI's own features, 2022
CircleCI featureCount of outagesDuration
CircleCI Insights2
0:20:00
CircleCI UI37:05:00
Docker Jobs716:40:00
Machine Jobs818:50:00
macOS Jobs912:15:00
Pipelines & Workflows612:45:00
Runner20:15:00
Windows Jobs67:15:00
Total4375:25:00

The two main conclusions were: 

  • MacOS Jobs, Machine Jobs, and Docker Jobs were affected by the outages more frequently than other features. 
  • Machine Job’s total outage duration was the longest at 18 hours and 50 minutes. This was closely followed by Docker Jobs, with a total duration of 16 hours and 40 minutes.

Taking this into account, we looked closer at the average number of outages (and duration) before and after the promise. 

We saw the following:

Average outage stats on CircleCI’s own features (without 3rd-party services), 2022
Before promiseAfter promiseChange
Outage count3.53.63⬆+0.13 (~3.7%)
Outage duration3:51:151:43:08⬇-2:08:07 (~56.7%)

When comparing the total outages and duration, we can see that CircleCI features are not solely to blame for the total number of outages and their duration. 

Even after the reliability promise in April, CircleCI’s features’ average outage number slightly increased by ⬆3.7%. But, we did see a significant drop in average duration, with an overall cut of ⬇56.7%.

CircleCI's features before and after the outage in April 2022, analysis by StatusGator

So far, the data suggests that the reliability promise did not reduce outage numbers. Even so, it is clear that the outages are being resolved quicker as the duration is lower. 

Let’s take a look at CircleCI Dependencies and Upstream Services outage statistics to investigate further. 

CircleCI 3rd-party Dependencies and Upstream Services Outages

CircleCI’s Dependencies and Upstream Services also affect uptime, so they must be considered in the analysis. As mentioned above, these are the third parties that CircleCI depends on (such as GitHub and AWS). 

CircleCI's third-party service outages, analysis by StatusGator
CircleCI's third-party service outages, analysis by StatusGator

The key conclusions we made from this dataset are: 

  • Dependencies and Upstream Services caused at least 1 outage per month (except for August and December). 
  • The biggest downtime input from dependencies and Upstream Services was in June and July – an additional 2 hours and 40 minutes in June and 2 hours and 35 minutes in July.
  • In May 2022, CircleCI went down 7 times due to the disruption of the third-party service providers.

An important fact is that partial system outages of Atlassian Bitbucket SSH, Google Cloud Storage, and GitHub Packages were the cause of CircleCI outages in February, July, and November, respectively. If this were not the case, CircleCI would have avoided major outages in those months.

Let’s take a look at the list of CircleCI’s third-party services that they depend on.

Outages across CircleCI's 3rd-party dependencies, 2022
CircleCI connection3rd-party servicesCount of outagesDuration
Upstream ServicesAtlassian Bitbucket API11:05:00
Upstream ServicesAtlassian Bitbucket SSH10:04:00
Upstream ServicesGitHub10:20:00
Upstream ServicesGitHub API Requests20:45:00
Upstream ServicesGitHub Packages42:25:00
Upstream ServicesGitHub Webhooks10:05:00
CircleCI DependenciesGoogle Cloud Networking41:21:00
CircleCI DependenciesGoogle Cloud Storage22:35:00
CircleCI DependenciesMailgun API11:30:00
CircleCI DependenciesMailgun Outbound Delivery22:40:00
CircleCI DependenciesMailgun SMTP11:30:00
Total2014:20:00

From this data, it’s clear that:

  • GitHub Packages and Google Cloud Networking affected the reliability of CircleCI in 2022 most often.
  • The longest outages that affected CircleCI were in connection with the downtime of Mailgun Outbound Delivery, Google Cloud Storage, and GitHub Packages.
  • In total, CircleCI saw 14 hours and 20 minutes of outages in 2022 (due to third-party services). 
Average outage stats on CircleCI features and 3rd-party services, 2022
Before promiseAfter promiseChange
Outage count12⬆+1 (+100%)
Outage duration0:28:451:05:00⬆+0:36:15 (+127%)

We can see that CircleCI Dependencies and Upstream Services caused an increase in average outages and duration after CircleCI went public with the promise of reliability.

If you wish to monitor your third-party services and provide the same level of transparency as CircleCI does, try StatusGator status pages to monitor your dependencies in a single place.

StatusGator Eats Support Tickets for Lunch

Conclusion

CircleCI came very close to keeping its promises.

CircleCI improved the average monthly duration of outages across its own features, but the average number of major outages increased by 18% (3.7% if counting CircleCI’s own features only). 

This should not turn users against CircleCI, since their transparency is admirable within the market. Making a promise with such a level of transparency and providing comments on outages is a brave move for CircleCI. 

Overall, we believe that Rob Zuber, CircleCI’s CTO, is working towards improving reliability despite external dependencies. Shout out to Rob for doing a great job with transparency. We hope 2023 will be better and CircleCI will show ever-improving reliability in 2023. Keep up the great work.

If you want to follow CircleCI, sign up to StatusGator to receive notifications and updates on CircleCI outages.