On July 7, 2025, between 21:14 UTC and 22:34 UTC, Copilot coding Agent was degraded and non-responsive to issue assignment. Impact was limited to internal GitHub staff because the feature flag gating a newly released feature was enabled on internal development setups and not in global GitHub production environments.
The incident was mitigated by disabling the feature flag for all users.
While our existing safeguards worked as intended—the feature flag allowed for immediate mitigation and the limited scope prevented broader impact—we are enhancing our monitoring to better detect issues that affect smaller user segments and reviewing our internal testing processes to identify similar edge cases before they reach production.
On July 7th, 2025, between 18:20 UTC and 22:10 UTC the Actions service was degraded for GitHub Larger Hosted and scale set runners. During this time window, 9% of GitHub Larger Hosted Runners and scale set jobs saw a delay of at least 5 minutes before being assigned to a runner. Impact was more apparent to customers that didn’t have pre-scaled runner pools or who infrequently queued jobs during the incident window. This was due to a change that unintentionally decreased the rate at which we notified our backend that new scale set runners were coming online, and was mitigated by reverting that change. To reduce the likelihood and impact time of a similar issue in the future, we are improving our detection of this failure mode so we catch it in earlier stages of development and rollout.
On 7/3/2025, between 3:22 AM and 7:12 AM UTC, customers were prevented from SSO authorizing Personal Access Tokens and SSH keys via the GitHub UI. Approximately 1300 users were impacted.A code change modified the content type of the response returned by the server, causing a lazily-loaded dropdown to fail to render, prohibiting the user from proceeding to authorize. No authorization systems were impacted during the incident, only the UI component. We mitigated the incident by reverting the code change that introduced the problem.We are making improvements to our release process and test coverage to catch this class of error earlier in our deployment pipeline. Further, we are improving monitoring to reduce our time to detection and mitigation of issues like this one in the future.
On July 2, 2025, between 1:35 AM UTC and 16:23 UTC, the GitHub Enterprise Importer (GEI) migration service experienced degraded performance and slower-than-normal migration queue processing times. This incident was triggered due to a migration including an abnormally large number of repositories, overwhelming the queue and slowing processing for all migrations.We mitigated the incident by removing the problematic migrations from the queue. Service was restored to normal operation as the queue volume was reduced.To ensure system stability, we have introduced additional concurrency controls that limit the number of queued repositories per organization migration, helping to prevent similar incidents in the future.
On July 2nd, 2025, between approximately 08:40 and 10:16 UTC, the Copilot service experienced degradation due to an infrastructure issue which impacted the Claude Sonnet 4 model, leading to a spike in errors. No other models were impacted.The issue was mitigated by rebalancing load within our infrastructure. GitHub is working to further improve the resiliency of the service to prevent similar incidents in the future.
Due to a degradation of one instance of our internal message delivery service, a percentage of jobs started between 06/30/2025 19:18 UTC and 06/30/2025 19:50 UTC failed, and are no longer retry-able. Runners assigned to these jobs will automatically recover within 24 hours, but deleting and recreating the runner will free up the runner immediately.
On June 30th, 2025, between approximately 18:20 and 19:55 UTC, the Copilot service experienced a degradation of the Claude Sonnet 3.7 model due to an issue with our upstream provider. Users encountered elevated error rates when using Claude Sonnet 3.7. No other models were impacted.The issue was resolved by a mitigation put in place by our provider. GitHub is working with our provider to further improve the resiliency of the service to prevent similar incidents in the future.
On June 26, 2025, between 17:10 UTC and 23:30 UTC, around 40% of attempts to create a repository from a template repository failed. The failures were an unexpected result of a gap in testing and observability.We mitigated the incident by rolling back the deployment.We are working to improve our testing and automatic detection of errors associated with failed template repository creation.
On June 26th, between 14:42UTC and 18:05UTC, the GitHub Enterprise Importer (GEI) service was in a degraded state, during which time, customers of the service experienced extended repository migration durations.Our investigation found that the combined effect of several database updates resulted in the severe throttling of GEI to preserve overall database health.We have taken steps to prevent additional impact and are working to implement additional safeguards to prevent similar incidents from occurring in the future.
This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.