Incident with Pull Requests, Git Operations and Issues
130 points by kevcampb 4 hours ago | 63 comments

a-squirrel-comm 3 hours ago
It looks like today there was maintenance scheduled to deprecate support for certain categories of SSH keys: https://github.blog/changelog/2026-09-22-security-improvemen...

> November 4, 2026: We’ll have a brownout of the removal of the ssh-rsa signature type (i.e., RSA keys using SHA-1) and the diffie-hellman-group-exchange-sha256 key exchange algorithm.

I guess something went sideways.

reply
igleria 3 hours ago
hey how much do I owe you for this INCREDIBLY important information? This is not sarcasm btw.
reply
a-squirrel-comm 3 hours ago
nothing :) hope they'll release a statement soon, this is was an assumption on my side, it might be totally wrong. I had intermittent issues with basic git pull/push ops locally so I went to dig a bit. ps: if you do have money to spare, consider donating to the EFF (https://www.eff.org) though B)
reply
igleria 3 hours ago
thanks a bunch! will do.
reply
qtqtqt 3 hours ago
How to check if one's SSH keys are still compatible?
reply
dist-epoch 2 hours ago
if any of your SSH key files are bigger than 500 bytes, you should rotate them
reply
SkiFire13 2 hours ago
Bigger or smaller?
reply
mschuster91 50 minutes ago
EC keys should be much smaller... this is only valid for id_rsa SSH keys
reply
rarisma 3 hours ago

       - - >  i wake up - -
       |                  |
       |                  V
       github has an outage
reply
eknkc 4 hours ago
Our deploys are tied to GitHub actions and it is becoming a liability at this point. What alternatives do people use? I mean for FOSS there are some decent stuff but we need a reliable provider for private repos.
reply
stack_framer 41 minutes ago
We use GitLab where I work. The only problem is it calls pull requests "merge requests," and I don't think I'll ever get used to saying "MR" instead of "PR."
reply
jaapz 4 minutes ago
Why are all you github people pulling on your requests?

I am requesting to merge my changes into the main branch.

reply
hhh 28 minutes ago
I have the opposite
reply
delf 33 minutes ago
I self-host with GitSocial.

https://gitsocial.org/

reply
bonetruck 51 minutes ago
Forgejo backed by Woodpecker CI running on Kubernetes (Talos) has been amazing! Ditch Github Actions syntax for the simplicity of Woodpecker and containers. You'll thank me later.
reply
jelder 2 hours ago
Git itself usually works, but we gave up on GHA and use https://www.rwx.com/ for tests, previews, and deployments. It’s 2x to 10x faster than GitHub Actions and testing uncommitted changes on real CI infrastructure is a total game changer.
reply
SkiFire13 2 hours ago
Some of our deploys use CircleCI instead, but they were still broken because they couldn't checkout from Github.
reply
saturn_vk 3 hours ago
Self-hosted Forgejo with a bunch of runners might also be interesting for you. The workflow syntax is _almost_ identical to the github one.
reply
pelagicAustral 4 hours ago
My deployments are tied to their container registry, so I'm equally screwed... Been thinking of going back to Dokku, I started working on a third wrapper around Dokku (yeah, I've literally done this twice already), and then self-host and self-provision... I honestly thinking seriously about this.
reply
misswaterfairy 4 hours ago
Genuine question: if you're hosting private repos, is self-hosting an option?

Gitea Enterprise would be worth looking at.

https://about.gitea.com/products/gitea-enterprise/

reply
eknkc 3 hours ago
Honestly, not really wanna deal with self hosting it. Would rather pay for a decent service at this point.
reply
import 3 hours ago
There is already a blacksmith sh if you don't want to selfhost your runners. Or codeberg as a full solution if I am not mistaken.
reply
michaelbuckbee 3 hours ago
+1 for Blacksmith. You need to have a Github org to use it as it won't work on personal accounts, but it feels like a magic want that just makes runners 2x as fast.

I have it for all my stuff (dozens of repos and hundreds of PRs/builds per week) and it's only once in the last year that I've exceeded the free resource levels. I don't know if it's just extra capacity or they're burning VC dollars in the hopes of inspiring posts like this, but either way I feel like I'm taking advantage of them at this point.

reply
crote 3 hours ago
Codeberg only allows F/LOSS software, for most companies that's a dealbreaker.
reply
tcfhgj 44 minutes ago
Private projects usually are not owned by companies
reply
myaccountonhn 3 hours ago
Post receive hook on a bare git repo with gitolite for access control.
reply
rozularen 3 hours ago
self hosted forgejo has been my saviour. im thinking in acquiring a new NUC server to increase compute and storage to my homelab to add some new runners
reply
embedding-shape 2 hours ago
You don't even have to migrate off GitHub if you don't want to.

Begin by refactoring the deployments so they're driven by checked in shellscripts, that accepts a git reference to deploy (prefer immutable commit SHAs over mutable branches/tags). Make it use the stuff it already has checked in, and make the actual GitHub Actions stuff as small and concise as possible, ideally each "run" should just call a shellscript with the git reference.

Then move so the deploys happen not on GitHub Actions itself, but on a "intermediate deploy host" that has the source code and access to the credentials you need, and make this host have one secure entrypoint, where someone (or something) can enter and run shellscripts, after pushing their code as a normal git push. Then switch so the deploys from GitHub Actions instead use this new host to do the deploys. Lock down this host hard.

When GitHub and/or GitHub Actions go down, the developer responsible for deploying stuff, should be able to get temporary git and ssh access to this host, can push any git data straight to the host, and their user on this host can run the deploy scripts. Keep in mind that nothing in your deployment process should depend on stuff that isn't also available on this "deploy host", if you have multiple synced repositories and so on.

Now things mostly work the same as people are used to, with the added benefit that when things go down again (like next week), you'll still be able to continue to deploy stuff.

I've done this refactor/migration for tens of companies in the last year, more or less following this exact template for all of them, with no issues reported from either of them, but lots of praise :)

reply
import 3 hours ago
gitea is amazing.
reply
raddan 3 hours ago
Seconded. Also very easy to set up. I’ve been running it for years, along with webhooks to kick off testing and website deployment jobs. I especially like the swagger browser that generate API calls for you to try in curl. I use the API pretty extensively.
reply
infra_tinker 2 hours ago
[flagged]
reply
thesdev 4 hours ago
Finally a GitHub outage during European workday. Yaaay.
reply
nope1000 2 hours ago
Although it is lunchtime :/
reply
y-curious 4 minutes ago
Isn’t most of the European workday lunch time? :P

I joke, the QoL is really nice

reply
cashbreaker 3 hours ago
I was thinking I'm going crazy with me ssh keys not working on both of my machines.
reply
techterrier 3 hours ago
https is working still!

git -c credential.helper= -c 'credential.helper=!gh auth git-credential' pull https://github.com/myorg/my-repo.git

reply
sognetic 4 hours ago
Can't even pull no more using my key. That's a new low, I'm used to the website being janky nowadays but this is seriously essential
reply
ayhanfuat 4 hours ago
At least they have a very clear error message so you know what is going on.

> git@github.com: Permission denied (publickey). fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

reply
chinathrow 3 hours ago
That's standard git, not coming from GH.
reply
b1tshift 3 hours ago
Just rolled new keys and though I'm going crazy that it's not working
reply
alper 27 minutes ago
Same old same old.

What were the 60'000 people exiting Microsoft doing if not improving Github?

reply
igleria 4 hours ago
Can't they just... sell github back like when the germans sold the nuclear plant back to mr burns?
reply
solyin 4 hours ago
As of 2026-11-04T10:25:34Z the domain MicrosoftPleaseGiveGithubToGoogle.com is still unclaimed.
reply
selectnull 4 hours ago
Yes, really at this point it would be best if MS sold Github to Google and they kill it within a year.
reply
marvstazar 3 hours ago
Pretty good timing now that Rails has released a security update :|

https://rubyonrails.org/2026/11/4/Rails-8.0.5.2-and-8.1.4.1-...

reply
fauzanhilmi 4 hours ago
For a moment I thought my ssh key got breached. Fricking hell Github!
reply
misnome 3 hours ago
At this point, even when there isn't an incident, push/pull have been so unreliable as to fail about 30% of the time.

I can live with the occasional outage but this is a constant problem.

reply
w3ll_w3ll_w3ll 3 hours ago
I feel there is a problem with your connectivity or with the Github datacenter you are being routed to. That has not been my experience.
reply
swe_dima 3 hours ago
Temporary workaround is to do

git config --global url."https://github.com/".insteadOf git@github.com:

so it pushes / pulls using HTTPS

when things are back to normal run:

git config --global --unset url."https://github.com/".insteadOf

reply
feukon 3 hours ago
The day GitHub will no longer be free is approaching... they just can't support the abnormal amount of things being pushed rn
reply
crote 3 hours ago
For most people here GitHub is already a paid platform. Our companies rely on it as a core part of our development workflows, and we pay a decent bunch of money for it. If it was only the free part of GH having issues, it wouldn't get even remotely near this kind of exposure.

Developers are expensive. If I can't ship a feature on time because GitHub issues keeps forcing me to switch to different tasks to work around git/PR/ticket/Actions downtime, then that costs my company real money - despite us not being the source of the AI DDoS that GitHub has happily been welcoming.

To me, the situation is very simple: either 1) rein in the AI deluge, or 2) isolate paying customers from it, or we'll move to an alternative.

For businesses the current state of unreliability would be unacceptable if it was completely free, the fact that it is actually paid just adds insult to injury.

reply
fredsted 3 hours ago
Just set up a Gitea instance, pushed my code and switched my deployment script to it. Took 30 minutes. Github pulls still aren't working.
reply
grishka 3 hours ago
I managed to push a commit over ssh but then the client threw an error before the push fully completed. I can see the commit on the web, but my client still thinks it isn't pushed. And now when I try to fetch I get "permission denied" after a long wait, which is scary. I first thought something got hacked.

Maybe they need a better AI model for their vibe coding. That'll fix everything, right? At least that's what my friends keep telling me when I say that the free LLMs I played around with a bit are incredibly dumb.

reply
chinathrow 4 hours ago
git clone works sometimes, sometimes not.
reply
dude250711 4 hours ago
These days, perhaps, normal undisrupted operation should start counting as an incident...
reply
esher 4 hours ago
Again and again?
reply
crote 3 hours ago
GitHub is down only on days ending in -day.
reply
anon7000 4 hours ago
That is how explosive demand growth works after all. They just aren’t keeping up with increasing capacity or rate limiting people
reply
rvz 3 hours ago
Another incident. Last time we had one was 5 days ago [0]

It's up to you if you want more of this instability. Perhaps you should consider a self-hosted alternative.

[0] https://news.ycombinator.com/item?id=50307843

reply
larsnystrom 3 hours ago
Why haven’t they asked an AI agent to fix all their bugs yet? /s
reply
Ampersander 37 minutes ago
Why does anyone even use Github anymore? Just generate a better forge with AI
reply
refactor_master 3 hours ago
Prompting the wrong things is just a skill issue.
reply
malikolivier 3 hours ago
GitHub outages can be a drain on productivity. If your stack is pulling artifacts from GitHub, and this consistently fails during GitHub outage, you can consider having a file-cache in between.

Some of our customers unexpectedly started using the service we're running as a file-cache around GitHub to mitigate GitHub outages.

We wrote about this use case in our blog: https://www.stablebuild.com/blog/protect-your-builds-from-gi...

reply
jaapz 2 minutes ago
Self promotion is generally frowned upon here
reply