Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

atproto firehose throws errors #1059

Closed
mackuba opened this issue May 17, 2024 · 10 comments
Closed

atproto firehose throws errors #1059

mackuba opened this issue May 17, 2024 · 10 comments
Labels
infra

Comments

@mackuba
Copy link

mackuba commented May 17, 2024

When trying to connect to the firehose and read some events from it, I sometimes get errors, which look for me like this:

#<Faye::WebSocket::API::ErrorEvent:0x0000000106d44318>

This is even after I started receiving some data - events are coming, then they stop, nothing happens for a bit, and then error, maybe more errors, and then after a while it continues printing events.

I inspected the details of the error, and it looks like the server is sometimes returning status 502:

@message="Error during WebSocket handshake: Unexpected response code: 502",
@pathname="/xrpc/com.atproto.sync.subscribeRepos?cursor=124415",
@status=502

@headers={
      "content-type"=>"text/html; charset=UTF-8",
      "referrer-policy"=>"no-referrer",
      "content-length"=>"332",
      "date"=>"Fri, 17 May 2024 20:35:11 GMT",
      "connection"=>"close"
    },

@buffer="\n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>502 Server Error</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Server Error</h1>\n<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>\n<h2></h2>\n</body></html>\n"
@snarfed
Copy link
Owner

snarfed commented May 18, 2024

Ugh, thanks, not ideal. Will look.
@snarfed snarfed added now infra and removed now labels May 19, 2024
@snarfed
Copy link
Owner

snarfed commented May 24, 2024 •
edited
Loading

I've made some significant infra improvements over the last couple days, including to the (outgoing) firehose's stability, and I'm not seeing it serve these 502s any more. @mackuba hopefully you're seeing the same thing now?
@mackuba
Copy link
Author

mackuba commented May 24, 2024

Just tried it but still not looking great:

Screen Shot 2024-05-24 at 21 59 51
@snarfed
Copy link
Owner

snarfed commented May 24, 2024 •
edited
Loading

Hmm! That's surprising. My logs show that BF was serving them pretty consistently for a while, but they stopped altogether a few days ago. The last time I see that we served a 502 to subscribeRepos was 5/21 07:39:04 UTC.

image
@mackuba
Copy link
Author

mackuba commented May 24, 2024

Could it be served by something before your app (Nginx or whatever) when it can't get a response from your app?
@snarfed
Copy link
Owner

snarfed commented May 24, 2024

Yup, definitely. It currently restarts about once a day (for reasons 😐), and any open connections when that happens get 502ed. Looks like you caught it at one of those restarts. Try again? Apart from those, it's not serving many/any other 502s as far as I can tell.
@mackuba
Copy link
Author

mackuba commented May 24, 2024

Hmm… looks good at the moment, but I'll try a few more times at different times :)
@mackuba
Copy link
Author

mackuba commented May 24, 2024

Btw, what's the buffer size for your firehose? It looks like it goes back <24h?
@snarfed
Copy link
Owner

snarfed commented May 24, 2024

5000 seqs, but now that it's behaving better I should probably drop that limit and allow full history.
snarfed added a commit that referenced this issue May 28, 2024
@snarfed
hub: bump up ROLLBACK_WINDOW drastically to effectively all of history
973e29f 
for #1059
@snarfed
Copy link
Owner

snarfed commented Jun 6, 2024

Hi again! I fixed #1091 today, which should stop the daily restarts, so I think these 502 errors should be fully gone now. Websocket subscriptions will still close with HTTP 101 after 1 hour, but that seems reasonable, clients should hopefully handle that and reconnect ok.

Let me know if you still see this problem!
@snarfed snarfed closed this as completed Jun 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infra
2 participants
@mackuba @snarfed