Powered by Outside

We're Back

Dec 19, 2024 at 17:30
by Brian Park  
photo
Server Farm Employee Unplugs Unknown Cable to Charge eMTB Before Going Away on Holiday. Photo: Kelly Sikkema


Over the past several days Pinkbike experienced an extended outage. No, we weren't hacked, and no, we didn't get paid by a shadowy cabal of MTB brands to avoid reporting on some of the difficult news that's come out this week. (shadowy cabal: you have my number)

bigquotesPinkbike and Trailforks have had prolonged outages over the past several days due to challenges during a planned upgrade.

Our team has been working around the clock to bring Pinkbike and Trailforks back. The sites are now operational and stable, and the improvements to enhance performance, reliability, and durability are ongoing. To ensure this doesn’t happen again, we are implementing more reliable backups, better recovery processes, and improved scheduled disaster testing.

We appreciate your patience and understanding during this time.—Devin Lehman, Product Manager


Needless to say it's been a weird few days. We almost had to go and interact with the real world for a moment. We've got a backlog of stories to get published and a ton of news to catch up on so bear with us as we get back up to speed! It's good to be back.

Author Info:
brianpark avatar

Member since Dec 29, 2010
253 articles
Must Read This Week
Results: Elite Finals at the Loudenvielle DH World Cup
106383 views
Jackson Goldstone Shares Why he was Disqualified at the Loudenvielle World Cup
88669 views
Results: Elite Q1 Qualifying at the Loudenvielle DH World Cup
79427 views
Results: Elite Q2 Qualifying at the Loudenvielle DH World Cup
75336 views
How to Watch the 2026 Mountain Bike World Cups
59021 views
First Ride: Specialized’s New DH Bike Has a Wild Drivetrain & a Lot to Prove
47300 views
Norco's New $4,500 Torrent DH - Ready for Big Moves & the Bike Park
42263 views
First Ride: A Lot Has Changed About The New Revel Ranger
33571 views

149 Comments
  • 3191
 Condolences to my family for enduring my MTB talk for 2 days
  • 190
 There is SO much to talk about!!!!!
  • 20
 @mariomtblt Those brave souls kept us going....
  • 70
 Thoughts and prayers
  • 30
 @racecase: so much room for activities!!!
  • 71
 @racecase: My wife, who is very into cycling, asked me last night to stop talking to her about GT and Rocky Mountain since it was bumming her out.
  • 50
 According to my GF's daughter, I'm autistic. So, I guess they enjoyed the break from me talking about trains.
  • 10
 Unplugged "WAS" the thing in '90 and will be soon now.
BTW this thing doesn't fit in EU and not only.
220 V doesn't feel any better...
  • 2300
 Dude, I have had the most productive 3 days at work in a looong time.
  • 390
 You only hurt yourself.
  • 20
 @cmi85: this is so true
  • 40
 @Hookem34 100% this. It wild what you can get done when you don't check PB home page every 11 minutes.
  • 61
 @dmackyaheard: You can make it 11 minutes between each check in?? That is impressive!! Pat yourself on the back!! Salute
  • 260
 Felt so weird without the comments section. I seriously spent 70% less time on pinkbike
  • 10
 @sambobcat: it would make for an interesting analysis on traffic with and without user interaction. I betting page accesses were up, engagement cratered.
  • 150
 I realized that I don’t care as much as I think I do about mtb news. The real value (and time suck) is in the comments section!
  • 40
 @scoon: they’re going to expect this productivity moving forward…
You’re F-d
  • 1565
 My wife: "What's wrong? You keep checking your phone looking concerned"
Me: "nothing, it's ok"
My wife: "is there something you need to tell me?"
Me: "no, ahhh, no I'm fine. It's fine."
My wife: "is it someone else? Something's definitely wrong! I'm getting nervous!"
Me: "Pinkbike's broken"
Wife launches cushion at head. "You and bl00dy Pinkbike again!"
  • 822
 Give it to us straight. Yall were trying to migrate to the www.outsideonline.com/home interface and Levy ripped down to Pinkbike HQ screaming incomprehensible things about aliens and high pivots before pouring Faygo all over the office toaster oven he had long accused of calling the shots around Pinkbike HQ.

Of course it didn't do anything because everything is on the cloud somewhere, but it was clearly a sign, and reason enough to keep what's working going.
  • 560
 This is accurate.

Edit: except it was sugar free Monster not Faygo.
  • 80
 Here for conspiracy theories like this. More, please.
  • 302
 Mike Levy
  • 240
 No I have it on solid authority that they took the comments down for a few days in order to publish their best value bikes list with bikes costing over 4k and not having to deal with the pushback!
  • 60
 This is 100% what happened. I was half expecting some horrid slick 'outside' interface to appear on refresh hahah
  • 30
 @Mugen: you've got it backwards! comments == engagement, and emotional engagement is the stickiest engagement. Expensive "value bikes" provoke reactions and comment battles, that means more traffic and then PROFIT!
  • 20
 @plyawn: stickiest engagement? You shoulda seen the honeymoon, pilgrim.
  • 573
 Praise baby jebus. I didn't know what to think of the awards without comments. Please tell me which to be most mad about!
  • 31
 suspension product and then innovation
  • 88
 Thank you for censoring the name of Christ. Definitely saved some angry replies and aneurisms induced.
  • 460
 this has been the scariest 3 days of my life. waking up in the morning only to be able to read nsmb comment section be super polite and understanding. i need anger and unjustified hatred of new products in my mtb news community
  • 80
 Go to mtb news and run the comments through Ai to translate: more anger, more witchhunts less humor and sarcasm. Your welcome (maybe not on the site though).
  • 130
 @Vinnijussi: You will never find a more wretched hive of scum and villainy
  • 40
 As a confirmed green site user it has been upsetting to feel my sense of superiority diminish as I've not been able able to read the drivel that passes PB comments for 3 days. /s
  • 30
 You’re wrong and I totally disagree!

(Thanks. I needed that.)
  • 280
 The real disaster averted. The Grim Donut Game database was separate and not restored. But I got it back working from another backup!
www.pinkbike.com/sandbox/grimdonutgame
  • 31
 THIS IS THE WAY!
  • 50
 I forgot about this. That's my day gone.
  • 210
 Unsuccessful Levy coup. Fahk.
  • 150
 Pinkbike without comments shouldn't ever be a thing again.
  • 140
 How would a shadowy cabal of MTB brands even pay you; they're all bankrupt
  • 110
 Are we gonna talk about GT though?
  • 90
 Might as well say my BIKE ate my HOMEWORK!!!
  • 116
 As a database nerd I'm really curious about the root cause (I respect/understand if a post-mortem is never released).
If you guys are on AWS, RDS has automated blue/green deployments for major version upgrades (at least for MySQL/Postgres). Was it a major version upgrade that failed? Naughty dev updating/deleting data without a where clause? Migrating database systems? Corrupt backup (yikes)?
If it was indeed the database backend that failed and you guys need some help figuring out how to make stuff more highly available and redundant, happy to help. I've had a few long nights dealing with production outages :-D
  • 231
 Real men test in production
  • 52
 I don't think they're looking for a personel solution in the comments...
  • 30
 no doubt a lot of navel gazing in recent scrums
  • 180
 @seismicninja: Hey boss, there's a guy by the name of @turboshart who says he can solve all our DB issues! Smile
  • 21
 @Woody25: sandboxes are for kids
  • 90
 @Woody25: everyone has a test environment. Some people are lucky enough to have a production one as well.
  • 40
 Based on some WHOIS IP lookups, it looks like Pinkbike has registered their own IP block directly. That makes it hard to tell where the data is actually hosted. They also appear to leverage a global BGP network with SD-WAN capabilties. So I suspect that have multiple cache nodes that are spread out geographically. That's easiest to do with AWS. But AWS also has their own BGP and SD-WAN solutions, but PB is using a different global transit network that does not seem to partner with AWS. But I do still think its in AWS. And Trailforks is behind a CloudFlare proxy - so also hard to figure out the backend. As an IT guy, I feel for the guys behind the scenes that scrambled for days, probably without much sleep, to figure out how to recover from this. I bet they'll stop using Symantec Backup Exec to tape drives now!
  • 10
 @rallyimprezive: Last time I checked it was someone's garage or so. Radek didn't believe in cloud Smile
  • 20
 @rallyimprezive: I'm pretty sure I read a job ad where they run their own servers and infrastructure
  • 10
 @MrDuck: Elsewhere in these comments, someone who seems to have knowledge said "PB uses Outside Auth servers on a different AWS."
  • 90
 Please never let this happen again I was a mess
  • 90
 [object Object]
  • 70
 Pinkbike without comments was a wierd experience. Glad its back to normal!
  • 24
 I actually really liked it!
  • 60
 I think as compensation for the mental hardship endured by all PB commentors, we are entitled to Friday Fails EVERY Friday in 2025.
  • 30
 Was the site reverted to a back up just before the outage? I notice some comments were lost. Hopefully it was just comments and not necessarily all content after the nearest back up.
  • 120
 I made perhaps my best comment on the Canyon post, and when I went back to check the responses, it was gone, and the site was down. I just re-posted it, but now it’s too late to garner much attention.
  • 70
 @WeaselSqueezer: I won’t lie to you, WeaselSqueezer… I went to take a look because I thought your mentioning of a comment lost in the shuffle was ridiculous. But it was indeed an excellent reference.
  • 90
 @WeaselSqueezer: agreed, excellent post. Please accept props and a response here as a consolation prize.
  • 60
 "Looks like I picked the wrong week to quit drinking"
  • 50
 Don't look now, but Levy's on the Velo podcast talking about gravel something or other.
  • 30
 Shhhhh Smile
  • 51
 The NYSE / TSX markets will never recover from the damage done these last 3 days.
  • 40
 I recently learned how often I open an article on here and instinctively go down to the comments.
  • 51
 I have a feeling the “all mountain” riders had something to do with it…
  • 50
 Those of us in IT know this as a Resume Generating Event.
  • 40
 Oh so we're not boycotting the comments section to protest the lack of an advent calendar?
  • 20
 it was a strange 3 days. i get the news in my life not from pb posts, but from its comment section. it was great having a power outage and to have a break, truly a blessing in disguise...
  • 10
 I'm seeming ads on the Homepage even though I am a paid subscriber. Is this intentional now or do I have to use a different browser to block the ads instead?

Also, seems there's some sort of glitch with that Norco ad as it Autoplays at full volume each time I go to the homepage even though I mute it and Autoplay is off on my profile. It's irritating as it isn't exactly a quiet intro to the video...
  • 10
 I don't think anything has changed re ads, but the homepage autoplay should definitely be muted. I can't replicate your issue, can you DM me your browser/system info and I'll have someone do some digging.

Is anyone else having the video start playing with audio on?
  • 20
 @brianpark: done
  • 20
 Ahem, “Partner Content”, not ads.
  • 50
 I don't always test my changes, but when I do, I test them in production.
  • 40
 I've worked as a software developer for more that 20 years, never heard of anything like this, must have been a major fuckup
  • 40
 I couldn’t look at bikes I can’t afford!
  • 40
 so much grass was touched it will need counselling.
  • 30
 “Why won’t this column f’n drop???” Been there before on a hot rollback, gotta drop that constraint first Big Grin
  • 10
 It is ok to not have a comment section for a while, but I was hoping to read that Taj-illustrated anti-ride article I hadn't got to the first time around. It was gone as well. Glad it is back!
  • 10
 How would the plug from two days ago fix the Android app that has been broken for weeks? You are clearly spinning over the deepening incompetence of Pinkbike's technology team and decisions.
  • 30
 Longest 3 days of my life. I did manage to get a years worth of backlogged chores done during the outage though.
  • 20
 It would be interesting how page engagement differed (suffered) while the comments section was down.
  • 20
 I think @henryquinney extended his full compression settings to the PB internet connection..
  • 10
 Seem to have lost one of my rides on trailforks. Nothing in the app suggested it would just be gone and not to use the app. Did I miss something?
  • 30
 You might just have to wait for it to retry uploading from the app. You can goto in Settings -> Manual Sync -> Queue -> List and force sync it.
  • 20
 My wife regularly unplugs stuff rolling around in her wheelchair
  • 100
 If your wife ever leaves you, you can steal her wheelchair. I guarantee she will come crawling back.
  • 50
 @Ride406orDie:
Oh, you better believe I've tested that out
  • 20
 What's the difficult news from this week???
  • 90
 GT and RMB in challenging financial situations. Stories coming shortly.
  • 10
 GT temp shuttered (and RM filing for debt relief, but not outright bankruptcy)
  • 30
 @brianpark: got it. When PB goes down I don't have a clue WTF is going on. I wouldn't even know where to go for news, are libraries still a thing?
  • 20
 video or I call BULLSHIT.
  • 60
 Make it Autoplay
  • 30
 @manybrouce: unmuted
  • 10
 The big cleans is completed. All your shitty comments have been purged from the system.
  • 10
 I've been so confused the last few days without the Pinker Cabal telling me what to think.
  • 20
 shadowy cabal: you have my number


LoL
  • 20
 Pinkbike out, World Economy goes up!
  • 10
 Why is this story permanently in my feed? Comes back every time I delete it.
  • 10
 And that's the excuse you're going with...c'mon we need our MTB FIX!!!
  • 99
 This is the only ebike content that should ever on this site, being used as a joke. Because they're a joke.
  • 54
 Pinkbike without the comments…feature not a glitch.
  • 20
 I WANT MY 2 DOLLAR$!!!
  • 10
 Thou cannot silence the Pinkbike community!
  • 30
 Really?
Bugger.
  • 22
 And to be honest revert to 24 hours back up should not take that long while u figure out otherwise
  • 10
 Don't lie. You just wanted a break from the comment section....
  • 11
 Good way of losing traffic Curiosity doesn't kill people, stupid does mind.
  • 10
 I boosted a post and lost 2 days of boost on buysell
  • 10
 @brianpark why TF subscriptions being cancelled?
  • 10
 Better not hold my breathMike Levy
  • 10
 It was hugely improved without comments.
  • 10
 I took up Pickle ball for 3 dayz....
  • 10
 Awesome!
  • 10
 Here for the comments!
  • 21
 No comment.
  • 10
 dont abandon us again
  • 12
 PB might be back but a message from somebody inquiring about an item I had for sale has disappeared
  • 11
 I wonder if they scrubbed all comments etc.
  • 10
 AWESOME
  • 47
 whats up with the pb mod on reddit saying all our passwords were leaked?
  • 110
 oh no! Someone might steal my password and shit post!
  • 20
 Not AFAIK. Can you please DM me a link?
  • 120
 How do you know they're really a PB mod? Because they said so on Reddit? I hope you never get an email from a Nigerian prince.
  • 10
 @warmerdamj: they had a tag given by the r/mtb mods
  • 60
 @warmerdamj: nvm looks like they can self assign. Im calling BS on myself @brianpark. Sorry!
  • 130
 This was not a hack. And passwords are not stored on the PB servers for past 3 years. PB uses Outside Auth servers on a different AWS.
  • 20
 @canadaka: well then is must be true lol
  • 20
 @canadaka: *passwords* are stored? Or password hashes?
Below threshold threads are hidden







Copyright © 2000 - 2026. Pinkbike.com. All rights reserved.
dv56 0.043118
Mobile Version of Website