FPG Refuges Status Area: The 2020 FPN Diaspora

[aquafox]

Welp, took them long enough.
I honestly thought FPN was over, but I am glad they will be back.

[kaliuzhkin]

Don't count your chickens before they hatch. (or, Don't hatchet your Counts before they're chicken.)

FPN is not yet back.

[Ron Z]

Pessimism or wishful thinking?

[welch]

Closer. Wim & Company had to convert all the posts to the new format. That must have been a chore. Now working through the classifieds.

Update 2020-11-27 01:55 GMT / 2020-11-26 20:55 EST:
Configuration still going on, hopefully new Classifieds tomorrow (later today ), last website upgrade, and database upgrades to newer versions, still to be done, the latter once we are sure everything works just fine. We expect this to last another 1 to 1.5 days, so the end is in sight.

[welch]

A note from Wim. I cut the house=keeping talk about advertisements and membership.

Update 2020-11-30 02:45 GMT / 2020-11-29 21:45 EST:

Still some important and complex configuration going on, and fixing of problems encountered with certain image file groups.

Unfortunately it does take more time than we ever thought, but we are still making progress. Tomorrow morning (CET) we will try to make a new estimate for the ETA.

[Jon Szanto]

A note from Wim. I cut the house=keeping talk about advertisements and membership.

This is all so unfortunate and it seems that the biggest issue is that it is being handled by hobbyists. I can't imagine people who are involved in site design/maintenance would take a month to get something back online. Site management includes looking ahead to prepare for issues of scale, etc, and it seems like a project that grew and grew and got too big for the people involved to handle, but that was never a situation where skilled professionals were brought in to correct the issues.

This is merely based on observation of the situation, no back-channel knowledge. If any of the people involved actually are IT professionals, I really do question how it can have taken a month. That is eons in Net time. I wish them the best and hope it comes back soon, but I wonder if this won't happen again... because this is exactly how it has happened in the past.

[Ron Z]

Like I said before, I am amused at the judgements of those who have no idea what the back end of FPN is like, and how complicated it is. I've was watching discussions with moderators before the upgrade went to plan B. Wim works in IT. There are major changes in the hosting software that make a lot of what was used to make the board function rendered obsolete, some of which was no longer supported. All of those things have to be upgraded or replaced, and configured, while preserving the information it contains. Simply converting files takes time. For an idea of scale, IIRC Wim said that they get 5000 images a month in just the sales areas, not to mention all of the reviews and other images. Its not a commercial business or site - its one that was created for the benefit of the pen community. It costs money to run it, and money to pay people to do the tech stuff. The moderators are all volunteers. Maybe posting thanks for all the work when it is up and running instead of grousing would be worth considering.

[Jon Szanto]

Like I said before, I am amused at the judgements of those who have no idea what the back end of FPN is like, and how complicated it is. I've was watching discussions with moderators before the upgrade went to plan B. Wim works in IT. There are major changes in the hosting software that make a lot of what was used to make the board function rendered obsolete, some of which was no longer supported. All of those things have to be upgraded or replaced, and configured, while preserving the information it contains. Simply converting files takes time. For an idea of scale, IIRC Wim said that they get 5000 images a month in just the sales areas, not to mention all of the reviews and other images. Its not a commercial business or site - its one that was created for the benefit of the pen community. It costs money to run it, and money to pay people to do the tech stuff. The moderators are all volunteers. Maybe posting thanks for all the work when it is up and running instead of grousing would be worth considering.

Ron, I tried to couch my comments in as non-judgemental way as possible and indicated I don't know the background of the team involved. Since my joining FPN in 2011, every single time the site has gone down I have made certain to offer kudos to the team when it came back up. I am not dancing about this, but simply posting an open question about how they got to this point and what elements could cause such a long down-time. If I was one to laugh at the situation and make light of the amount of work involved, and if I somehow had it out for FPN in general, then I would not have been the one to start this very thread, welcoming those who needed a temporary place to hang their hat and talk pens.

I know this is all a frustration, and I am not meaning to add to that, merely trying to understand how this crash occurred. I appreciate your insight and I hope you understand that every questioning of the situation is not malicious in nature.

[silverlifter]

Like I said before, I am amused at the judgements of those who have no idea what the back end of FPN is like, and how complicated it is.

I doubt it is that complicated. There is nothing particularly complex about the site. The quantity of images is small, by modern standards. The admin may be an IT professional, but that could mean anything from a front end developer to a db admin (although, given the issues that they are reporting, that seems unlikely).

This looks to me, as someone who has worked in IT for decades now, like a poor choice of platform compounded by a shitload of technical debt and insufficient resources to fix the screw up.

I'm not blaming anyone, but I do find the ongoing commentary on the site about as credible and as amusing as "the dog ate my homework".

[pajaro]

Well, having had to upgrade systems to new software often made me write software to convert data, run a lot of conversion programs, and test and prove the process to skeptical people, so it could take a lot of time. Sometimes it is tricky, and sometimes it is scary. If you have to convert a lot of data, you have to plan it and then test and run it. Then you need to check it.

[welch]

I was a skilled enough IT guy that companies paid me to do it for about 35 years, going from 8080 assembly language programmer at a micro-computer startup through the rarified heights of "senior consultant" at the system by which banks move money across borders. When we did something like change from X.25 to TCP/IP, or add yet another security layer, we had a couple thousand engineers designing, testing, running "proof-of-concept" examples, maintaining fallback systems, and a few other things.

Best guess, and it's only a guess, FPN has run a series of server software, and has tried to carry over "legacy" features from one system to the next. That is a messy job.

As Ron Z. suggests, "Hats off!"

[silverlifter]

at the system by which banks move money across borders. When we did something like change from X.25 to TCP/IP, or add yet another security layer, we had a couple thousand engineers designing, testing, running "proof-of-concept" examples, maintaining fallback systems, and a few other things.

Yeah, but I doubt that FPN is written in COBOL...

[TSherbs]

I love when you guys computer talk

[Jon Szanto]

Yeah, but I doubt that FPN is written in COBOL...

BTW: while my first degree was in Music Performance, I later got a second degree in Software Engineering and spent a few years in that field concurrently. I've gotten my hands dirty on projects like these...

[azkid]

This looks to me, as someone who has worked in IT for decades now, like a poor choice of platform compounded by a shitload of technical debt and insufficient resources to fix the screw up.

That's exactly how this reads to me. The situation reminds me of the nightmare we had to deal with upgrading our SaaS product at the dot com I worked at in the early 2000s.

We had similar issues in the moderately large company I went to after that. Our IT systems were a decade or more old and just a mess. We actually started a major enterprise re-architecture to address the kludge-fest it had become but then we got sold off to a company with IT practices of 1980s BBS run by a teen. Anyhoo...

There's no blame. It just is. When you aren't a corporation paying the salaries of a team of full time, skilled IT professionals, but instead have a too-small team of volunteers versus the scale of the site, there's no time for planning, you put off upgrades, and eventually the bill comes due.

In the professional world, you'd have system admins, dbas, app admins, and tech support on a system like this, maybe even architects, capacity planners, performance engineers, product owners, project managers, or various others to guide the operations and plan for upgrades and maintenance.

You'd have monitoring software and notifications of performance problems, downtime, etc., with on call response to deal with it. If it were critical, the system would be running in some high availability architecture.

That all takes gobs of money and time. And a lot of expertise.

Anyway, I feel bad for em. I've been involved with some epic downtime slogs but nothing like what they're going through. It sounds like an absolutely demoralizing nightmare to me. So kudos to them for the ton of hard work.

[Ron Z]

BTW: while my first degree was in Music Performance

You too? I went into radio with the second degree. People would ask me "OH, what instrument do you play?" I'd answer, "Vox humanas." ...which should have been humanae vocis.

[welch]

at the system by which banks move money across borders. When we did something like change from X.25 to TCP/IP, or add yet another security layer, we had a couple thousand engineers designing, testing, running "proof-of-concept" examples, maintaining fallback systems, and a few other things.

Yeah, but I doubt that FPN is written in COBOL...

Ha! And neither is our system that keeps money in your account! (But none of us are allowed to tell you the languages, the operating systems, or even the name of the company!) I did, however, write some programs in COBOL on punch cards way, way back. Got a job writing to 8-bit micros and never did see a bit of COBOL.

[Jon Szanto]

I don't think it's reasonable to assume by default...

You are welcome to have that viewpoint.

[silverlifter]

at the system by which banks move money across borders. When we did something like change from X.25 to TCP/IP, or add yet another security layer, we had a couple thousand engineers designing, testing, running "proof-of-concept" examples, maintaining fallback systems, and a few other things.

Yeah, but I doubt that FPN is written in COBOL...

Ha! And neither is our system that keeps money in your account! (But none of us are allowed to tell you the languages, the operating systems, or even the name of the company!) I did, however, write some programs in COBOL on punch cards way, way back. Got a job writing to 8-bit micros and never did see a bit of COBOL.

FWIW, one of the universities here is now offering COBOL as part of its CSI degree, because the banks are still running those legacy systems and the engineers who maintain them are either retiring or punching out. Those few that are left are pulling salaries that make their bosses wince...

[welch]

Silverlifter wrote:

FWIW, one of the universities here is now offering COBOL as part of its CSI degree, because the banks are still running those legacy systems and the engineers who maintain them are either retiring or punching out. Those few that are left are pulling salaries that make their bosses wince...

Wow. And "wow" again. I remember the great "Y2K" scramble, when companies hunted up retired COBOL guys (trained when nearly all programmers were men) and threw them at all the mainframe - COBOL systems. So it's happening again?