No further news from the WCG team. Still down, very very deep down......
Since it's almost 23:00 hours, in Toronto now, WCG is probably going to stay down, at least over the (Canadian) night.

No further news from the WCG team. Still down, very very deep down......
Since it's almost 23:00 hours, in Toronto now, WCG is probably going to stay down, at least over the (Canadian) night.

The lack of news (and progress) is somewhat concerning.
Also, why would the forums be down at the same time?
I thought they were hosted separately.

---

The lack of news (and progress) is somewhat concerning.
Also, why would the forums be down at the same time?
I thought they were hosted separately.

It was when IBM ran the show. Don't know how much physical hardware Krembil consolidated project on, but my guess is it's a lot less than IBM.

Being 30+ hours since last Facebook update/post from Krembil, who have not been good at communicating things since they took over 18 months ago... We don't have a clue what's happening.

To: Grumpy Swede
Thank you for posting updates on WCG here. I don't use Facebook or Twitter so, with the forums down as well, this thread has become my source of information.
Cheers
Phillip

To: Grumpy Swede
Thank you for posting updates on WCG here. I don't use Facebook or Twitter so, with the forums down as well, this thread has become my source of information.
Cheers
Phillip

No problems Phillip. I do not "use" Facebook or Twitter either (I have no accounts), but one can always read the postings on most public Facebook and Twitter accounts.
Sadly though, it has not been any more updates on the WCG FB account. They really need to improve their communication skills. This silence towards us volunteers,
are simply not acceptable.

Just a few words would make a big difference. Like "We're having big problems with the RAID array, and it will take about xx hours/days to rebuild the array", or
"The entire RAID array is toast, and we have to rebuild it from backups, and it will take about xx hours/days" (Let's hope their backups has been handled in a professional way.)

To have this kind of SPOF (Single Point Of Failure), in such a big system as WCG, is just ridiculously unprofessional. When/if, they recover from this, the first thing they need to do,
is hire a real IT-professional, to look through their entire setup, and identify other weaknesses, and other SPOF's.

Just in, New Update:

"Update #2: The borrowed RAID card worked and the drive layout was recognized, so we have all data safe
(there is also a tape backup, but accessing that would be slower). Data center managed a full boot and we
expect we will resume operation later today."

Edit, added: the last part "we expect we will resume operation later today", is something I would take with more than one grain of salt

To have this kind of SPOF (Single Point Of Failure), in such a big system as WCG, is just ridiculously unprofessional. When/if, they recover from this, the first thing they need to do,
is hire a real IT-professional, to look through their entire setup, and identify other weaknesses, and other SPOF's.

I suspect the real issue is the amount of cash going in is a fraction of what there was when IBM had it. If they had more cash, I am sure their infrastructure would have more redundancy built in.

New Update, about 4-hours ago:

"Update #3: All disks have finished rebuilding and the MySQL and DB2 filesystems are mounted.
We are currently checking the /science filesystem integrity."

WCG is still down though......

New update, just in:

"Update #4: We have confirmed all the data is intact and have replaced the RAID controller,
but we are still having some issues with getting the new hardware production ready.
Unfortunately, data center staff will not be able to help us over the weekend.

Additionally, the deadline of all existing WUs that are partially done will be extended and accepted once the hardware change is done."

So, as I expected, WCG will likely continue to be down, until at least some time next week.
(I wouldn't be surprise though, if WCG was down for another week, or two)

A slight difference now on the WCG home page. You no longer get the normal homepage, but instead directly the Errror
"System error World Community Grid is currently experiencing an unexpected error. Please check Facebook or Twitter for more information."

Also the forum ink (or other links needing access to a database) , does no longer show "Error 500: javax.servlet.ServletException: net.myvietnam.mvncore.exception.DatabaseException: Error executing SQL in MVNForumPermissionWebHelper.getPermissionsForGroupGuest."
The forum link now also shows: "System error World Community Grid is currently experiencing an unexpected error. Please check Facebook or Twitter for more information."

So, somebody is possibly up early in Toronto, trying to perhaps restart the system, and restore connections to the various databases, and filesystems.

Edit, added: No life from BOINC yet though. BOINC is still replying to a request with "Server error: feeder not running"

Thx for info ; )

And it's finally Monday. Let's see if anything at all happens with WCG today.
It's only 05:33 in Toronto yet, so maybe something happens in a couple of hours.

And it's finally Monday. Let's see if anything at all happens with WCG today.
It's only 05:33 in Toronto yet, so maybe something happens in a couple of hours.

More in hope than certainty! Do you have pigs that fly in Sweden? I just saw several Porcine Flying Objects overhead in South West France and distinctly heard them squealing "(B)oink, (B)oink"
Fingers crossed!
Phillip

More in hope than certainty! Do you have pigs that fly in Sweden? I just saw several Porcine Flying Objects overhead in South West France and distinctly heard them squealing "(B)oink, (B)oink"
Fingers crossed!
Phillip

I don't know the velocity of flying pigs, they may take a while to reach Toronto.

More in hope than certainty! Do you have pigs that fly in Sweden? I just saw several Porcine Flying Objects overhead in South West France and distinctly heard them squealing "(B)oink, (B)oink"
Fingers crossed!
Phillip

Well, we do have some flying pigs here in Sweden, most of them are politicians :-)
Yeah, fingers crossed for WCG returning before the next ice age.

Only a matter of hours until the next ice age in the north of the UK......

I'm very upset over the current state of affairs. I've been crunching files since the start of WCG. I'm now doing Rosetta files hoping WCG will come back soon. Glad to have found this forum since FB and the WCG forum are down.

New Update from WCG on Facebook, a couple of minutes ago:

"Hello everyone, hope you had a great weekend. We are still working with data centre to resolve the hardware failure so we can
restart the storage, BOINC and website ASAP. We will post updates as we receive them. Thank you for your patience."

... Another day wasted because Krembil's incompetence and running around chasing it's tail :(

It's soon 14:00 hours in Toronto, and I seriously doubt that we'll see any working WCG today either.
Sigh......