Workunits downloading, with usual re-tries.
Website down with Error 500: (was up for a while last night).

Paul.

20.5 hours later, and the site is still down. Not a peep from the team on Facebook or Twitter.
And of course, the BOINC part is slow on downloads, and full of HTTP errors as usual.

But, in 11 days, that doesn't matter for me at least. Nov 1st will my new electricity contract begin,
and with up to 5 times as high cost per kWh as now, crunching will be the last thing on my mind.

Finally, after almost 23 hours downtime, they posted the following on Facebook:

"We are currently suffering a communication issue between the website and our database,
resulting in an inability to log in properly (or access the website in some browsers).
We apologize and will notify you when it has been resolved."

Well, the site came back for a few hours, but again it went down after the staff abandoned their workplace, and went home.
No reaction on Facebook, which clearly shows that they do not have anyone checking the projects 24/7, or use some
automatic means to get a message when things go wrong.

Not good for such a big project as WCG. If things doesn't improve soon, WCG will not continue to be a big project though...

And there's no sign of any new work today, either - except the occasional one or two stragglers limping in.

I hope Krembil are aware - but I fear that they are not - that for a successful scientific research project under BOINC, all the moving parts have to be working at the same time. Work has to be generated, allocated, downloaded, processed, uploaded, validated, and assimilated. The entire pipeline moves at the speed of the slowest. And if one piece breaks, they're all broken.

I did get 19 new (_0) OPNG tasks from batch 0160806 at 09:45 UTC+2, but they are crunched, and gone. The last two uploaded and reported at 10:39 UTC+2.
Since then, nothing, nada, not even any resends.

Krembil/Jurisica really have to get their act together, or just throw in the towel, and admit that they're not up to running WCG.

I just got some more _0 OPNG with no retries, 7 of 8 slots filled.

Paul.

After my 19 OPNG's at 09:45 UTC+2, I've only received 2 (_2) OPNG resends (not asking for CPU tasks)

Website still down. Apart from fixing their inability to tame the IBM WCG system, they really need to work
on their communication skills. Not a peep from them on their social media sites, regarding this latest kaboom.

Website still down. Apart from fixing their inability to tame the IBM WCG system, they really need to work
on their communication skills. Not a peep from them on their social media sites, regarding this latest kaboom.

No one in a large organization communicates bad results, only good ones.

New post from WCG on Facebook:

Access to the WCG website is restricted for a short time while we work to expand the storage capacity of the WCG. This should not prevent workunits from being sent out.
This will help identify the underlying system issues and crashes that we have been having trouble with for the past few weeks.

And the website is up again. However I doubt it will stay up for many hours.
No tasks (OPNG) for many hours now.

Poof, and website down again. Rinse and repeat.....
It's quite obvious by now, that they haven't got a clue about why this happens time after time.

Geeze, that was surprising. Someone is actually working in the WCG team on a Saturday.
The website is back up again. But of course, I do not expect it to last for long.

Website down, scheduler still responding.

Paul.

Updated "news" from WCG
2022-10-27 Update (Workunits & storage update)

Hi everyone, we’re happy to see that volunteers are receiving more OPN1 workunits than last week. We recently increased our DB2 storage pool and switched to a more coarse-grained scheduling method for creating and packaging new workunits for each project. This change may have temporarily disrupted WU scheduling, but we will need to monitor further and likely explore additional possible causes before we can consider the issue resolved.

Another (less optimistic) theory is that other tasks, specifically OPNG, were the cause of our recent storage issues and database-wide system errors. We have no solid evidence yet, only an observation that there is typically a decline in available OPNG work around the same time the download issues are less prevalent. A high load on the storage server and scheduler coincide with the database crashes and a phenomenon whereby the download/upload server groups intermittently register as down from the perspective of our load balancer.

We continue to monitor the system to determine what the best course of action is to stabilize our internal network.

Thank you for your support, patience and understanding.

WCG team at Krembil Research Institute

[Oct 27, 2022 8:39:27 AM]

And the WCG website just crashed again.

Scheduler still respomding, so far.

Paul.

Website back up but still no ARP work.

And down again.

And the download servers are playing hard to get.