Website down, scheduler still OK, no retries on recent downloads.

Paul.

And still down, 6 hours after your post :-(

Website up, scheduler responding OK.

Paul.

Sigh.....

503 Service Unavailable
No server is available to handle this request

Edit, and shortly thereafter:

System error
World Community Grid is currently experiencing an unexpected error. Please check Facebook or Twitter for more information.

It's no longer an "unexpected error". I suggest they change the message to "World Community Grid is currently experiencing an expected error.

Incredible that they can't get it to work longer than max 1-day, and only if they restrict how much work they send out.

And down again of course.....

More propaganda from Krembil:

WCG’s 18th Anniversary

Celebrating WCG’s achievements over the past 18 years.

Best part of propaganda [highlighted in bold by me] >>

The WCG is now managed by the Jurisica lab at UHN. As an academic group, we face a significant financial challenge in providing the same level of support to the global research community, still free of charge. One year may seem a long time for “WCG system migration”, but our small team had to transition from a research team that ran Mapping Cancer Markers on WCG to learning, building and configuring the WCG system on our servers, with our architecture and performance constraints (s the servers were not optimally configured for the WCG system)

And again.....Rinse and repeat......

And more rinse and repeat.

New post / WCG news from Krembil:
Thread: 2022-11-18 Update (Network and Storage)

Hi everyone, an update on network connection and storage.

We are working together with SHARCNET (an HPC site where WCG servers and storage reside) to resolve the network congestion events we have been experiencing. For volunteers, these events manifest as the arbitrary website/forums database downtime and constant interruptions to volunteers attempting to download workunits. At this time, we believe the root cause to be a limitation or bug in the OpenStack software through which our virtual environment is provisioned at SHARCNET.

To help ameliorate the worst effects of this issue, SHARCNET have re-routed all WCG traffic through a new network node. Effectively, this separates WCG traffic from that of other users and deployments unrelated to the WCG that are colocated at the SHARCNET HPC facility. We have already seen a benefit from this change, and it could help us to further diagnose and optimize additional performance issues.

We have also reduced the maximum concurrent connections permitted on the download servers at SHARCNET’s request, and reduced the maximum number of packages available at any one time for download. Although these adjustments suggest a lower throughput, they have been active since November 11 and are in fact helping the overall throughput of WCG by stabilizing the network to a degree. However, we are still seeing events inside our environment where the load balancer and servers behind it are sometimes unable to communicate with each other.

Importantly, the bandwidth that the WCG environment is provided with at SHARCNET is nowhere near saturated during these events. It is not an issue of capacity. We are working to resolve this and will provide an update on our progress as soon as we have new information. Once resolved, we will be in a position to fully restart, and bring new projects to the Grid.

The new and faster storage server is physically installed at SHARCNET now and will be connected to the rest of the WCG servers next week. The primary benefit of the new storage array is the SSD storage that comes with it, which will increase performance of many key components that currently rely on NFS shares of logical volumes that are composed of HDD storage only.

If you have any comments or questions, please leave them in this thread for us to answer. Thank you for your support, patience and understanding.

WCG team at Krembil Research Institute

And time again for another "System error"

Geeze, this is really getting old.....
503 Service Unavailable

And as usual followed by:
System error
World Community Grid is currently experiencing an unexpected error. Please check Facebook or Twitter for more information.

Of course there is no information about this on Facebook.....

Down, 503 etc.

Paul.

Web and the BOINC part of WCG, both down this time. So, this time the entire WCG project is DEAD.
The web part does not respond at all, and the BOINC part responds with

6372	World Community Grid	2022-12-03 03:27:02	update requested by user	
6373	World Community Grid	2022-12-03 03:27:04	Sending scheduler request: Requested by user.	
6374	World Community Grid	2022-12-03 03:27:04	Requesting new tasks for CPU and NVIDIA GPU	
6375	World Community Grid	2022-12-03 03:27:27	Scheduler request failed: Couldn't connect to server	
6376			        2022-12-03 03:27:28	Project communication failed: attempting access to reference site	
6377			        2022-12-03 03:27:29	Internet access OK - project servers may be temporarily down.	

EDIT: Seems as if it's WCG's IT-provider sharcnet.ca that is totally down.

"SHARCNET is a consortium of universities in Ontario, Canada, that aggregate funding to purchase supercomputer systems,
which are shared among the members to perform research, rather than individually purchasing smaller systems at each university.
It was formed to allow members access to larger, faster, and more modern computer resources than they would otherwise be able
to afford, and to retain researchers at their organizations. SHARCNET is part of the larger Compute Canada umbrella."

Geeze, WCG via sharcnet came back for short while, but now the WCG site is gone again with the usual:
System error
World Community Grid is currently experiencing an unexpected error. Please check Facebook or Twitter for more information.

BOINC, and Sharcnet is working though, so this time it seems to be the "normal" WGG problem.

News from WCG

2022-12-08 Update (OPNG workunits, storage update, and missing devices on Contributions page)
OPNG workunits
Over the past 2 weeks many volunteers noticed the lack of OPNG workunits. The shortage is not a technical issue, but rather it has a “science-based” reason. Here is a short message the OPN team shared with us:

“The scientists are analyzing the available data to choose a set of protein conformations for the next rounds of molecular docking in GPUs. The focus is on PLPro, one of the SARS-CoV-2 proteases. The dockings will include covalent and non-covalent compounds.”

When new OPNG tasks will be prepared, we will make a small announcement informing everyone.

SSD delay
Unfortunately, the new storage upgrade that we mentioned in this previous update has been delayed by a late shipment of the SSDs, with no guaranteed ETA for implementation after arrival at this time. We are in constant communication with the SHARCNET team as to the status of the new drives, and will update you when significant progress has been made.

Missing devices in Contributions page
Several volunteers experienced missing devices and results in the Contributions page. This issue has been problematic for some time and we are now putting more effort into solving it. For the time being, we have scheduled a batch update from the BOINC database directly, bypassing the message passing via IBM MQ that notifies the website database when new devices have been inserted into the BOINC database. Insertion into the BOINC database occurs at time of first contact with our scheduler. The new record is then placed on a service-to-service queue, to which the webserver and website database server are subscribed. When processed, this message should result in visibility of the device in both databases and to users on the website immediately.

The fact that only some devices do not show up as a result of this process and remain “hidden” is what we must understand before we can fully resolve the issue. For now, re-running the batch update procedure regularly will help to recover visibility of most devices newly added to WCG. We highly encourage those having this issue to send an email to [email protected] with screenshots and extra details you can provide.

If you have any comments or questions, please leave them in this thread for us to answer. Thank you for your support, patience and understanding.

WCG team at Krembil Research Institute

And ARP tasks just back after an absence. Downloads are so dodgy it makes my bored band upload limit of 100KB/s seem fast!

My experience is virtually the same today.

One ARP data file been in constant backoffs/stuck download for almost 9 hours so far. Don't matter how many times mash the retry, it ends with backoff.

My experience is virtually the same today.

One ARP data file been in constant backoffs/stuck download for almost 9 hours so far. Don't matter how many times mash the retry, it ends with backoff.

7 running, 7 inqueue and one uploading at the moment. I am trying to manage it so only one task is downloading at a time so that downloads happening from lots of tasks at once don't mean that much longer till all files from the first task finish downloading.

Anyway, a few more thousand CPDN tasks coming next week so ARP will go to back burner.

The sad thing is that I don’t take ARP but my WCG is still screwed because of the download problem - one held file overnight and the whole of WCG stops until the blockage is cleared.

It's almost 10 months now, since the migration from IBM to Krembil/Jurisica begun, and we're not even close to anything that even resembles
how WCG worked before the migration. Nothing has gone according to plans....

WCG Data Transfer Underway, Stress Test of New Infrastructure Scheduled For Feb 28th

lol.... Another day, more "stuck" downloads for ARP tasks