I've currently running BOINC 6.12.34. I am subscribed to 4 projects.

I've recently been watching how BOINC manages tasks. I've read the discussion of project debt and scheduling for this version and I'm still puzzled by how it works.

All of the discussion of task scheduling talk about the normal mode being to use debt to select a project to run. Nothing talks about how you select tasks once a project is selected. It seems obvious that once you select a project you should run the available task with the earliest deadline, but that's not the way it works.

First, I currently have 22 work units for Cosmology@Home. More than half (13) are partially completed. The last time BOINC switch into the project, instead of picking up one of the partially finished units, it started a new one that wasn't the earliest deadline.

Can someone please explain how tasks are selected to run withing a project?

Thanks

---

I'll answer that with a quote of how John McLeod VII over at Seti explained how BOINC follows the FIFO track at all times that it can:

The BOINC CPU scheduler can be in either of two modes at any given time for each processor independently (OK, this slide over multi CPU tasks a bit, it is covered in the code though).

At each task start or task switch opportunity the CPU scheduler estimates the completion times of all tasks in the system, and compares that to 90% of the time from now to the computation completion deadline (which is earlier than the report deadline that shows in the UI - more on that in a bit). If the estimated completion time is >= 90% of the computation deadline, then that task is marked as requiring high priority.

For each high priority task in deadline order from earliest to latest, assign the task to a CPU (or GPU as the case may be) until either the supply of CPUs (or GPUs) is exhausted or all of the high priority tasks are assigned to a CPU (or GPU).

For the remaining CPUs and GPUs, assign tasks in Round Robin order by project and FIFO within each project.

The two different operational modes do several things:

1) Meet all deadlines if it is at all possible to do so.
2) Ensures that long running tasks with distant deadlines are not starved.
3) Cycles through the different projects in what is hopefully an interesting order.

The computation deadline is the report deadline - (task switch interval + Connect Every X days + possibly 1 day (version dependent, removed in later versions)). This ensures that the task should be completed early enough to be updated and reported during a connection that falls between the completion of the task and the report date.

The estimation takes into account factors such as fraction of the time BOINC gets to do computation, and resource shares of all projects with tasks on the computer among other things. Which leads to the non-obvious outcome that low resource projects that actually get to download work typically get a lot of high priority CPU time to complete by deadline.

---

My problem is that in this case it isn't completing all the work units.

My machine was shut off for several days. As a result in Cosmology (long work units all about the same length) ordered by deadline I have 7 work units, then a several day gap, then more work units. It appears that the early wus will not finish by deadline.

At this point, I have Boinc running 4 high priority wus from Cosmology, but these are NOT 4 units from the early group. BOINC has run and completed units from the late group and even downloaded additional units. But seemingly will not run any of the early units.

I've finally shut off downloading, more units have finished, but not from the early group.

I'd offer to volunteer to work on the scheduler, but I understand it is rewritten in the next version.

---

But with 13 tasks from 1 project started and the scheduler then gives the CPU to a task that hasn't been started then it's clear the scheduler is not following the FIFO rule. If 13 tasks are already started then, according to John, they must have started because they were in first. If they were in first then why doesn't the scheduler give the CPU back to one of them? Instead it starts a task that hasn't started because, according to John, it came in later.

I am not deep into this issue but have recognized some seemingly weird behavior of the scheduler myself. As far as I have come it is not just the FIFO rule, but instead the scheduler takes the (estimated) time needed to complete a task and compares it to other tasks (of the same projekt of course). Therefore if a partially completed task does only need "little" time to complete and a new task would get close to its deadline or even over it the new task gets CPU time so all tasks can complete within their deadlines.

Well, that is my theory, at last..... but I think it is reasonable and would be practicable, too.

And anyway, it seems to work so far.

My problem is that in this case it isn't completing all the work units.

My machine was shut off for several days. As a result in Cosmology (long work units all about the same length) ordered by deadline I have 7 work units, then a several day gap, then more work units. It appears that the early wus will not finish by deadline.

At this point, I have Boinc running 4 high priority wus from Cosmology, but these are NOT 4 units from the early group. BOINC has run and completed units from the late group and even downloaded additional units. But seemingly will not run any of the early units.

I've finally shut off downloading, more units have finished, but not from the early group.

That would be weird indeed. Please let it run and complete and tell us how it finished. I would be really interested, maybe it offers some insight into the schedulers' work.

3) Cycles through the different projects in what is hopefully an interesting order.

May I hear more about 'interesting order', please?

---

I'm counting for science,
points just make me sick.

3) Cycles through the different projects in what is hopefully an interesting order.

May I hear more about 'interesting order', please?

The interesting order depends on work availability at the attached projects,

Claggy

The interesting order depends on work availability at the attached projects,

Does it mean that client has some paranormal wisdom about WU availabitility in future? Or not? I'm confused.

And a bit of observation on FIFO. Before PG challenges I nomorework every project attached except SZTAKI that has four week deadlines. At a challenge start I suspend it and WUs are finished eventually after challenge end. This time, three days before the Equinox SZTAKI stops distributing work, so I've resumed ABC in order to keep cores busy. ABC has two week deadlines, WOO-RLLs are quite time consuming, thus I decided that as soon as I see that started WU doesn't finish before the challenge end I can safely resume ABC.

So, within 48 hours before bottom deadline a client has three cores and three days of work. What could possibly go wrong? The client starts to empty the queue from the end. That was scary. Within eight hours before the top deadline three hours of work weren't even touched. Those three hours were finished and uploaded, however they weren't reported. I had to manualy report them two minutes after deadline. Hopefully they weren't sent to someone in that time.

Shortly, I fail to see any FIFO.

---

I'm counting for science,
points just make me sick.

Those three hours were finished and uploaded, however they weren't reported. I had to manualy report them two minutes after deadline. Hopefully they weren't sent to someone in that time.

That sucks. Use the report_tasks_immediately option in a cc_config.xml file.

That is a good issue to speak of - I never understood what the reporting feature does. Maybe someone can explain this to me, because I always took it that when results are uploaded the project is notified (shouldn't that be always in this way?) automatically....

So what's the difference anyway? Why need to report separately? Who goes this report to?


And moreover, why is there even a report_tasks_immediately option for this anyway, this should be done always and immediately by default (aka hardcoded), right?

Thanks guys.....

Uploading is just moving data from your hard drive to a hard drive on the server.
Reporting accesses the database, whereby it must be noted that when you report one task that it takes as much overhead on the database server as when you report ten tasks or one hundred tasks. So projects and BOINC prefer it if you report tasks in bunches, to keep things nice and tidy.

The RRI option is there to allow reports to be done immediately on projects like WCG which make daily backups of their data. That way you have a good chance that your reported tasks were in before the backup.

Other projects don't like RRI that much. Einstein and CPDN have been reported as disliking RRI, whereby you can lose data.

So use at your own discretion.

---

"Daily Backups"? Maybe a blond moment, No, the RRI was [reluctantly activated again] on request of WCG for large contributors who re-image daily [DeepFreeze], regardless if work is finished or not... call it a Daily Restore [to virgin set-up state] ;-).

Uploading is just moving data from your hard drive to a hard drive on the server.
Reporting accesses the database, whereby it must be noted that when you report one task that it takes as much overhead on the database server as when you report ten tasks or one hundred tasks. So projects and BOINC prefer it if you report tasks in bunches, to keep things nice and tidy.

The RRI option is there to allow reports to be done immediately on projects like WCG which make daily backups of their data. That way you have a good chance that your reported tasks were in before the backup.

Other projects don't like RRI that much. Einstein and CPDN have been reported as disliking RRI, whereby you can lose data.

So use at your own discretion.

Thanks Ageless, that really cleared things up.
:)

Those three hours were finished and uploaded, however they weren't reported. I had to manualy report them two minutes after deadline. Hopefully they weren't sent to someone in that time.

I think you are on the safe side - AFAIK most projects do not take their deathlines strictly but give a bit time to it for exactly those cases.

Sory, I forgot to mention that in my earlier post.

Those three hours were finished and uploaded, however they weren't reported. I had to manualy report them two minutes after deadline. Hopefully they weren't sent to someone in that time.

I think you are on the safe side - AFAIK most projects do not take their deathlines strictly but give a bit time to it for exactly those cases.

Sory, I forgot to mention that in my earlier post.

More accurately, projects do set a replacement task immediately ready in the distributor, WCG at least, and if the No Reply still shows up, a message is send to the repair men not to start the task i.e. abort. If the No Reply shows up before someone fetches the repair task, it will be changed to 'Other' status and withdrawn. Some members will see that and come to the forums to ask what was that about :D

It's rather odd that the client did not report immediately of it's own volitions with that deadline exceed. Maybe that could get incorporated into the coming client if not already.

--//--