EDIT:: (I was looking at my screen after closing a web session and caught a MOO work unit get a computation error.
"Output File dnetc_r72_1323592031_308_768_4_0 for task dnetc_r72_1323592031_308_768_4 Exceeds Size Limit
FileSize: 54240.000000 Bytes. Limit: 51200.000000 Bytes".
So this could be the problem, it shows as a Failed Upload and File Transfer Error when reported)

Conan.

Another thing is a packet limit error. Here's the end of one of my WU stderrs:

[Dec 14 08:03:11 UTC] RC5-72: 1 packet (2.00 stats units) remains in in.r72
Projected ideal time to completion: 0.00:00:04.00
[Dec 14 08:03:11 UTC] RC5-72: 387 packets (764.00 stats units) are in
out.r72
[Dec 14 08:03:20 UTC] RC5-72: Completed CA:5BBA5EB5:00000000 (3.00 stats units)
0.00:00:08.45 - [1,523,938,721 keys/s]
[Dec 14 08:03:20 UTC] RC5-72: Loaded CA:5BBA5ECC:00000000:2*2^32
[Dec 14 08:03:20 UTC] RC5-72: Summary: 388 packets (767.00 stats units)
0.00:35:59.60 - [1,525.39 Mkeys/s]
[Dec 14 08:03:20 UTC] RC5-72: 0 packets remain in in.r72
[Dec 14 08:03:20 UTC] RC5-72: 388 packets (767.00 stats units) are in
out.r72
[Dec 14 08:03:27 UTC] RC5-72: Completed CA:5BBA5ECC:00000000 (2.00 stats units)
0.00:00:05.66 - [1,516,852,305 keys/s]
[Dec 14 08:03:27 UTC] Shutdown - packet limit exceeded.
[Dec 14 08:03:27 UTC] RC5-72: Summary: 389 packets (769.00 stats units)
0.00:36:05.26 - [1,525.37 Mkeys/s]
[Dec 14 08:03:27 UTC] RC5-72: 0 packets remain in in.r72
[Dec 14 08:03:27 UTC] RC5-72: 389 packets (769.00 stats units) are in
out.r72
[Dec 14 08:03:27 UTC] *Break* Shutting down...
[Dec 14 08:03:27 UTC] Shutdown complete.
00:03:27 (1444): called boinc_finish

</stderr_txt>
<message>
upload failure: <file_xfer_error>
<file_name>dnetc_r72_1323738425_389_769_1_0</file_name>
<error_code>-131</error_code>
</file_xfer_error>

It happens at varying lengths so it must have something to do with the file size as well as it looks like its adding extra computation results to the out file from ghost packets.
I've been lucky so far. Only 3 bad WUs in last 2 days (all on 1 box). Another thing also is the in and out files in the slot folders are always garbled (or encrypted?) chars but the checkpoint file is that way only sometimes while clear text on others. This is on 3 different boxes, 1 with 4850 and 2 with 5830s. Any thoughts?

NeoMetal

I'm now playing with a manual fix for that issue:

1. Request a cache of 1 day of Moo WUs.
2. Shutdown BOINC after all WUs have been downloaded.
3. Manually edit the file 'client_state.xml' and replace the line

<max_nbytes>51200.000000</max_nbytes>

by

<max_nbytes>61200.000000</max_nbytes>

for every Moo! <file_info> section in that file.
4. Restart BOINC.

I'm quite confident that this will fix the issue but of course only time will tell if it really does...

I'm now playing with a manual fix for that issue:

1. Request a cache of 1 day of Moo WUs.
2. Shutdown BOINC after all WUs have been downloaded.
3. Manually edit the file 'client_state.xml' and replace the line

<max_nbytes>51200.000000</max_nbytes>

by

<max_nbytes>61200.000000</max_nbytes>

for every Moo! <file_info> section in that file.
4. Restart BOINC.

I'm quite confident that this will fix the issue but of course only time will tell if it really does...

lol ... I'll Detach and go to another Project before going thru all that work on 4 Different Box's ... I can live with the 10-15 Percent error rate for a few more day's, but at the rate it's going that's going to go up to 25% or more so it'll be time to move on until the problems fixed ...

---

Actually, it looks like Teemu is already working on a solution as I'm not getting any new GPU work:

15.12.2011 01:25:05 | Moo! Wrapper | Requesting new tasks for ATI GPU
15.12.2011 01:25:07 | Moo! Wrapper | Scheduler request completed: got 0 new tasks
15.12.2011 01:25:07 | Moo! Wrapper | No tasks sent

Server Status page is showing plenty of available tasks but they seem to be CPU only...

Hi,

Thanks for PM me and sorry that it took this long to notice but I'm working on this problem now. This seems to also affect out throughput. :(

This seems to be related to the upstream resending old work which leads to high fragmentation in the work they send to us which in turn increases the packets in our work units.

The error -131 means the output file is bigger than what we accept. Since more packets mean bigger files, that'll explain why we've now hit this limit. I've increased this limit for new work we generate and next I'll see what I can do with the existing work that keeps failing. (These will eventually fail permanently but that can take some time so I'd rather get them fixed or aborted centrally.)

-w

Across 8 rigs I only have a total of 6 error -131's.

HD5850 - 3
HD4870 - 2
HD4770 - 1

So either I am very lucky or it's about to hit me ....

---

I iz also got icons!

Hi hi,

Looks like these big packet counts also affected our work generation (or there was some other problem) and I had to fix that as well. We seem to be running out of huge work units at the moment but it should eventually catch up, I hope. I'll keep an eye on that.

The -131 error codes should now also be fixed for any existing work units as well. At least as much they can be. Until all the faulty work units have cycled from your systems there might still be some errors generated. If you are in hurry or badly affected, a project reset to trigger a resending of work will hopefully update the wu metadata stored by your BOINC Client.

-w

Thanks Teemu.

I'll try that rset again.

I don't think I've had a Computation Error since Aborting all work & slowly getting new work, can't be positive on that though but don't think I have ... Thanks Teemu

Seems the Wu's are taking a longer time to finish though ...

---

I thin we were seeing 2 things happening -

1. The -131 error where a WU wet full time but errored so no credit (for me this rose from 5% towards 10%+. Hopefully, as Teemu said, this may now be cured, but I need to leave it 24 hours to check (I did a project reset like Ste\/e.

2. The WUs seemed to need about 5% to 10% more crunch time, and were getting a marginally lower credit at 6.5K rather than 7K.

I think the latter is what Ste\/e was posting about. I can confirm the same on 2 rigs.

Although the actual number of faulty work units was not overly large about 10% seems to be the number, my performance here has dropped markedly.

My run times are as good as before the errors, my throughput is the same (or appears to be the same) as normal, the amount awarded per result seems to be the same
Yet my RAC is dropping faster every day, which corresponds to the drop in output that I have noticed as well.
RAC is down at least 60,000 or more, with RAC being an average figure it shows it has been happening for a little while now.
This is despite most things all appearing to be running fine.

I don't get why there has been such a big drop.
When I checked on Boincstats everyone has had a major drop in output.

Puzzled

Conan

---

I'm having the same longer crunch times
A 768 used to take about 30 minutes on a 5850 / 6950
Now it is up to 38 minutes

Seems the Wu's are taking a longer time to finish though ...

Since there's more packets in work units due to fragmentation then that means more context switches for the GPU when D.net Client moves through the packets. This can slow down the performance for obvious reasons and I wouldn't be surprised to see higher CPU usage as well. Just like having the client interrupted all the time can increase CPU usage due to increased GPU traffic.

I'm not sure I can do anything about this other than hope we blow through these old fragmented areas quickly so we can get to fresh blocks. Block fetching for the work generation is entirely handled by non-opensource upstream code so I'm kinda at mercy of that code. :(

That said, I'll try to look if there's more going on to affect our RACs as I find that concerning. Could be it's just an effect of not catching this problem sooner. It has been going on since at least from 10th of December. :(

-w

Seems the Wu's are taking a longer time to finish though ...

Since there's more packets in work units due to fragmentation then that means more context switches for the GPU when D.net Client moves through the packets. This can slow down the performance for obvious reasons and I wouldn't be surprised to see higher CPU usage as well. Just like having the client interrupted all the time can increase CPU usage due to increased GPU traffic.

I'm not sure I can do anything about this other than hope we blow through these old fragmented areas quickly so we can get to fresh blocks. Block fetching for the work generation is entirely handled by non-opensource upstream code so I'm kinda at mercy of that code. :(

That said, I'll try to look if there's more going on to affect our RACs as I find that concerning. Could be it's just an effect of not catching this problem sooner. It has been going on since at least from 10th of December. :(

-w

Since these WUs are taking 10%-15% longer to complete how about raising the credit 10%-15% to compensate then go back to old credit when they are finished (or not). I know some of the other WUs are mixed in but those would just be bonuses during this time. A simple short term fix I would think.

NeoMetal

The increase in time vary from card to card.
On my HD6950 it is about 10 - 15% longer but on the older HD5850 that is normally 5% slower than the 6950 the time increased to about 30 - 35%.
Both cards are optimized to run on the fastest GPU core with a free CPU core available on both PCs
My personal RAC dropped from 835k (on the 10th) to currently 775k & is still falling rapidly
I checked my failed WUs & found only 4 so this can't be the reason

Everthing all normal here now...Cool! CrunchTimes are right on, heat, the roaring hummmmm of the fans!
THANKS!!!

---

My WU´s failed to approx 50%. Hoping for better result. I trust Teemu that he is/has fixed this!

---

Slowly but surely recover!
I´m happy.

---

The -131 errors seem to have been corrected by Teemu.

Still showing marginally extended crunch times for WUs, which I hope will rectify over time.

Some crunchers seem totally unaffected, which is weird.

I noticed a credit drop of around 500 per WU. No big deal per se, but just wondering if anyone has noticed a slight drop. Like 5% or so.

---