Posts by Metod, S56RKO

1) Message boards : Number crunching : Forcing work on one GPU only (Message 2849) Posted 7 Mar 2012 by Metod, S56RKO Post: On my only dual gpu machine I AM running both on Moo right now, but I used to run one on Collatz. OK, so you have a home work to do: when you will split your GPU power to two projects again, check the output of one task - not the first one of course, rather one that gets downloaded after you do the split. I'd be quite interested in that output.
2) Message boards : Number crunching : Forcing work on one GPU only (Message 2842) Posted 6 Mar 2012 by Metod, S56RKO Post: I don't know how DNETC behaves in Windows and how you can actually verify proper operation (with regard to exclusion), however I proved in my previous posts that it doesn't work as it should in Linux. By having 2 gpu's in one machine and each crunching for a different project, one being Moo. Nope. If you check the output of a task from your dual-GPU machine (e.g. this one), you will see that DNETC application started two worker threads (#a and #b) and assigned work to both. Compare this to output from task being run on any other of your machines (such as this one) which only starts one worker thread. Which means that DNETC application is as greedy on Windows as on Linux. You should not trust BOINC manager when checking if project application behaves ... BOINC manager has no idea about that.
3) Message boards : Number crunching : Forcing work on one GPU only (Message 2836) Posted 6 Mar 2012 by Metod, S56RKO Post: I'm using BOINC 7.0.18. And cc_config.xml works just fine if placed in BOINC data directory. I'm using some debug options and as far as I observed, BOINC respects exclusion of Moo! form GPU number 1. Again: it's not up to project application to actually read the cc_config.xml file, it's up to BOINC to pass appropriate settings and it's up to project application just to obey them. For this part, OS choice doesn't matter at all. E.g. how is project application supposed to know on which GPU is it supposed to run unless it gets this application from BOINC CC? I don't know how DNETC behaves in Windows and how you can actually verify proper operation (with regard to exclusion), however I proved in my previous posts that it doesn't work as it should in Linux. Nevertheless, it's not up to us volunteers to fix the application, it's up to Moo! project and DNETC project.
4) Message boards : Number crunching : Forcing work on one GPU only (Message 2831) Posted 5 Mar 2012 by Metod, S56RKO Post: The exclude option means that BOINC CC will not assign Moo! task to that particular GPU. If the task does as instructed, then that's the end of story. Not so with DNETC application which is greedy and uses all of the resources of the same kind (CPU or GPU) and doesn't care about what BOINC CC is instructing it to do. The application doesn't know that some particular GPU is off-limits because nobody told it so ... and it doesn't care either as it is right now. Here's process list from my Linux box: boinc 19783 0.0 0.0 11284 1136 ? SNl 18:02 0:04 ../../projects/moowrap.net/dnetc_wrapper_1.3_x86_64-pc-linux-gnu__cuda31 --device 0 boinc 19785 10.0 0.0 79853508 63552 ? SNl 18:02 11:28 dnetc -ini dnetc.ini -runoffline -multiok=1 -e email@somewhere.net The first line is Moo! wrapper and BOINC CC passes instruction to use only device 0 (NVIDIA GPU). The second line is actual DNETC application which is not BOINC aware (that's why we need wrapper). Note, that information about which GPU to use got lost between the two. BTW, the actual DNETC application is instructed not to top up the buffer (option -runoffline) which would otherwise be done in a DNETC way. The application will terminate after it empties the work buffer - that one had been prepared by Moo! server as a task. Now a part of output of the DNETC application: 18:02:54 (19783): wrapper: starting v1.3.9.7 18:02:54 (19783): device: 2 x GeForce GT 430 (driver version unknown, CUDA version 4.20, compute capability 2.1, 1024MB, 1001MB available, 280 GFLOPS peak) 18:02:54 (19783): checkpoint interval: 25 min (task 414750 GFLOPS, 12 min) 18:02:54 (19783): wrapper: running dnetc517-linux-amd64-cuda31 (-ini dnetc.ini -runoffline -multiok=1) - attempt 1/10 dnetc v2.9108-517-CTR-10070313 for CUDA 3.1 on Linux (Linux 2.6.36.1). Using email address (distributed.net ID) 'email@somewhere.net' [Mar 05 18:02:54 UTC] Automatic processor detection found 2 processors. [Mar 05 18:02:54 UTC] Loading crunchers with work... [Mar 05 18:02:54 UTC] Automatic processor type detection found a GeForce GT 430 (64 SPs) processor. [Mar 05 18:02:54 UTC] RC5-72: using core #0 (CUDA 1-pipe 64-thd). [Mar 05 18:02:54 UTC] RC5-72 #a: Loaded CD:B80D28C8:00000000:482^32 [Mar 05 18:02:54 UTC] RC5-72 #b: Loaded CD:B80D4442:00000000:642^32 At the top there's some output from the wrapper application (it already omits the info about assigned GPU device). Then DNETC starts: it detected 2 GPUs and says that it started two threads. To add insult to injury thread b (which is executed by off-limits NVIDIA device 1) got a larger work batch (64 vs. 48) and that one is competing for GPU cycles with another application (from a different BOINC project) which actually behaves and only runs on the designated GPU device. Which means that while DNETC crunching will run full-speed on GPU 0, it will crawl on GPU 1. My estimation is that GPU 0 will sit idle at least 50% of time because DNETC application will wait for thread b to finish. This is where DNETC project should step towards BOINC community. Unless there's some option for the DNETC application that would instruct i only to use particular device (CPU or GPU), Moo! won't be able to tie the two distributed computing worlds together. Second yes you DO need the exclude option for other projects or they will run more than one unit at a time using all gpu's available to it, just putting one workunit on each gpu. BOINC CC takes care not to over commit resources. If there's one Moo! running (assumed to run on one GPU but this is not the case as illustrated above), then it won't assign another task to use the same resource. And that works just fine for the rest of GPU projects I participate. And, BTW, you got it mixed up a bit for your third argument. A BOINC task as served by project (Moo!) server consists of (set of) executable as well as data file. Without forcing it other way BOINC will only process data file using designated executable. In Moo! case data file contains work cache and it won't be processed by application other than the one defined when task had been served (either CPU or GPU). This means it's perfectly safe to mix CPU and GPU tasks from the same project on the same machine as BOINC will not confuse them (it beter doesn't, there are projects that serve completely different applications and data files. You can immagine disaster if BOINC CC would mix them at will). No, I'm no running CPU tasks from Moo! ... and I'm quite sure I won't be running GPU tasks on this dual-GPU machine either until this issue gets resolved.
5) Message boards : Number crunching : Forcing work on one GPU only (Message 2820) Posted 4 Mar 2012 by Metod, S56RKO Post: Thank you for your effort. I will try a few aspects of your proposed solution after task backlog gets cleared a bit - right now I excluded second GPU from any BOINC projects, giving all GPU power to Moo! (BOINC doesn't intend to use it but DNETC uses it). The workaround certainly helps to reduce congestion between two Moo! tasks trying to consume same GPU resources (as I mentioned before, if two dnetc applications are running concurrently, they both use excessive amount of CPU). There are a few problems with this work-around ... 1. it doesn't make DNETC application behave BOINC-friendly: it will still grab all GPUs it sees, thus starving other projects. One might not want to dedicate all the resources to distributed computing (in general) 2. it doesn't prevent DNETC from wasting resources due to non-synchronous processing of work batches. BTW, the same problem does arise also with CPU application when one has many CPUs in machine (think dual hexa core HT server, like dual Intel X5670 or similar), where it hits even harder BTW, I don't think it's necessary to exclude the second GPU from other BOINC projects. They mostly (if not all) only require single GPU and play well alongside each other. One might want to do it only if they want to see Moo! running all the time. Your workaround might also work around the BOINC CC bug which prevents from running multi-GPU tasks ... but will have to test and see.
6) Message boards : Number crunching : Forcing work on one GPU only (Message 2815) Posted 3 Mar 2012 by Metod, S56RKO Post: An addition to my previous post: the way DNETC handles multiple GPUs is not good even if one only runs tasks from Moo! I observed that tasks don't contain homogenous work batches. For example, a '192' task is typically composed of several work batches and size of those varies from 1 to 64 - the sum is 192. When DNETC starts, it finds 2 GPUs so it starts two worker threads. Then it assigns a work batch to each of them. Then it waits for worker thread to finish work batch and assigns next one. If work batches are not uniform in size or GPUs are of different speed, then both worker threads don't finish their jobs at the same time. At the end of task one GPU sits idle until the other one finishes its work batch. This waiting time can vary between almost zero up to total processing time (in my case that's more than an hour for a '192' task). My observation is that in average almost 10% of GPU cycles get wasted. This phenomenon does not happen if DNETC is runs 'natively' in sense that it also maintains cache of work. It only happens if DNETC is left to run down the buffer - this happen all the time with Moo! If DNETC would limit itself to only use one GPU, BOINC CC would start two DNETC tasks, and upon finishing of one task (on one GPU) it would immediately start another one. Hence no time would get wasted.
7) Message boards : Number crunching : Forcing work on one GPU only (Message 2813) Posted 3 Mar 2012 by Metod, S56RKO Post: mikey wrote: The format here is to run one task on all available gpu's in the machine, that is one of the premises of the project. YES it can be changed, but yours IS doing what it was designed to do. I fully understand that and I fully accept that from the original Distributed.net client. I've been running the original DNETC client back in 1997 (RC5-56 contest), so I do know and understand their way of distributing work. However, BOINC has a bit different nature in a way that different projects nicely co-exist. In this case there's "higher authority" (BOINC core client) which assigns part of resources and project application is supposed to obey that. Most of projects (a least the ones I participate at) behave according to this spirit, while DNETC client does not. It's not that BOINC per se could not fulfill the premises of the project you're mentioning, it's just that it would be done a bit differently. I would expect from Moo! more than only wrap DNETC tasks for use in BOINC environment, but to try (in any possible way) to also adapt it to BOINC way of distributed computing. I don't think that it's Moo! project management to blame completely. I just made an observation and asked/proposed about possible solutions. I don't expect that everybody agrees with me as I fully understand that my case is not typical one: not everybody has got more than one GPU installed and of those not everybody participates in other projects as well.
8) Message boards : Number crunching : Forcing work on one GPU only (Message 2810) Posted 2 Mar 2012 by Metod, S56RKO Post: Me too! ;-) I have two Nvidia GT430's in machine with BOINC 7.0.18. First of all, BOINC doesn't behave when it gets task which requests 2 GPUs a the same time. It won't run those tasks even though both GPUs are idle. I've reported this at BOINC developers' forum and will see where this will head. Second: I think wrapper should run dnetc application with cmd line argument -multiok=0. The application is designed to grab all available GPUs regardless of what BOINC designates. I guess it would be safe to prevent running more than one dnetc apps in parallel: * they both compete for same GPU resources * on my host both apps also consumed around 40% of CPU each. If I instruct BOINC only to use one GPU (so that only one Moo! task is run at a time), dnetc still consumes both GPUs while only consuming around 1% of CPU. The only question is then how would BOINC behave if Moo! wrapper would return immediately. It would probably re-start the same task again and again. Maybe wrapper should detect such state and just wait for other wrappers to finish their run before starting dnetc application?