Forcing work on one GPU only

\n studio-striking\n

Message boards : Number crunching : Forcing work on one GPU only
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
GFrost

Send message
Joined: 16 May 11
Posts: 2
Credit: 7,455,289
RAC: 0
Message 2379 - Posted: 23 Jan 2012, 13:04:27 UTC

Hi, is there a way to force moo to only do work on one GPU instead of all available.
If I understand things correctly this might be possible with an app_info file but I don't know what to write in it and where to put it.
I wonder if someone can help me with it. My GPUs are GTX470.

The reason I would like to do this is because I have multiple GPU WUs from diffirent projects and when boinc want's to switch and calculate moo WU's one GPU sits idle for upp to 60 minutes, several times a day.
ID: 2379 · Rating: 0 · rate: Rate + / Rate - Report as offensive
juice3

Send message
Joined: 6 Dec 11
Posts: 60
Credit: 306,719,331
RAC: 0
Message 2402 - Posted: 25 Jan 2012, 17:22:02 UTC

Do you want to do this for just moo? Or are you running other projects as well?
ID: 2402 · Rating: 0 · rate: Rate + / Rate - Report as offensive
GFrost

Send message
Joined: 16 May 11
Posts: 2
Credit: 7,455,289
RAC: 0
Message 2405 - Posted: 25 Jan 2012, 22:17:13 UTC - in response to Message 2402.  

Yes, I'm running other projects as well as moo, but I just want to force Moo since the other projects only use one GPU core per WU and not "all available".
ID: 2405 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Wizzo

Send message
Joined: 1 Jan 12
Posts: 13
Credit: 21,324,276
RAC: 0
Message 2429 - Posted: 29 Jan 2012, 2:53:20 UTC

If you use an appinfo you will run one unit PER gpuat the same time. I have 2 6990's, and moo wants to run 4 app at a time.
ID: 2429 · Rating: 0 · rate: Rate + / Rate - Report as offensive
DarkRyder

Send message
Joined: 23 Jun 11
Posts: 87
Credit: 798,452,366
RAC: 0
Message 2431 - Posted: 29 Jan 2012, 5:35:37 UTC

is there any way to force Moo Wrapper to do 1 workunit per gpu?
ID: 2431 · Rating: 0 · rate: Rate + / Rate - Report as offensive
mikey
Avatar

Send message
Joined: 22 Jun 11
Posts: 2080
Credit: 1,826,336,240
RAC: 3,658
Message 2435 - Posted: 29 Jan 2012, 11:56:24 UTC - in response to Message 2431.  

is there any way to force Moo Wrapper to do 1 workunit per gpu?


Yes thru an app_info.xml file you can do it. But Moo is designed to run using everything gpu it finds all working on one workunit, kind of like a super computer.
ID: 2435 · Rating: 0 · rate: Rate + / Rate - Report as offensive
DarkRyder

Send message
Joined: 23 Jun 11
Posts: 87
Credit: 798,452,366
RAC: 0
Message 2441 - Posted: 30 Jan 2012, 4:49:51 UTC

can you tell me how to do it please?
ID: 2441 · Rating: 0 · rate: Rate + / Rate - Report as offensive
mikey
Avatar

Send message
Joined: 22 Jun 11
Posts: 2080
Credit: 1,826,336,240
RAC: 3,658
Message 2443 - Posted: 30 Jan 2012, 13:37:50 UTC - in response to Message 2441.  

can you tell me how to do it please?


Read this thread:
http://moowrap.net/forum_thread.php?id=205#2315

There is lots of info there and you MUST be careful, you could lose lots of units and be put on workunit lockdown if you mess things up to bad! Be SURE to run thru ALL of your current units and set your cache to 0.01 before you make changes!
ID: 2443 · Rating: 0 · rate: Rate + / Rate - Report as offensive
DarkRyder

Send message
Joined: 23 Jun 11
Posts: 87
Credit: 798,452,366
RAC: 0
Message 2447 - Posted: 30 Jan 2012, 15:17:14 UTC - in response to Message 2443.  

seems that shows how to limit the project to run on gpu0 and not the rest. Just to make sure you know what i'm wanting to do. I have 3 gpus in a system. I would like to run a single workunit from moowrapper on each gpus at the same time. Not spread 1 workunit to 3 different gpus. Is there any way to do this?
ID: 2447 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Profile Zydor
Avatar

Send message
Joined: 5 May 11
Posts: 233
Credit: 351,414,150
RAC: 0
Message 2449 - Posted: 30 Jan 2012, 15:59:33 UTC
Last modified: 30 Jan 2012, 16:23:39 UTC

Thats what Mikey pointed you at, that app_info puts one per gpu.

This is the one I am using with 2x5970s, they run one Moo WU per GPU

<app_info>
<app>
<name>dnetc</name>
<user_friendly_name>Distributed.net Client</user_friendly_name>
</app>
<file_info>
<name>dnetc_wrapper_1.3_windows_intelx86__ati14.exe</name>
<executable/>
</file_info>
<file_info>
<name>dnetc518-win32-x86-stream.exe</name>
<executable/>
</file_info>
<file_info>
<name>dnetc-gpu-1.3.ini</name>
</file_info>
<file_info>
<name>job-ati14-1.00.xml</name>
</file_info>
<app_version>
<app_name>dnetc</app_name>
<version_num>103</version_num>
<platform>windows_intelx86</platform>
<avg_ncpus>0.1</avg_ncpus>
<max_ncpus>0.5</max_ncpus>
<plan_class>ati14</plan_class>
<flops>1157115231469.729200</flops>
<file_ref>
<file_name>dnetc_wrapper_1.3_windows_intelx86__ati14.exe</file_name>
<main_program/>
</file_ref>
<file_ref>
<file_name>dnetc518-win32-x86-stream.exe</file_name>
<copy_file/>
</file_ref>
<file_ref>
<file_name>dnetc-gpu-1.3.ini</file_name>
<open_name>dnetc.ini</open_name>
<copy_file/>
</file_ref>
<file_ref>
<file_name>job-ati14-1.00.xml</file_name>
<open_name>job.xml</open_name>
<copy_file/>
</file_ref>
<coproc>
<type>ATI</type>
<count>1</count>
</coproc>
</app_version>
</app_info>

Create it inside Notepad (no other editor - use Notepad), and save file as a txt format file called app_info.xml, into the Moo Project Directory. Dont forget to run down the cache before using it else you will trash existing WUs. BAM must be closed down and restarted for the app_info to take effect.

EDIT:
In an app_info, the part that contrrols number of WUs running on the gpu is this bit:

<coproc>
<type>ATI</type>
<count>1</count>
</coproc>

A count of one means one per GPU. If it was put as 0.5, it would be 2 per GPU .... and so on, so for example(s)

Count of ...

1 is 1 per GPU
0.5 is two per GPU
0.33 is three per GPU
0.25 is 4 per GPU.

Do the Maths you'll get the idea

(Dont run more than two Moo WU per GPU, utterly pointless, just creates other issues for zero purpose, and usually one per GPU in a multi GPU setup solves the fragmentation / utilisation problem.)

Watch the heat carefully when first starting this, expect to have to downclock GPUs by about 30 on previous setting to keep heat under control - more work flying around, hence the need to reduce GPU speed slighly to compensate - be careful ... you will overheat if you dont watch this the first time you use the app_info.

Regards
Zy
ID: 2449 · Rating: 0 · rate: Rate + / Rate - Report as offensive
DarkRyder

Send message
Joined: 23 Jun 11
Posts: 87
Credit: 798,452,366
RAC: 0
Message 2452 - Posted: 30 Jan 2012, 18:08:42 UTC - in response to Message 2449.  

Thanks so much man! you guys are awesome!
ID: 2452 · Rating: 0 · rate: Rate + / Rate - Report as offensive
juice3

Send message
Joined: 6 Dec 11
Posts: 60
Credit: 306,719,331
RAC: 0
Message 2454 - Posted: 31 Jan 2012, 4:33:31 UTC

Talk about heat. I have my systems in a 50 degree cold room with massive air flow and I'm still seeing rise of 12-20 degree C over ambient depending on load on the CPU from other tasks.

And that's on liquid...
ID: 2454 · Rating: 0 · rate: Rate + / Rate - Report as offensive
DarkRyder

Send message
Joined: 23 Jun 11
Posts: 87
Credit: 798,452,366
RAC: 0
Message 2492 - Posted: 2 Feb 2012, 20:54:22 UTC - in response to Message 2454.  

what kinda hardware are you runnin ?
ID: 2492 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Metod, S56RKO

Send message
Joined: 1 Mar 12
Posts: 8
Credit: 7,361,960
RAC: 0
Message 2810 - Posted: 2 Mar 2012, 19:09:37 UTC - in response to Message 2449.  
Last modified: 2 Mar 2012, 19:10:43 UTC

Me too! ;-)

I have two Nvidia GT430's in machine with BOINC 7.0.18.

First of all, BOINC doesn't behave when it gets task which requests 2 GPUs a the same time. It won't run those tasks even though both GPUs are idle. I've reported this at BOINC developers' forum and will see where this will head.

Second: I think wrapper should run dnetc application with cmd line argument -multiok=0. The application is designed to grab all available GPUs regardless of what BOINC designates. I guess it would be safe to prevent running more than one dnetc apps in parallel:
    * they both compete for same GPU resources
    * on my host both apps also consumed around 40% of CPU each. If I instruct BOINC only to use one GPU (so that only one Moo! task is run at a time), dnetc still consumes both GPUs while only consuming around 1% of CPU.



The only question is then how would BOINC behave if Moo! wrapper would return immediately. It would probably re-start the same task again and again. Maybe wrapper should detect such state and just wait for other wrappers to finish their run before starting dnetc application?


Metod ...
ID: 2810 · Rating: 0 · rate: Rate + / Rate - Report as offensive
mikey
Avatar

Send message
Joined: 22 Jun 11
Posts: 2080
Credit: 1,826,336,240
RAC: 3,658
Message 2811 - Posted: 2 Mar 2012, 23:31:59 UTC - in response to Message 2810.  

Me too! ;-)

I have two Nvidia GT430's in machine with BOINC 7.0.18.

First of all, BOINC doesn't behave when it gets task which requests 2 GPUs a the same time. It won't run those tasks even though both GPUs are idle. I've reported this at BOINC developers' forum and will see where this will head.

Second: I think wrapper should run dnetc application with cmd line argument -multiok=0. The application is designed to grab all available GPUs regardless of what BOINC designates. I guess it would be safe to prevent running more than one dnetc apps in parallel:
    * they both compete for same GPU resources
    * on my host both apps also consumed around 40% of CPU each. If I instruct BOINC only to use one GPU (so that only one Moo! task is run at a time), dnetc still consumes both GPUs while only consuming around 1% of CPU.



The only question is then how would BOINC behave if Moo! wrapper would return immediately. It would probably re-start the same task again and again. Maybe wrapper should detect such state and just wait for other wrappers to finish their run before starting dnetc application?



The format here is to run one task on all available gpu's in the machine, that is one of the premises of the project. YES it can be changed, but yours IS doing what it was designed to do.
ID: 2811 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Metod, S56RKO

Send message
Joined: 1 Mar 12
Posts: 8
Credit: 7,361,960
RAC: 0
Message 2813 - Posted: 3 Mar 2012, 14:54:28 UTC - in response to Message 2811.  

mikey wrote:
The format here is to run one task on all available gpu's in the machine, that is one of the premises of the project. YES it can be changed, but yours IS doing what it was designed to do.


I fully understand that and I fully accept that from the original Distributed.net client. I've been running the original DNETC client back in 1997 (RC5-56 contest), so I do know and understand their way of distributing work.

However, BOINC has a bit different nature in a way that different projects nicely co-exist. In this case there's "higher authority" (BOINC core client) which assigns part of resources and project application is supposed to obey that. Most of projects (a least the ones I participate at) behave according to this spirit, while DNETC client does not.

It's not that BOINC per se could not fulfill the premises of the project you're mentioning, it's just that it would be done a bit differently. I would expect from Moo! more than only wrap DNETC tasks for use in BOINC environment, but to try (in any possible way) to also adapt it to BOINC way of distributed computing.

I don't think that it's Moo! project management to blame completely. I just made an observation and asked/proposed about possible solutions. I don't expect that everybody agrees with me as I fully understand that my case is not typical one: not everybody has got more than one GPU installed and of those not everybody participates in other projects as well.
Metod ...
ID: 2813 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Metod, S56RKO

Send message
Joined: 1 Mar 12
Posts: 8
Credit: 7,361,960
RAC: 0
Message 2815 - Posted: 3 Mar 2012, 20:11:58 UTC - in response to Message 2813.  

An addition to my previous post: the way DNETC handles multiple GPUs is not good even if one only runs tasks from Moo! I observed that tasks don't contain homogenous work batches. For example, a '192' task is typically composed of several work batches and size of those varies from 1 to 64 - the sum is 192. When DNETC starts, it finds 2 GPUs so it starts two worker threads. Then it assigns a work batch to each of them. Then it waits for worker thread to finish work batch and assigns next one. If work batches are not uniform in size or GPUs are of different speed, then both worker threads don't finish their jobs at the same time. At the end of task one GPU sits idle until the other one finishes its work batch. This waiting time can vary between almost zero up to total processing time (in my case that's more than an hour for a '192' task). My observation is that in average almost 10% of GPU cycles get wasted.

This phenomenon does not happen if DNETC is runs 'natively' in sense that it also maintains cache of work. It only happens if DNETC is left to run down the buffer - this happen all the time with Moo!

If DNETC would limit itself to only use one GPU, BOINC CC would start two DNETC tasks, and upon finishing of one task (on one GPU) it would immediately start another one. Hence no time would get wasted.
Metod ...
ID: 2815 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Tex1954

Send message
Joined: 25 Jul 11
Posts: 5
Credit: 23,397,372
RAC: 6
Message 2816 - Posted: 4 Mar 2012, 0:24:52 UTC - in response to Message 2815.  
Last modified: 4 Mar 2012, 0:29:12 UTC

I can report the same problem running Moo! Wrapper on the ATI HD6990 card.

If I use the above app_info.xml file and say one GPU per task, it will load and run two tasks... displays showing 1ATI per WU. But often during crunching, the GPU usage will drop or oscillate between 90-99% as if it is unstable.

Also, the Dnet application looks to use ALL GPU's in real time, so if you suspend one task, the other starts using both GPU's even though app_info doesn't prescribe that. That causes more problems as eventually it's trying to run two tasks on two GPU's at the same time.

This tells me both applications are polling both GPU's at the same time trying to compete for GPU resources... and explains the large differences I see in task completion times. One would expect a simple doubling of times, but that doesn't seem to be the case... it can be a lot worse than that...

This behavior also seen when starting a collatz task on one GPU and Moo on the another. If the collatz task is suspended, the Moo! task starts using both GPU's!

So, one can see this isn't nice behavior and contrary to BOINC methodology in general.


:)
ID: 2816 · Rating: 0 · rate: Rate + / Rate - Report as offensive
mikey
Avatar

Send message
Joined: 22 Jun 11
Posts: 2080
Credit: 1,826,336,240
RAC: 3,658
Message 2817 - Posted: 4 Mar 2012, 12:08:42 UTC - in response to Message 2813.  
Last modified: 4 Mar 2012, 12:13:55 UTC

mikey wrote:
The format here is to run one task on all available gpu's in the machine, that is one of the premises of the project. YES it can be changed, but yours IS doing what it was designed to do.


I fully understand that and I fully accept that from the original Distributed.net client. I've been running the original DNETC client back in 1997 (RC5-56 contest), so I do know and understand their way of distributing work.

However, BOINC has a bit different nature in a way that different projects nicely co-exist. In this case there's "higher authority" (BOINC core client) which assigns part of resources and project application is supposed to obey that. Most of projects (a least the ones I participate at) behave according to this spirit, while DNETC client does not.

It's not that BOINC per se could not fulfill the premises of the project you're mentioning, it's just that it would be done a bit differently. I would expect from Moo! more than only wrap DNETC tasks for use in BOINC environment, but to try (in any possible way) to also adapt it to BOINC way of distributed computing.

I don't think that it's Moo! project management to blame completely. I just made an observation and asked/proposed about possible solutions. I don't expect that everybody agrees with me as I fully understand that my case is not typical one: not everybody has got more than one GPU installed and of those not everybody participates in other projects as well.


Okay now that we have that out of the way try this:
<cc_config>
<options>
<use_all_gpus>1</use_all_gpus>
<exclude_gpu>
<url>http://http:/moowrap.net/</url>
<device_num>0</device_num>
</exclude_gpu>
<exclude_gpu>
<url>http://boinc.thesonntags.com/collatz/</url>
<device_num>1</device_num>
</exclude_gpu>
<report_results_immediately>1</report_results_immediately>
</options>
</cc_config>

Change it to fit YOUR projects AND it ONLY works with Boinc version 7!!! I just saw that Boinc version 7.0.18 is the best to use, use Donnie in the app_info thread is who told me this works. What it does is tell gpu 1 to NOT work on Moo and gpu zero to NOT work on Collatz and then to return the results immediately. What you do is copy the above in Notepad in Windows, do NOT use a word processing program JUST NOTEPAD, and save the file with the file name of cc_config.xml but as a TXT file!!! Then stop Boinc, put the file in the Boinc/data directory and restart Boinc. The Boinc/data directory can be found in the Event Log or Messages tab near the top.
ID: 2817 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Metod, S56RKO

Send message
Joined: 1 Mar 12
Posts: 8
Credit: 7,361,960
RAC: 0
Message 2820 - Posted: 4 Mar 2012, 13:22:15 UTC - in response to Message 2817.  
Last modified: 4 Mar 2012, 13:43:55 UTC

Thank you for your effort. I will try a few aspects of your proposed solution after task backlog gets cleared a bit - right now I excluded second GPU from any BOINC projects, giving all GPU power to Moo! (BOINC doesn't intend to use it but DNETC uses it). The workaround certainly helps to reduce congestion between two Moo! tasks trying to consume same GPU resources (as I mentioned before, if two dnetc applications are running concurrently, they both use excessive amount of CPU).

There are a few problems with this work-around ...
    1. it doesn't make DNETC application behave BOINC-friendly: it will still grab all GPUs it sees, thus starving other projects. One might not want to dedicate all the resources to distributed computing (in general)
    2. it doesn't prevent DNETC from wasting resources due to non-synchronous processing of work batches. BTW, the same problem does arise also with CPU application when one has many CPUs in machine (think dual hexa core HT server, like dual Intel X5670 or similar), where it hits even harder



BTW, I don't think it's necessary to exclude the second GPU from other BOINC projects. They mostly (if not all) only require single GPU and play well alongside each other. One might want to do it only if they want to see Moo! running all the time.

Your workaround might also work around the BOINC CC bug which prevents from running multi-GPU tasks ... but will have to test and see.


Metod ...
ID: 2820 · Rating: 0 · rate: Rate + / Rate - Report as offensive
1 · 2 · Next

Message boards : Number crunching : Forcing work on one GPU only


 
Copyright © 2011-2024 Moo! Wrapper Project