Based on the screenshot you are calling the program correctly. Can you tell me what version of Docker you are running and Galaxy you are running? There have been a number of fixes in both programs over the last six months to a year. Docker is known to consume a lot of space in /var - it probably doesn't have much to do with Galaxy or this tool right. It looks like you Docker command line does not include -rm - which was added recently to have Docker dispose of instances after it done with them. This might help with your /var space issue - though in general Docker does use a lot a relatively large amount of disk under /var - it is the nature of the technology and many traditional HPC clusters for instance may allocate relatively small /var partitions. Upgrading Galaxy might also help the permission thing - I don't see a '-u <uid>' argument in your command-line - which is another newer default to help ensure the resulting files have the correct permissions (especially when sudo is disabled - which it seems it is in your configuration). Hope this helps, -John On Mon, Jan 26, 2015 at 7:02 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
Our sysadmin just notified me docker was taking up large amounts of space in /var/lib/docker/devicemapper/devicemapper/data and /var/lib/docker/devicemapper/devicemapper/metadata
I noticed the following DEBUG message in paster.log:
galaxy.jobs.runners.local DEBUG 2015-01-23 13:30:19,503 (12) executing job script: /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/galaxy_12.sh galaxy.jobs DEBUG 2015-01-23 13:30:19,552 (12) Persisting job destination (destination id: docker_local)
Are these related? (the files I'm trying to concatenate are small text files so that's not it)
On Mon, Jan 26, 2015 at 11:47 AM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
See attached for screenshot!
Thanks for the links.
On Mon, Jan 26, 2015 at 11:37 AM, John Chilton <jmchilton@gmail.com> wrote:
Can you send me a screenshot of the Tool form right before you hit submit. It is odd that two commands are executed barely differing at all. There were some recent UI changes and I want to make sure that you are passing two inputs once and not submitting one input twice in the newly named "batch mode"?
container.sh is generated in command_factory.py using containers.py - here are some relevant links.
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/jobs/command_...
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/co...
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/do...
-John
On Mon, Jan 26, 2015 at 2:17 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
I'm still trying to debug this.
How do I change the working directory? I can't seem to find out how all those directory mounts get passed to docker - the catDocker.xml appears to generate a 'cat' command that I assume ends up in ()/job_working_directory/000/12/container.sh but I can't find where that's happening - container.sh isn't listed in any file under galaxy-dist. job_conf.xml lists the directories that get passed as variables - again, when and where are those defined?
Thanks,
-Jeltje
On Fri, Jan 23, 2015 at 1:35 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
I tried the second solution (setting everything to rw) and while I can confirm that the command is changed accordingly, the results are the same: Two empty output files and the same error message.
Command from paster.log: docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:rw -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:rw -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:rw -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw -w
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12 --net none busybox:ubuntu-14.04
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/container.sh; return_code=$?; if [ -f
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file ] ; then cp
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_19.dat ; fi; sh -c "exit $return_code"
-Jeltje
On Fri, Jan 23, 2015 at 12:24 PM, John Chilton <jmchilton@gmail.com> wrote:
My first thought is the bug mentioned here -
https://lists.galaxyproject.org/pipermail/galaxy-dev/2014-November/020892.ht... along with potential work arounds. Are you able to confirm that this is or is not the problem at all?
-John
On Fri, Jan 23, 2015 at 2:37 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote: > Hi, > > I'm following instructions at > https://github.com/apetkau/galaxy-hackathon-2014. > > When I try to run the concatenate-datasets on two input files, two > odd > things happen. First, TWO nearly identical commands are generated > in > the > History panel, only differing in their output filename. In > paster.log, > only > the second one shows up. Second, the program fails, with this info > in > the > History: > > An error occurred with this dataset: > Galaxy slots passed through contain as 1 > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh: > line 2: can't create > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_14.dat > > This happens for both commands in the History panel, only the > output > filename in the other error is dataset_15.dat > > Oddly, the dataset_14.dat and dataset_15.dat are both created > during > this > command, they just end up empty. > > Paster.log: > > galaxy.jobs.runners DEBUG 2015-01-23 11:08:35,973 (10) command is: > docker > inspect busybox:ubuntu-14.04 > /dev/null 2>&1 > [ $? -ne 0 ] && docker pull busybox:ubuntu-14.04 > /dev/null 2>&1 > > docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:ro > -v > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:ro > -v > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:rw > -v > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw > -w > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10 > --net none busybox:ubuntu-14.04 > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh; > return_code=$?; if [ -f > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file > ] ; then cp > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_15.dat > ; fi; sh -c "exit $return_code" > galaxy.jobs.runners.local DEBUG 2015-01-23 11:08:36,040 (10) > executing > job > script: > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/galaxy_10.sh > galaxy.jobs DEBUG 2015-01-23 11:08:36,091 (10) Persisting job > destination > (destination id: docker_local) > > > What could be happening here, and how do I fix it? > > Thanks, > > -Jeltje > > > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/