concatenate_datasets docker image can't create output
Hi, I'm following instructions at https://github.com/apetkau/galaxy-hackathon-2014. When I try to run the concatenate-datasets on two input files, two odd things happen. First, TWO nearly identical commands are generated in the History panel, only differing in their output filename. In paster.log, only the second one shows up. Second, the program fails, with this info in the History: An error occurred with this dataset: Galaxy slots passed through contain as 1 /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh: line 2: can't create /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_14.dat This happens for both commands in the History panel, only the output filename in the other error is dataset_15.dat Oddly, the dataset_14.dat and dataset_15.dat are both created during this command, they just end up empty. Paster.log: galaxy.jobs.runners DEBUG 2015-01-23 11:08:35,973 (10) command is: docker inspect busybox:ubuntu-14.04 > /dev/null 2>&1 [ $? -ne 0 ] && docker pull busybox:ubuntu-14.04 > /dev/null 2>&1 docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:ro -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:ro -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:rw -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw -w /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10 --net none busybox:ubuntu-14.04 /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh; return_code=$?; if [ -f /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file ] ; then cp /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_15.dat ; fi; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2015-01-23 11:08:36,040 (10) executing job script: /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/galaxy_10.sh galaxy.jobs DEBUG 2015-01-23 11:08:36,091 (10) Persisting job destination (destination id: docker_local) What could be happening here, and how do I fix it? Thanks, -Jeltje
My first thought is the bug mentioned here - https://lists.galaxyproject.org/pipermail/galaxy-dev/2014-November/020892.ht... along with potential work arounds. Are you able to confirm that this is or is not the problem at all? -John On Fri, Jan 23, 2015 at 2:37 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
Hi,
I'm following instructions at https://github.com/apetkau/galaxy-hackathon-2014.
When I try to run the concatenate-datasets on two input files, two odd things happen. First, TWO nearly identical commands are generated in the History panel, only differing in their output filename. In paster.log, only the second one shows up. Second, the program fails, with this info in the History:
An error occurred with this dataset: Galaxy slots passed through contain as 1 /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh: line 2: can't create /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_14.dat
This happens for both commands in the History panel, only the output filename in the other error is dataset_15.dat
Oddly, the dataset_14.dat and dataset_15.dat are both created during this command, they just end up empty.
Paster.log:
galaxy.jobs.runners DEBUG 2015-01-23 11:08:35,973 (10) command is: docker inspect busybox:ubuntu-14.04 > /dev/null 2>&1 [ $? -ne 0 ] && docker pull busybox:ubuntu-14.04 > /dev/null 2>&1
docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:ro -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:ro -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:rw -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw -w /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10 --net none busybox:ubuntu-14.04 /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh; return_code=$?; if [ -f /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file ] ; then cp /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_15.dat ; fi; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2015-01-23 11:08:36,040 (10) executing job script: /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/galaxy_10.sh galaxy.jobs DEBUG 2015-01-23 11:08:36,091 (10) Persisting job destination (destination id: docker_local)
What could be happening here, and how do I fix it?
Thanks,
-Jeltje
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
I tried the second solution (setting everything to rw) and while I can confirm that the command is changed accordingly, the results are the same: Two empty output files and the same error message. Command from paster.log: docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:rw -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:rw -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:rw -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw -w /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12 --net none busybox:ubuntu-14.04 /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/container.sh; return_code=$?; if [ -f /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file ] ; then cp /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_19.dat ; fi; sh -c "exit $return_code" -Jeltje On Fri, Jan 23, 2015 at 12:24 PM, John Chilton <jmchilton@gmail.com> wrote:
My first thought is the bug mentioned here -
https://lists.galaxyproject.org/pipermail/galaxy-dev/2014-November/020892.ht... along with potential work arounds. Are you able to confirm that this is or is not the problem at all?
-John
On Fri, Jan 23, 2015 at 2:37 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
Hi,
I'm following instructions at https://github.com/apetkau/galaxy-hackathon-2014.
When I try to run the concatenate-datasets on two input files, two odd things happen. First, TWO nearly identical commands are generated in the History panel, only differing in their output filename. In paster.log, only the second one shows up. Second, the program fails, with this info in the History:
An error occurred with this dataset: Galaxy slots passed through contain as 1
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh:
line 2: can't create
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_14.dat
This happens for both commands in the History panel, only the output filename in the other error is dataset_15.dat
Oddly, the dataset_14.dat and dataset_15.dat are both created during this command, they just end up empty.
Paster.log:
galaxy.jobs.runners DEBUG 2015-01-23 11:08:35,973 (10) command is: docker inspect busybox:ubuntu-14.04 > /dev/null 2>&1 [ $? -ne 0 ] && docker pull busybox:ubuntu-14.04 > /dev/null 2>&1
docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:ro
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:ro
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:rw
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw
-w
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10
--net none busybox:ubuntu-14.04
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh;
return_code=$?; if [ -f
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file
] ; then cp
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_15.dat
; fi; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2015-01-23 11:08:36,040 (10) executing job script:
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/galaxy_10.sh
galaxy.jobs DEBUG 2015-01-23 11:08:36,091 (10) Persisting job destination (destination id: docker_local)
What could be happening here, and how do I fix it?
Thanks,
-Jeltje
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
I'm still trying to debug this. How do I change the working directory? I can't seem to find out how all those directory mounts get passed to docker - the catDocker.xml appears to generate a 'cat' command that I assume ends up in ()/job_working_directory/000/12/container.sh but I can't find where that's happening - container.sh isn't listed in any file under galaxy-dist. job_conf.xml lists the directories that get passed as variables - again, when and where are those defined? Thanks, -Jeltje On Fri, Jan 23, 2015 at 1:35 PM, Jeltje van Baren < jeltje.van.baren@gmail.com> wrote:
I tried the second solution (setting everything to rw) and while I can confirm that the command is changed accordingly, the results are the same: Two empty output files and the same error message.
Command from paster.log: docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:rw -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:rw -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:rw -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw -w /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12 --net none busybox:ubuntu-14.04 /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/container.sh; return_code=$?; if [ -f /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file ] ; then cp /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_19.dat ; fi; sh -c "exit $return_code"
-Jeltje
On Fri, Jan 23, 2015 at 12:24 PM, John Chilton <jmchilton@gmail.com> wrote:
My first thought is the bug mentioned here -
https://lists.galaxyproject.org/pipermail/galaxy-dev/2014-November/020892.ht... along with potential work arounds. Are you able to confirm that this is or is not the problem at all?
-John
Hi,
I'm following instructions at https://github.com/apetkau/galaxy-hackathon-2014.
When I try to run the concatenate-datasets on two input files, two odd things happen. First, TWO nearly identical commands are generated in the History panel, only differing in their output filename. In paster.log, only the second one shows up. Second, the program fails, with this info in
On Fri, Jan 23, 2015 at 2:37 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote: the
History:
An error occurred with this dataset: Galaxy slots passed through contain as 1
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh:
line 2: can't create
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_14.dat
This happens for both commands in the History panel, only the output filename in the other error is dataset_15.dat
Oddly, the dataset_14.dat and dataset_15.dat are both created during
this
command, they just end up empty.
Paster.log:
galaxy.jobs.runners DEBUG 2015-01-23 11:08:35,973 (10) command is: docker inspect busybox:ubuntu-14.04 > /dev/null 2>&1 [ $? -ne 0 ] && docker pull busybox:ubuntu-14.04 > /dev/null 2>&1
docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:ro
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:ro
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:rw
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw
-w
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10
--net none busybox:ubuntu-14.04
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh;
return_code=$?; if [ -f
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file
] ; then cp
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_15.dat
; fi; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2015-01-23 11:08:36,040 (10) executing job script:
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/galaxy_10.sh
galaxy.jobs DEBUG 2015-01-23 11:08:36,091 (10) Persisting job destination (destination id: docker_local)
What could be happening here, and how do I fix it?
Thanks,
-Jeltje
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Can you send me a screenshot of the Tool form right before you hit submit. It is odd that two commands are executed barely differing at all. There were some recent UI changes and I want to make sure that you are passing two inputs once and not submitting one input twice in the newly named "batch mode"? container.sh is generated in command_factory.py using containers.py - here are some relevant links. https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/jobs/command_... https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/co... https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/do... -John On Mon, Jan 26, 2015 at 2:17 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
I'm still trying to debug this.
How do I change the working directory? I can't seem to find out how all those directory mounts get passed to docker - the catDocker.xml appears to generate a 'cat' command that I assume ends up in ()/job_working_directory/000/12/container.sh but I can't find where that's happening - container.sh isn't listed in any file under galaxy-dist. job_conf.xml lists the directories that get passed as variables - again, when and where are those defined?
Thanks,
-Jeltje
On Fri, Jan 23, 2015 at 1:35 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
I tried the second solution (setting everything to rw) and while I can confirm that the command is changed accordingly, the results are the same: Two empty output files and the same error message.
Command from paster.log: docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:rw -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:rw -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:rw -v /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw -w /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12 --net none busybox:ubuntu-14.04 /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/container.sh; return_code=$?; if [ -f /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file ] ; then cp /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_19.dat ; fi; sh -c "exit $return_code"
-Jeltje
On Fri, Jan 23, 2015 at 12:24 PM, John Chilton <jmchilton@gmail.com> wrote:
My first thought is the bug mentioned here -
https://lists.galaxyproject.org/pipermail/galaxy-dev/2014-November/020892.ht... along with potential work arounds. Are you able to confirm that this is or is not the problem at all?
-John
On Fri, Jan 23, 2015 at 2:37 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
Hi,
I'm following instructions at https://github.com/apetkau/galaxy-hackathon-2014.
When I try to run the concatenate-datasets on two input files, two odd things happen. First, TWO nearly identical commands are generated in the History panel, only differing in their output filename. In paster.log, only the second one shows up. Second, the program fails, with this info in the History:
An error occurred with this dataset: Galaxy slots passed through contain as 1
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh: line 2: can't create
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_14.dat
This happens for both commands in the History panel, only the output filename in the other error is dataset_15.dat
Oddly, the dataset_14.dat and dataset_15.dat are both created during this command, they just end up empty.
Paster.log:
galaxy.jobs.runners DEBUG 2015-01-23 11:08:35,973 (10) command is: docker inspect busybox:ubuntu-14.04 > /dev/null 2>&1 [ $? -ne 0 ] && docker pull busybox:ubuntu-14.04 > /dev/null 2>&1
docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:ro -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:ro -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:rw -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw -w
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10 --net none busybox:ubuntu-14.04
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh; return_code=$?; if [ -f
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file ] ; then cp
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_15.dat ; fi; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2015-01-23 11:08:36,040 (10) executing job script:
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/galaxy_10.sh galaxy.jobs DEBUG 2015-01-23 11:08:36,091 (10) Persisting job destination (destination id: docker_local)
What could be happening here, and how do I fix it?
Thanks,
-Jeltje
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
See attached for screenshot! Thanks for the links. On Mon, Jan 26, 2015 at 11:37 AM, John Chilton <jmchilton@gmail.com> wrote:
Can you send me a screenshot of the Tool form right before you hit submit. It is odd that two commands are executed barely differing at all. There were some recent UI changes and I want to make sure that you are passing two inputs once and not submitting one input twice in the newly named "batch mode"?
container.sh is generated in command_factory.py using containers.py - here are some relevant links.
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/jobs/command_...
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/co...
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/do...
-John
I'm still trying to debug this.
How do I change the working directory? I can't seem to find out how all those directory mounts get passed to docker - the catDocker.xml appears to generate a 'cat' command that I assume ends up in ()/job_working_directory/000/12/container.sh but I can't find where that's happening - container.sh isn't listed in any file under galaxy-dist. job_conf.xml lists the directories that get passed as variables - again, when and where are those defined?
Thanks,
-Jeltje
On Fri, Jan 23, 2015 at 1:35 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
I tried the second solution (setting everything to rw) and while I can confirm that the command is changed accordingly, the results are the
same:
Two empty output files and the same error message.
Command from paster.log: docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:rw
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:rw
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:rw
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw
-w
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12
--net none busybox:ubuntu-14.04
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/container.sh;
return_code=$?; if [ -f
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file
] ; then cp
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_19.dat
; fi; sh -c "exit $return_code"
-Jeltje
On Fri, Jan 23, 2015 at 12:24 PM, John Chilton <jmchilton@gmail.com> wrote:
My first thought is the bug mentioned here -
https://lists.galaxyproject.org/pipermail/galaxy-dev/2014-November/020892.ht...
along with potential work arounds. Are you able to confirm that this is or is not the problem at all?
-John
On Fri, Jan 23, 2015 at 2:37 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
Hi,
I'm following instructions at https://github.com/apetkau/galaxy-hackathon-2014.
When I try to run the concatenate-datasets on two input files, two odd things happen. First, TWO nearly identical commands are generated in the History panel, only differing in their output filename. In
On Mon, Jan 26, 2015 at 2:17 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote: paster.log,
only the second one shows up. Second, the program fails, with this info in the History:
An error occurred with this dataset: Galaxy slots passed through contain as 1
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh:
line 2: can't create
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_14.dat
This happens for both commands in the History panel, only the output filename in the other error is dataset_15.dat
Oddly, the dataset_14.dat and dataset_15.dat are both created during this command, they just end up empty.
Paster.log:
galaxy.jobs.runners DEBUG 2015-01-23 11:08:35,973 (10) command is: docker inspect busybox:ubuntu-14.04 > /dev/null 2>&1 [ $? -ne 0 ] && docker pull busybox:ubuntu-14.04 > /dev/null 2>&1
docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:ro
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:ro
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:rw
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw
-w
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10
--net none busybox:ubuntu-14.04
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh;
return_code=$?; if [ -f
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file
] ; then cp
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_15.dat
; fi; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2015-01-23 11:08:36,040 (10) executing job script:
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/galaxy_10.sh
galaxy.jobs DEBUG 2015-01-23 11:08:36,091 (10) Persisting job destination (destination id: docker_local)
What could be happening here, and how do I fix it?
Thanks,
-Jeltje
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Our sysadmin just notified me docker was taking up large amounts of space in /var/lib/docker/devicemapper/devicemapper/data and /var/lib/docker/devicemapper/devicemapper/metadata I noticed the following DEBUG message in paster.log: galaxy.jobs.runners.local DEBUG 2015-01-23 13:30:19,503 (12) executing job script: /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/galaxy_12.sh galaxy.jobs DEBUG 2015-01-23 13:30:19,552 (12) Persisting job destination (destination id: docker_local) Are these related? (the files I'm trying to concatenate are small text files so that's not it) On Mon, Jan 26, 2015 at 11:47 AM, Jeltje van Baren < jeltje.van.baren@gmail.com> wrote:
See attached for screenshot!
Thanks for the links.
On Mon, Jan 26, 2015 at 11:37 AM, John Chilton <jmchilton@gmail.com> wrote:
Can you send me a screenshot of the Tool form right before you hit submit. It is odd that two commands are executed barely differing at all. There were some recent UI changes and I want to make sure that you are passing two inputs once and not submitting one input twice in the newly named "batch mode"?
container.sh is generated in command_factory.py using containers.py - here are some relevant links.
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/jobs/command_...
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/co...
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/do...
-John
I'm still trying to debug this.
How do I change the working directory? I can't seem to find out how all those directory mounts get passed to docker - the catDocker.xml appears to generate a 'cat' command that I assume ends up in ()/job_working_directory/000/12/container.sh but I can't find where that's happening - container.sh isn't listed in any file under galaxy-dist. job_conf.xml lists the directories that get passed as variables - again, when and where are those defined?
Thanks,
-Jeltje
On Fri, Jan 23, 2015 at 1:35 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
I tried the second solution (setting everything to rw) and while I can confirm that the command is changed accordingly, the results are the
same:
Two empty output files and the same error message.
Command from paster.log: docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:rw
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:rw
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:rw
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw
-w
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12
--net none busybox:ubuntu-14.04
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/container.sh;
return_code=$?; if [ -f
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file
] ; then cp
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_19.dat
; fi; sh -c "exit $return_code"
-Jeltje
On Fri, Jan 23, 2015 at 12:24 PM, John Chilton <jmchilton@gmail.com> wrote:
My first thought is the bug mentioned here -
https://lists.galaxyproject.org/pipermail/galaxy-dev/2014-November/020892.ht...
along with potential work arounds. Are you able to confirm that this is or is not the problem at all?
-John
On Fri, Jan 23, 2015 at 2:37 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
Hi,
I'm following instructions at https://github.com/apetkau/galaxy-hackathon-2014.
When I try to run the concatenate-datasets on two input files, two odd things happen. First, TWO nearly identical commands are generated in the History panel, only differing in their output filename. In
On Mon, Jan 26, 2015 at 2:17 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote: paster.log,
only the second one shows up. Second, the program fails, with this info in the History:
An error occurred with this dataset: Galaxy slots passed through contain as 1
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh:
line 2: can't create
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_14.dat
This happens for both commands in the History panel, only the output filename in the other error is dataset_15.dat
Oddly, the dataset_14.dat and dataset_15.dat are both created during this command, they just end up empty.
Paster.log:
galaxy.jobs.runners DEBUG 2015-01-23 11:08:35,973 (10) command is: docker inspect busybox:ubuntu-14.04 > /dev/null 2>&1 [ $? -ne 0 ] && docker pull busybox:ubuntu-14.04 > /dev/null 2>&1
docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:ro
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:ro
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:rw
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw
-w
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10
--net none busybox:ubuntu-14.04
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh;
return_code=$?; if [ -f
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file
] ; then cp
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_15.dat
; fi; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2015-01-23 11:08:36,040 (10) executing job script:
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/galaxy_10.sh
galaxy.jobs DEBUG 2015-01-23 11:08:36,091 (10) Persisting job destination (destination id: docker_local)
What could be happening here, and how do I fix it?
Thanks,
-Jeltje
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Based on the screenshot you are calling the program correctly. Can you tell me what version of Docker you are running and Galaxy you are running? There have been a number of fixes in both programs over the last six months to a year. Docker is known to consume a lot of space in /var - it probably doesn't have much to do with Galaxy or this tool right. It looks like you Docker command line does not include -rm - which was added recently to have Docker dispose of instances after it done with them. This might help with your /var space issue - though in general Docker does use a lot a relatively large amount of disk under /var - it is the nature of the technology and many traditional HPC clusters for instance may allocate relatively small /var partitions. Upgrading Galaxy might also help the permission thing - I don't see a '-u <uid>' argument in your command-line - which is another newer default to help ensure the resulting files have the correct permissions (especially when sudo is disabled - which it seems it is in your configuration). Hope this helps, -John On Mon, Jan 26, 2015 at 7:02 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
Our sysadmin just notified me docker was taking up large amounts of space in /var/lib/docker/devicemapper/devicemapper/data and /var/lib/docker/devicemapper/devicemapper/metadata
I noticed the following DEBUG message in paster.log:
galaxy.jobs.runners.local DEBUG 2015-01-23 13:30:19,503 (12) executing job script: /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/galaxy_12.sh galaxy.jobs DEBUG 2015-01-23 13:30:19,552 (12) Persisting job destination (destination id: docker_local)
Are these related? (the files I'm trying to concatenate are small text files so that's not it)
On Mon, Jan 26, 2015 at 11:47 AM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
See attached for screenshot!
Thanks for the links.
On Mon, Jan 26, 2015 at 11:37 AM, John Chilton <jmchilton@gmail.com> wrote:
Can you send me a screenshot of the Tool form right before you hit submit. It is odd that two commands are executed barely differing at all. There were some recent UI changes and I want to make sure that you are passing two inputs once and not submitting one input twice in the newly named "batch mode"?
container.sh is generated in command_factory.py using containers.py - here are some relevant links.
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/jobs/command_...
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/co...
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/do...
-John
On Mon, Jan 26, 2015 at 2:17 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
I'm still trying to debug this.
How do I change the working directory? I can't seem to find out how all those directory mounts get passed to docker - the catDocker.xml appears to generate a 'cat' command that I assume ends up in ()/job_working_directory/000/12/container.sh but I can't find where that's happening - container.sh isn't listed in any file under galaxy-dist. job_conf.xml lists the directories that get passed as variables - again, when and where are those defined?
Thanks,
-Jeltje
On Fri, Jan 23, 2015 at 1:35 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
I tried the second solution (setting everything to rw) and while I can confirm that the command is changed accordingly, the results are the same: Two empty output files and the same error message.
Command from paster.log: docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:rw -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:rw -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:rw -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw -w
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12 --net none busybox:ubuntu-14.04
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/container.sh; return_code=$?; if [ -f
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file ] ; then cp
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_19.dat ; fi; sh -c "exit $return_code"
-Jeltje
On Fri, Jan 23, 2015 at 12:24 PM, John Chilton <jmchilton@gmail.com> wrote:
My first thought is the bug mentioned here -
https://lists.galaxyproject.org/pipermail/galaxy-dev/2014-November/020892.ht... along with potential work arounds. Are you able to confirm that this is or is not the problem at all?
-John
On Fri, Jan 23, 2015 at 2:37 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote: > Hi, > > I'm following instructions at > https://github.com/apetkau/galaxy-hackathon-2014. > > When I try to run the concatenate-datasets on two input files, two > odd > things happen. First, TWO nearly identical commands are generated > in > the > History panel, only differing in their output filename. In > paster.log, > only > the second one shows up. Second, the program fails, with this info > in > the > History: > > An error occurred with this dataset: > Galaxy slots passed through contain as 1 > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh: > line 2: can't create > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_14.dat > > This happens for both commands in the History panel, only the > output > filename in the other error is dataset_15.dat > > Oddly, the dataset_14.dat and dataset_15.dat are both created > during > this > command, they just end up empty. > > Paster.log: > > galaxy.jobs.runners DEBUG 2015-01-23 11:08:35,973 (10) command is: > docker > inspect busybox:ubuntu-14.04 > /dev/null 2>&1 > [ $? -ne 0 ] && docker pull busybox:ubuntu-14.04 > /dev/null 2>&1 > > docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:ro > -v > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:ro > -v > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:rw > -v > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw > -w > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10 > --net none busybox:ubuntu-14.04 > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh; > return_code=$?; if [ -f > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file > ] ; then cp > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_15.dat > ; fi; sh -c "exit $return_code" > galaxy.jobs.runners.local DEBUG 2015-01-23 11:08:36,040 (10) > executing > job > script: > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/galaxy_10.sh > galaxy.jobs DEBUG 2015-01-23 11:08:36,091 (10) Persisting job > destination > (destination id: docker_local) > > > What could be happening here, and how do I fix it? > > Thanks, > > -Jeltje > > > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/
The docker version is 1.1.2. I'm not entirely sure how to get the galaxy version, but hg sum resulted in parent: 14955:c25df9f3d31e tip Update tag latest_2014.10.06 for changeset ff6e36d7a238 branch: stable So docker may be a bit too old. To get some more control over my OS, I did a fresh Ubuntu install on a separate laptop and got docker (1.41) and galaxy (downloaded stable release on Friday). Now I can get the docker image to run, but I STILL get two jobs in the history panel. One results in the concatenated files, the other has the text 'Work dir output' (and nothing else). Screenshot attached. I'll try to talk the sysadmins into upgrading docker and see if that fixes things. Thanks! -Jeltje On Fri, Jan 30, 2015 at 10:22 AM, John Chilton <jmchilton@gmail.com> wrote:
Based on the screenshot you are calling the program correctly.
Can you tell me what version of Docker you are running and Galaxy you are running? There have been a number of fixes in both programs over the last six months to a year. Docker is known to consume a lot of space in /var - it probably doesn't have much to do with Galaxy or this tool right. It looks like you Docker command line does not include -rm - which was added recently to have Docker dispose of instances after it done with them. This might help with your /var space issue - though in general Docker does use a lot a relatively large amount of disk under /var - it is the nature of the technology and many traditional HPC clusters for instance may allocate relatively small /var partitions.
Upgrading Galaxy might also help the permission thing - I don't see a '-u <uid>' argument in your command-line - which is another newer default to help ensure the resulting files have the correct permissions (especially when sudo is disabled - which it seems it is in your configuration).
Hope this helps,
-John
On Mon, Jan 26, 2015 at 7:02 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
Our sysadmin just notified me docker was taking up large amounts of space in /var/lib/docker/devicemapper/devicemapper/data and /var/lib/docker/devicemapper/devicemapper/metadata
I noticed the following DEBUG message in paster.log:
galaxy.jobs.runners.local DEBUG 2015-01-23 13:30:19,503 (12) executing job script:
galaxy.jobs DEBUG 2015-01-23 13:30:19,552 (12) Persisting job destination (destination id: docker_local)
Are these related? (the files I'm trying to concatenate are small text files so that's not it)
On Mon, Jan 26, 2015 at 11:47 AM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
See attached for screenshot!
Thanks for the links.
On Mon, Jan 26, 2015 at 11:37 AM, John Chilton <jmchilton@gmail.com> wrote:
Can you send me a screenshot of the Tool form right before you hit submit. It is odd that two commands are executed barely differing at all. There were some recent UI changes and I want to make sure that you are passing two inputs once and not submitting one input twice in the newly named "batch mode"?
container.sh is generated in command_factory.py using containers.py - here are some relevant links.
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/jobs/command_...
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/co...
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/do...
-John
On Mon, Jan 26, 2015 at 2:17 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
I'm still trying to debug this.
How do I change the working directory? I can't seem to find out how
all
those directory mounts get passed to docker - the catDocker.xml appears to generate a 'cat' command that I assume ends up in ()/job_working_directory/000/12/container.sh but I can't find where that's happening - container.sh isn't listed in any file under galaxy-dist. job_conf.xml lists the directories that get passed as variables - again, when and where are those defined?
Thanks,
-Jeltje
On Fri, Jan 23, 2015 at 1:35 PM, Jeltje van Baren <jeltje.van.baren@gmail.com> wrote:
I tried the second solution (setting everything to rw) and while I
can
confirm that the command is changed accordingly, the results are the same: Two empty output files and the same error message.
Command from paster.log: docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:rw
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:rw
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:rw
-v
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw
-w
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12
--net none busybox:ubuntu-14.04
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/container.sh;
return_code=$?; if [ -f
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file
] ; then cp
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_19.dat
; fi; sh -c "exit $return_code"
-Jeltje
On Fri, Jan 23, 2015 at 12:24 PM, John Chilton <jmchilton@gmail.com
wrote: > > My first thought is the bug mentioned here - > > > https://lists.galaxyproject.org/pipermail/galaxy-dev/2014-November/020892.ht... > along with potential work arounds. Are you able to confirm that
/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/galaxy_12.sh this
> is or is not the problem at all? > > -John > > On Fri, Jan 23, 2015 at 2:37 PM, Jeltje van Baren > <jeltje.van.baren@gmail.com> wrote: > > Hi, > > > > I'm following instructions at > > https://github.com/apetkau/galaxy-hackathon-2014. > > > > When I try to run the concatenate-datasets on two input files, two > > odd > > things happen. First, TWO nearly identical commands are generated > > in > > the > > History panel, only differing in their output filename. In > > paster.log, > > only > > the second one shows up. Second, the program fails, with this info > > in > > the > > History: > > > > An error occurred with this dataset: > > Galaxy slots passed through contain as 1 > > > > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh: > > line 2: can't create > > > > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_14.dat > > > > This happens for both commands in the History panel, only the > > output > > filename in the other error is dataset_15.dat > > > > Oddly, the dataset_14.dat and dataset_15.dat are both created > > during > > this > > command, they just end up empty. > > > > Paster.log: > > > > galaxy.jobs.runners DEBUG 2015-01-23 11:08:35,973 (10) command is: > > docker > > inspect busybox:ubuntu-14.04 > /dev/null 2>&1 > > [ $? -ne 0 ] && docker pull busybox:ubuntu-14.04 > /dev/null 2>&1 > > > > docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v > > > > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:ro > > -v > > > > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:ro > > -v > > > > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:rw > > -v > > > > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw > > -w > > > > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10 > > --net none busybox:ubuntu-14.04 > > > > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh; > > return_code=$?; if [ -f > > > > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file > > ] ; then cp > > > > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file > > > > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_15.dat > > ; fi; sh -c "exit $return_code" > > galaxy.jobs.runners.local DEBUG 2015-01-23 11:08:36,040 (10) > > executing > > job > > script: > > > > > > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/galaxy_10.sh > > galaxy.jobs DEBUG 2015-01-23 11:08:36,091 (10) Persisting job > > destination > > (destination id: docker_local) > > > > > > What could be happening here, and how do I fix it? > > > > Thanks, > > > > -Jeltje > > > > > > > > ___________________________________________________________ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > https://lists.galaxyproject.org/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/
participants (2)
-
Jeltje van Baren -
John Chilton