Re: [galaxy-dev] Trouble setting up a local instance of Galaxy

Hi Starr

I will try to answer some of your questions. Next time I recommend to ask just one question (with a corresponding subject line). This will make it much easier for the community to help you, and for others with similar problems to find the correct e-mail thread.

see below for my answers...

On 11/06/2013 11:43 PM, Hazard, E. Starr wrote:

Hello, I am a new user of Galaxy. I have a Galaxy instance running (sort of) on a local research cluster. I issued the command "hg update stable” today and it retrieved no files. SO presume I am up-to-date on the stable release. I start the instance as a user named “galaxy”. Right now I am still running in “local” mode. I hope to migrate to DRMAA LSF eventually. I have tried to set up ProFTP to upload files but have not succeeded so I use Galaxy Web-upload. The upload was working nicely and I had added a couple of new tools and they were working with the uploaded files. Getting LSF/DRMAA to work was giving me fits and ultimately I deleted all my history files in an effort to start over. Presently, files being uploaded appear in history as say job 1 ( in a new history) The job status in the history panel of the web GUI changes from purple to yellow then then to red indicating some sort of error. There is no viewable error text captured, but I can click on the “eye” icon and see the first megabyte of the data (for tiny files I can see the entire content and it’s intact). In the Galaxy file system, however, these files appear but have a different number , say, dataset_399.dat

On my system the uploaded files appear in /PATH/galaxy-dist/database/files/000

My first question is why is the data going into the “000” subdirectory and not one “owned’ by the user who is uploading?

all files are owned by galaxy (independent of who is uploading them and/or generate new files). The first 1000 files are stored in '000', the next 1000 files in '001', and so on.

Access permission is handled by the SQLite database (unless you have already switched to a PostgreSQL database).

My second question is why is the dataset being labeled as dataset_399.dat and not dataset_001.dat?

The number is given by the SQLite database. When you want a clean start, you need to remove all files from ~/database/files/000 and start with an empty database.

My third question is why do the uploaded files not appear as selectable options ( say I have paired-end fastq files and tool wants to have choices about filenames)? This problem is present for programs that seek one input file as well.

When you uploaded the files, was the 'format' properly recognized?

I presume that Galaxy is confused because the numbering in history is not the same as the numbering in the file upload archive (e.g. /PATH/galaxy-dist/database/files/000 in my case) so my last question is how do I “reset” my system to get the dataset and history numbers to be the same?

The numbering in the history has nothing to do with the file numbering. See above wrt my comment about a 'clean start'

Regards, Hans-Rudolf

Here’s how I launch the Galaxy instance

sh /shared/app/Galaxy/galaxy-dist/run.sh -v --daemon --pid-file=Nov6Localdaemon.pid.txt --log-file=Nov6Local1639daemon.log.txt

Entering daemon mode

Here are the last lines of the log

Starting server in PID 26236.

serving on 0.0.0.0:8089 view at http://127.0.0.1:8089

galaxy.tools.actions.upload_common DEBUG 2013-11-06 16:48:49,624 Changing ownership of /shared/app/Galaxy/galaxy-dist/database/tmp/upload_file_data_QZGHm4 with: /usr/bin/sudo -E scripts/external_chown_script.py /shared/app/Galaxy/galaxy-dist/database/tmp/upload_file_data_QZGHm4 hazards 502

galaxy.tools.actions.upload_common WARNING 2013-11-06 16:48:49,750 Changing ownership of uploaded file /shared/app/Galaxy/galaxy-dist/database/tmp/upload_file_data_QZGHm4 failed: sudo: no tty present and no askpass program specified

galaxy.tools.actions.upload_common DEBUG 2013-11-06 16:48:49,751 Changing ownership of /shared/app/Galaxy/galaxy-dist/database/tmp/tmpEsyGfO with: /usr/bin/sudo -E scripts/external_chown_script.py /shared/app/Galaxy/galaxy-dist/database/tmp/tmpEsyGfO hazards 502

galaxy.tools.actions.upload_common WARNING 2013-11-06 16:48:49,775 Changing ownership of uploaded file /shared/app/Galaxy/galaxy-dist/database/tmp/tmpEsyGfO failed: sudo: no tty present and no askpass program specified

galaxy.tools.actions.upload_common INFO 2013-11-06 16:48:49,805 tool upload1 created job id 170

galaxy.jobs DEBUG 2013-11-06 16:48:50,678 (170) Persisting job destination (destination id: local)

galaxy.jobs.handler INFO 2013-11-06 16:48:50,698 (170) Job dispatched

galaxy.jobs.runners.local DEBUG 2013-11-06 16:48:50,994 (170) executing: python /shared/app/Galaxy/galaxy-dist/tools/data_source/upload.py /depot/shared/app/Galaxy/galaxy-dist /shared/app/Galaxy/galaxy-dist/database/tmp/tmpTq22ot /shared/app/Galaxy/galaxy-dist/database/tmp/tmpEsyGfO 406:/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/dataset_406_files:/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/galaxy_dataset_406.dat

galaxy.jobs DEBUG 2013-11-06 16:48:51,030 (170) Persisting job destination (destination id: local)

galaxy.jobs.runners.local DEBUG 2013-11-06 16:48:53,335 execution finished: python /shared/app/Galaxy/galaxy-dist/tools/data_source/upload.py /depot/shared/app/Galaxy/galaxy-dist /shared/app/Galaxy/galaxy-dist/database/tmp/tmpTq22ot /shared/app/Galaxy/galaxy-dist/database/tmp/tmpEsyGfO 406:/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/dataset_406_files:/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/galaxy_dataset_406.dat

galaxy.jobs.runners.local DEBUG 2013-11-06 16:48:53,463 executing external set_meta script for job 170: /depot/shared/app/Galaxy/galaxy-dist/set_metadata.sh /shared/app/Galaxy/galaxy-dist/database/files /shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170 . /depot/shared/app/Galaxy/galaxy-dist/universe_wsgi.ini /shared/app/Galaxy/galaxy-dist/database/tmp/tmpTq22ot /shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/galaxy.json /shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/metadata_in_HistoryDatasetAssociation_410_pUkPU9,/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/metadata_kwds_HistoryDatasetAssociation_410_iouZ6k,/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/metadata_out_HistoryDatasetAssociation_410_3c6Nj2,/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/metadata_results_HistoryDatasetAssociation_410_A6L2LO,/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/galaxy_dataset_406.dat,/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/metadata_override_HistoryDatasetAssociation_410_8Wub22

galaxy.jobs.runners.local DEBUG 2013-11-06 16:48:54,789 execution of external set_meta for job 170 finished

galaxy.jobs DEBUG 2013-11-06 16:48:54,815 (170) Changing ownership of working directory with: /usr/bin/sudo -E scripts/external_chown_script.py /shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170 galaxy 50982

galaxy.jobs ERROR 2013-11-06 16:48:54,839 (170) Failed to change ownership of /shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170, failing

Traceback (most recent call last):

File "/depot/shared/app/Galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 868, in finish

 self.reclaim_ownership()

File "/depot/shared/app/Galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 1452, in reclaim_ownership

 self._change_ownership( self.galaxy_system_pwent[0], str(

self.galaxy_system_pwent[3] ) )

File "/depot/shared/app/Galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 1438, in _change_ownership

 assert p.returncode == 0

AssertionError

galaxy.jobs DEBUG 2013-11-06 16:48:54,876 fail(): Moved /shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/galaxy_dataset_406.dat to /shared/app/Galaxy/galaxy-dist/database/files/000/dataset_406.dat

ls -ltr /shared/app/Galaxy/galaxy-dist/database/files/000

-rw------- 1 galaxy galaxy 93 Nov6 16:48 dataset_406.dat

From http://dev.list.galaxyproject.org/Submitting-jobs-as-a-real-user-without-usi...

I tried this as sudo user

scripts/external_chown_script.py /shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170 svcgalaxy 100000

Traceback (most recent call last):

File "scripts/external_chown_script.py", line 16, in <module>

 import drmaa

File "build/bdist.linux-x86_64/egg/drmaa/__init__.py", line 41, in

<module>

File "build/bdist.linux-x86_64/egg/drmaa/wrappers.py", line 43, in

<module>

RuntimeError: could not find drmaa library. Please specify its full path using the environment variable DRMAA_LIBRARY_PATH

I have no idea why that refers to DRMAA anything because in this case I was running local In fact however I get that same error when I start galaxy with this ”job_conf.xml”

cat job_conf.xml

<?xml version="1.0"?>

<job_conf>

 <plugins>

     <!-- "workers" is the number of threads for the runner's work

queue.

          The default from <plugins> is used if not defined for a

<plugin>.

       -->

     <plugin id="drmaa" type="runner"

load="galaxy.jobs.runners.drmaa:DRMAAJobRunner" workers="8"/>

 </plugins>

  <handlers default="handlers">

          <!-- Additional job handlers - the id should match the

name of a

                            [server:<id>] in universe_wsgi.ini.

                                     -->

     <handler id="main" tags="handlers"/>

   </handlers>

 <destinations default="drmaa">

     <!-- Destinations define details about remote resources and how

jobs

          should be executed on those remote resources.

      -->

     <destination id="drmaa" runner="drmaa" tags="hpcc3.musc.edu">

         <!-- "TODO: The real user options should maybe not be

considered runner params."

          -->

         <param

id="galaxy_external_runjob_script">scripts/drmaa_external_runner.py</param>

         <param

id="galaxy_external_killjob_script">scripts/drmaa_external_killer.py</param>

         <param

id="galaxy_external_chown_script">scripts/external_chown_script.py</param>

     </destination>

 </destinations>

</job_conf>

The other difference is that in the DRMAA case, the GUI upload job in the HISTORY panel goes “grey” and stays that way; nothing is ever uploaded.

Summary: In local mode I can upload files but there are errors in HISTORY and I cannot get tools to see the uploaded files. In DMAA mode I cannot get the GUI upload to happen at all.

BTW in user “galaxy”’s .bashrc file I have this line

export DRMAA_LIBRARY_PATH=/opt/lsf/7.0/linux2.6-glibc2.3-x86_64/lib/libdrmaa.so

I’m baffled.

Thanks

Starr

/shared/app/Galaxy/galaxy-dist/database/files

---

Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/