September 2012 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Using $NSLOTS in tools to control thread number
by Peter Cock 19 Oct '12

19 Oct '12

On Fri, Jun 15, 2012 at 4:38 PM, James Taylor <james(a)jamestaylor.org> wrote: > This is exactly what I think we should do (and have for a long time), but I > think the variable should be something like: > > GALAXY_CPUS > > (threads is not accurate, a multithread or multiprocess job might want to use > this info, something even more abstract than CPUS might make sense, but > SLOTS has never made sense to me). I agree that a Galaxy specific name makes a lot of sense, and that the SGE term "slots" is a bit odd. Using CPUS however is potentially ambiguous with CPUs vs cores - my desktop has two quad core CPUs, i.e. 2 CPUs but 8 cores. Where do you think this number should come from? A new entry in the runner URL is simple albeit potentially redundant with cluster-specific entries in the runner URL. As to the alternative (doing it automatically), for PBS and SGE determining the number of cores from the cluster configuration and/or parsing the cluster runner URL sounds doable - what about the other backends? Peter

2 2

fastQC report bug
by David Hoover 12 Oct '12

12 Oct '12

Hi, I have a very recent version of Galaxy installed locally, and when I run FastQC:Read QC, it creates an HTML file with missing links to the plots and images within the HTML. The HTML source looks identical to the HTML source from the main Galaxy server, so there must be something wrong with my local install. Is there some option or modification that I need to enable to allow access to the generated HTML/PNG files in the database/files/000/database_###_files directory? David Hoover Helix Systems Staff

1 1

Issues up/downloading datasets after file_path change
by Josh Nielsen 04 Oct '12

04 Oct '12

Hello all, I recently migrated the location of the Galaxy file_path directory (~/database/files/) along with the temp, job_working_directory, and pbs directory locations (all under ~/database/) to a new storage system mount point and I properly updated the change in the universe_wsgi.ini file but have run into some problems with managing our datasets. I can run Galaxy jobs just fine but I cannot view or download the datasets in the output of the jobs after they have run - when it tries to open it from the new path - and I get an actual browser error when trying to fetch the dataset. On a hunch I created a symbolic link named after the old path in the / directory pointing to the new location (and pointed file_path in the universe_wsgi.ini file back to it - which looks exactly like the old directory path), and sure enough this works as a (temporary) fix to view/download the files. However doing that seems to, in turn, mess with the file uploading function which will fail every time with a "OSError: [Errno 2] No such file or directory" message. So I'm caught between two broken configurations depending on the path I point to (the new path or the symlink alternative). I guess in the latter situation it does not like traversing a symbolic link (which is the only explanation that I can think of). The root of the problem needs to be fixed though in that Galaxy seems to be "remembering" the old path from somewhere (the Galaxy database?) such that it will not let me simply migrate the data to a new location and change the path variables in universe_wsgi.ini. Any suggestions on the proper changes that need to be made to fix this permanently? Thanks, Josh Nielsen

2 3

Improved error logging in TaskWrapper
by Peter Cock 03 Oct '12

03 Oct '12

Hi all, I've been running into some sporadic errors on our Cluster while using the latest development Galaxy, and the error handling has made this quite difficult to diagnose. For a user perspective, the jobs seem to run, get submitted to the cluster, and finish, and the data looks OK via the 'eye' view icon, but is red in the history with: 0 bytes An error occurred running this job: info unavailable Furthermore, the stdout and stderr via the 'info' icon are blank. For watching the log (and adding more diagnosis lines), what is happening is the job is being split and sent out to the cluster fine, and starts running. If one of the tasks fails (and this seems to be happening due to some sort of file system error on our cluster), Galaxy spots this, and kills the rest of the jobs. That's good. The problem is it fails to record any record of why the job died. This is my suggestion for now - it would be nice to go further and fill the info text show in the history peep as well?: $ hg diff diff -r 4de1d566e9f8 lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py Fri Sep 21 11:02:50 2012 +0100 +++ b/lib/galaxy/jobs/__init__.py Fri Sep 21 11:59:27 2012 +0100 @@ -1061,6 +1061,14 @@ log.error( "stderr for job %d is greater than 32K, only first part will be logged to database" % task.id ) task.stderr = stderr[:32768] task.command_line = self.command_line + + if task.state == task.states.ERROR: + # If failed, will kill the other tasks in this job. Record this + # task's stdout/stderr as should be useful to explain the failure: + job = self.get_job() + job.stdout = ("(From one sub-task:)\n" +task.stdout)[:32768] + job.stderr = ("(From one sub-task:)\n" +task.stderr)[:32768] + self.sa_session.flush() log.debug( 'task %d ended' % self.task_id ) Regards, Peter

2 2

Error when running cleanup_datasets.py
by Liisa Koski 02 Oct '12

02 Oct '12

Hello, I am trying to run the cleanup scripts on my local installation but get stuck when trying to run the following: ./scripts/cleanup_datasets/cleanup_datasets.py universe_wsgi.ini -d 10 -5 -r Deleting library dataset id 7225 Traceback (most recent call last): File "./scripts/cleanup_datasets/cleanup_datasets.py", line 524, in <module> if __name__ == "__main__": main() File "./scripts/cleanup_datasets/cleanup_datasets.py", line 124, in main purge_folders( app, cutoff_time, options.remove_from_disk, info_only = options.info_only, force_retry = options.force_retry ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 247, in purge_folders _purge_folder( folder, app, remove_from_disk, info_only = info_only ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 497, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 497, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 495, in _purge_folder _purge_dataset_instance( ldda, app, remove_from_disk, info_only = info_only ) #mark a DatasetInstance as deleted, clear associated files, and mark the Dataset as deleted if it is deletable File "./scripts/cleanup_datasets/cleanup_datasets.py", line 376, in _purge_dataset_instance ( dataset_instance.__class__.__name__, dataset_instance.id, dataset_instance.dataset.id ) AttributeError: 'NoneType' object has no attribute 'id' Any help would be much appreciated. Thanks, Liisa

2 2

LDAP integration and migration of users
by Joachim Jacob 02 Oct '12

02 Oct '12

Hi all, I wonder if somebody with experience could enlighten me on these issues regarding authentication of users. - is it possible to have LDAP integration simultaneously with local Galaxy users. What I want to reach is when LDAP integration is working, that I can still add users through the Galaxy admin interface (users that are not present in the LDAP). - is there a way to transfer all histories, workflows and visualisation to another user? For example, an existing user receives a new email and password, and want to transfer all his/her stuff to this new account. Can this be done? I was thinking about when a user gets transferred to the LDAP, he/she would be able to transfer all data from the old account to the new account. Thanks, Joachim -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib

2 2

Orphan dataset created by history Export to File (possibly a bug?)
by Derrick Lin 02 Oct '12

02 Oct '12

hi guys, I have been trying to clean up some old histories and datasets on our local galaxy. I have spotted one very old dataset file with big size didn't get removed no matter what. I went to check the reports webapp, that dataset isn't on the largest unpurged data files list (despite it has bigger size). Then I moved on to checking the database, no history or library associates with that dataset. Finally, I found its trace in job_to_input_dataset, so I could identified the job id and found that it's the file created by history Export to File. I also find relevant entry in job_export_history_archive. Since such datasets are not associated with any history, library, the documented dataset cleanup method does not work on them. Any suggestion? Regards, Derrick

4 6

How to rotate Galaxy log file
by Jennifer Jackson 01 Oct '12

01 Oct '12

repost to galaxy-dev On 9/7/12 6:39 PM, Lukasz Lacinski wrote: > Dear All, > > I use an init script that comes with Galaxy in the contrib/ subdirectory > to start Galaxy. The log file > > --log-file /home/galaxy/galaxy.log > > specified in the script grows really quickly. How to logrotate the file? > > Thanks, > Lukasz > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ -- Jennifer Jackson http://galaxyproject.org

8 10

speed and large file question (after tying galaxy to Pittsburgh)
by Joseph Hargitai 01 Oct '12

01 Oct '12

Hi, General: Is there a quantifiable difference in data uploads, speed of processing, general availability after the connection was/is established to Pittsburgh? Specific: If we loaded data to a common directory on an XSEDE resource, would our students be able to fetch this data in a more timely manner to their galaxy accounts then from desktops etc... For us it would be easy to populate a common directory once then our users could fetch/wget etc... from there, or if possible could such directory be directly visible to our set of galaxy users? (this is what we do locally) - but one would need admin priv for that on the main instance. Or would you just do this with one galaxy user sharing out the files for the rest of a group working together? thanks. joe

2 1

Size of the genome indexes repos in Galaxy Main
by Derrick Lin 01 Oct '12

01 Oct '12

Hi guys, Just a quick question about the total size of the genome indexes just made available via rsync server? Cheers, Derrick

2 4