Getting example_watch_folder.py to work...
Hi, I'm trying to get the example_watch_folder.py to run but it seems to fail, and I'm not sure why? I run: ./example_watch_folder.py 64f3209856a3cf4f2d034a1ad5bf851c http://barium-rbh/csiro/api/ /home/galaxy/galaxy-drop/input /home/galaxy/galaxy-drop/output "My API Import" f597429621d6eb2b I got the workflow Id from : http://barium-rbh/csiro/api/workflows which gave me: [ { "id": "f597429621d6eb2b", "name": "extract", "url": "/csiro/api/workflows/f597429621d6eb2b" }, { "id": "f2db41e1fa331b3e", "name": "FULL CTE", "url": "/csiro/api/workflows/f2db41e1fa331b3e" } ] The output I get from the command line is: {'outputs': ['ba0fa2aed4052bce'], 'history': 'ba03619785539f8c'} The files I put into /home/galaxy/galaxy-drop/input get copied to /home/galaxy/galaxy-drop/output But nothing else happens. If I go to http://barium-rbh/csiro/api/histories I can see: { "id": "ba03619785539f8c", "name": "colin.nii.gz - extract", "url": "/csiro/api/histories/ba03619785539f8c" }, However when I go to: http://barium-rbh/csiro/api/histories/ba03619785539f8c I get: { "contents_url": "/csiro/api/histories/ba03619785539f8c/contents", "id": "ba03619785539f8c", "name": "colin.nii.gz - extract", "state": "error", "state_details": { "discarded": 0, "empty": 0, "error": 1, "failed_metadata": 0, "new": 0, "ok": 0, "queued": 0, "running": 0, "setting_metadata": 0, "upload": 0 } } Any ideas much appreciated Thanks Neil
Further, it seems that it doesn't manage to get hold of the file specified in the input directory as I can see from the output: http://barium-rbh/csiro/api/histories/964b37715ec9bd22/contents/2faba7054d92... { "data_type": "html", "deleted": false, "download_url": "/csiro/datasets/2faba7054d92b2df/display?to_ext=html", "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_137.dat", "file_size": 194, "genome_build": "?", "id": "2faba7054d92b2df", "metadata_data_lines": null, "metadata_dbkey": "?", "misc_blurb": "error", "misc_info": "Wed May 8 15:07:27 2013\nbashScript is: /home/galaxy/galaxy-dist/tools/visualization/extractSlice-wrapper.sh\ninput_image is: None\ncat: None: No such file or directory\nFailed reading file /tmp/tmp.9LWrz6SaLy.nii.gz\nitk::ERROR: PNGImageIO(0x1abdcf0): PNGIma", "model_class": "HistoryDatasetAssociation", "name": "Extract 2D slice on None", "state": "error", "visible": true } input_image is: None Does anybody know why the file may not be getting read? It is being copied from the specified input directory to the output directory. I have set : allow_library_path_paste = True and added my user to: admin_users Thanks for any help Neil ________________________________ From: Burdett, Neil (ICT Centre, Herston - RBWH) Sent: Wednesday, May 08, 2013 2:46 PM To: galaxy-dev@lists.bx.psu.edu Subject: Getting example_watch_folder.py to work... Hi, I'm trying to get the example_watch_folder.py to run but it seems to fail, and I'm not sure why? I run: ./example_watch_folder.py 64f3209856a3cf4f2d034a1ad5bf851c http://barium-rbh/csiro/api/ /home/galaxy/galaxy-drop/input /home/galaxy/galaxy-drop/output "My API Import" f597429621d6eb2b I got the workflow Id from : http://barium-rbh/csiro/api/workflows which gave me: [ { "id": "f597429621d6eb2b", "name": "extract", "url": "/csiro/api/workflows/f597429621d6eb2b" }, { "id": "f2db41e1fa331b3e", "name": "FULL CTE", "url": "/csiro/api/workflows/f2db41e1fa331b3e" } ] The output I get from the command line is: {'outputs': ['ba0fa2aed4052bce'], 'history': 'ba03619785539f8c'} The files I put into /home/galaxy/galaxy-drop/input get copied to /home/galaxy/galaxy-drop/output But nothing else happens. If I go to http://barium-rbh/csiro/api/histories I can see: { "id": "ba03619785539f8c", "name": "colin.nii.gz - extract", "url": "/csiro/api/histories/ba03619785539f8c" }, However when I go to: http://barium-rbh/csiro/api/histories/ba03619785539f8c I get: { "contents_url": "/csiro/api/histories/ba03619785539f8c/contents", "id": "ba03619785539f8c", "name": "colin.nii.gz - extract", "state": "error", "state_details": { "discarded": 0, "empty": 0, "error": 1, "failed_metadata": 0, "new": 0, "ok": 0, "queued": 0, "running": 0, "setting_metadata": 0, "upload": 0 } } Any ideas much appreciated Thanks Neil
It looks like the problem may have to do with the file I am using ??? It is a *.nii.gz file. Does galaxy try to uncompress ? do something to *.gz files when using the api (as it tries to do in ~/galaxy-dist/tools/data_source/upload.py Does anyone know of where in the code I can edit so my mile can be uploaded? Thanks Neil From: Burdett, Neil (ICT Centre, Herston - RBWH) Sent: Wednesday, 8 May 2013 3:15 PM To: galaxy-dev@lists.bx.psu.edu Subject: RE: Getting example_watch_folder.py to work... Further, it seems that it doesn't manage to get hold of the file specified in the input directory as I can see from the output: http://barium-rbh/csiro/api/histories/964b37715ec9bd22/contents/2faba7054d92... { "data_type": "html", "deleted": false, "download_url": "/csiro/datasets/2faba7054d92b2df/display?to_ext=html", "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_137.dat", "file_size": 194, "genome_build": "?", "id": "2faba7054d92b2df", "metadata_data_lines": null, "metadata_dbkey": "?", "misc_blurb": "error", "misc_info": "Wed May 8 15:07:27 2013\nbashScript is: /home/galaxy/galaxy-dist/tools/visualization/extractSlice-wrapper.sh\ninput_image is: None\ncat: None: No such file or directory\nFailed reading file /tmp/tmp.9LWrz6SaLy.nii.gz\nitk::ERROR: PNGImageIO(0x1abdcf0): PNGIma", "model_class": "HistoryDatasetAssociation", "name": "Extract 2D slice on None", "state": "error", "visible": true } input_image is: None Does anybody know why the file may not be getting read? It is being copied from the specified input directory to the output directory. I have set : allow_library_path_paste = True and added my user to: admin_users Thanks for any help Neil ________________________________ From: Burdett, Neil (ICT Centre, Herston - RBWH) Sent: Wednesday, May 08, 2013 2:46 PM To: galaxy-dev@lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu> Subject: Getting example_watch_folder.py to work... Hi, I'm trying to get the example_watch_folder.py to run but it seems to fail, and I'm not sure why? I run: ./example_watch_folder.py 64f3209856a3cf4f2d034a1ad5bf851c http://barium-rbh/csiro/api/ /home/galaxy/galaxy-drop/input /home/galaxy/galaxy-drop/output "My API Import" f597429621d6eb2b I got the workflow Id from : http://barium-rbh/csiro/api/workflows which gave me: [ { "id": "f597429621d6eb2b", "name": "extract", "url": "/csiro/api/workflows/f597429621d6eb2b" }, { "id": "f2db41e1fa331b3e", "name": "FULL CTE", "url": "/csiro/api/workflows/f2db41e1fa331b3e" } ] The output I get from the command line is: {'outputs': ['ba0fa2aed4052bce'], 'history': 'ba03619785539f8c'} The files I put into /home/galaxy/galaxy-drop/input get copied to /home/galaxy/galaxy-drop/output But nothing else happens. If I go to http://barium-rbh/csiro/api/histories I can see: { "id": "ba03619785539f8c", "name": "colin.nii.gz - extract", "url": "/csiro/api/histories/ba03619785539f8c" }, However when I go to: http://barium-rbh/csiro/api/histories/ba03619785539f8c I get: { "contents_url": "/csiro/api/histories/ba03619785539f8c/contents", "id": "ba03619785539f8c", "name": "colin.nii.gz - extract", "state": "error", "state_details": { "discarded": 0, "empty": 0, "error": 1, "failed_metadata": 0, "new": 0, "ok": 0, "queued": 0, "running": 0, "setting_metadata": 0, "upload": 0 } } Any ideas much appreciated Thanks Neil
Hi, I'm still struggling to get the example_watch_folder.py to work. So any help much appreciated. I've created a simple workflow, which essentially opens a text file and then writes out the data to a html file. My xml file: <tool id="CopyTool" name="Copy Tool"> <description>Test</description> <command interpreter="perl">$__root_dir__/tools/copy/copy.pl --input_image "$inputImage" --output_html "$output" </command> <inputs> <param format="txt" name="inputImage" type="data" label="Input Image" /> </inputs> <outputs> <data format="html" name="output" label="${tool.name} on #echo os.path.basename( str ( $inputImage.name ) )#" /> </outputs> <help> Copy Tool </help> </tool> And copy.pl: #!/usr/bin/perl use strict; use warnings; use Getopt::Long; my $input_image ; my $output_html ; # Get options from command line (i.e. galaxy) my $result = GetOptions ( "input_image=s" => \$input_image, "output_html=s" => \$output_html ) or die "Bad options"; print "input_image is: $input_image\n"; open FILE,"$input_image" or die $!; my @lines = <FILE>; close FILE; my $numOfLines = scalar @lines; # Create the output HTML, with links to the files and the square gif. # open HTML, ">", $output_html or die "Failed to create output HTML file '$output_html': $! "; print HTML<<EOF; <html> <head> <style> iframe { border: 0px; background: #eeeeee ; } </style> </head> <body> <h1>Copy Files tool</h1> <h2>Generated $numOfLines files</h2> EOF # Put direct links to each output file foreach my $sub_filename ( @lines ) { print HTML "Direct link to the $sub_filename file.<br/>\n"; } print HTML "<br/><br/><br/>\n"; print HTML "</body></html>\n" ; close HTML ; So pretty basic. I run: ./example_watch_folder.py cce1b01926646d548f6ddc32ff01aa2e http://140.253.78.234/galaxy/api/ /home/galaxy/data_input /home/galaxy/data_output "My API Import" f2db41e1fa331b3e and get the following output: {'outputs': ['a799d38679e985db'], 'history': '33b43b4e7093c91f'} I can see the sample.txt file I placed in the /home/galaxy/data_input has been put into the database: /home/galaxy/galaxy-dist/database/files/000/dataset_8.dat and: http://140.253.78.234/galaxy/api/libraries/f2db41e1fa331b3e/contents/1cd8e2f... shows the file has been uploaded: { "data_type": "txt", "date_uploaded": "2013-05-09T04:41:10.579602", "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_8.dat", "file_size": 76, "genome_build": "?", "id": "1cd8e2f6b131e891", "ldda_id": "1cd8e2f6b131e891", "message": "", "metadata_data_lines": 8, "metadata_dbkey": "?", "misc_blurb": "8 lines", "misc_info": "uploaded txt file", "model_class": "LibraryDataset", "name": "sam2.txt", "template_data": {}, "uploaded_by": "test@test.com", "uuid": null } However, looking in the histories: http://140.253.78.234/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38... The output file is zero as the input file can not be found as shown below in bold ... { "accessible": true, "api_type": "file", "data_type": "html", "deleted": false, "display_apps": [], "display_types": [], "download_url": "/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38679e985db/display", "file_ext": "html", "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_9.dat", "file_size": 0, "genome_build": "?", "hid": 1, "history_id": "33b43b4e7093c91f", "id": "a799d38679e985db", "metadata_data_lines": null, "metadata_dbkey": "?", "misc_blurb": "error", "misc_info": "Thu May 9 14:42:10 2013input_image is: None\nNo such file or directory at /home/galaxy/galaxy-dist/tools/copy/copy.pl line 27.\n", "model_class": "HistoryDatasetAssociation", "name": "Copy Tool on None", "peek": null, "purged": false, "state": "error", "uuid": null, "visible": true, "visualizations": [] } Does anyone have any idea why the input_file is not passed in/obtained in the perl script? as the script obviously copies it to the database so part of the script is working? the file sam2.txt looks like this Thanks again, sorry for swamping the list with this issue Neil
Hi all, I finally resolved the issue I'll mention it here to help anyone else who has the same issue. In "Edit" mode on the workflow you add the "Workflow control->Inputs" to use the api. Final question.... Everything gets created and stored in the database, but the history column on the left of the browser doesn't get updated. Can user see if the task has completed successfully and all files uploaded from the browser? Thanks Neil From: Burdett, Neil (ICT Centre, Herston - RBWH) Sent: Thursday, 9 May 2013 2:56 PM To: galaxy-dev@lists.bx.psu.edu Subject: Getting example_watch_folder.py to work... Hi, I'm still struggling to get the example_watch_folder.py to work. So any help much appreciated. I've created a simple workflow, which essentially opens a text file and then writes out the data to a html file. My xml file: <tool id="CopyTool" name="Copy Tool"> <description>Test</description> <command interpreter="perl">$__root_dir__/tools/copy/copy.pl --input_image "$inputImage" --output_html "$output" </command> <inputs> <param format="txt" name="inputImage" type="data" label="Input Image" /> </inputs> <outputs> <data format="html" name="output" label="${tool.name} on #echo os.path.basename( str ( $inputImage.name ) )#" /> </outputs> <help> Copy Tool </help> </tool> And copy.pl: #!/usr/bin/perl use strict; use warnings; use Getopt::Long; my $input_image ; my $output_html ; # Get options from command line (i.e. galaxy) my $result = GetOptions ( "input_image=s" => \$input_image, "output_html=s" => \$output_html ) or die "Bad options"; print "input_image is: $input_image\n"; open FILE,"$input_image" or die $!; my @lines = <FILE>; close FILE; my $numOfLines = scalar @lines; # Create the output HTML, with links to the files and the square gif. # open HTML, ">", $output_html or die "Failed to create output HTML file '$output_html': $! "; print HTML<<EOF; <html> <head> <style> iframe { border: 0px; background: #eeeeee ; } </style> </head> <body> <h1>Copy Files tool</h1> <h2>Generated $numOfLines files</h2> EOF # Put direct links to each output file foreach my $sub_filename ( @lines ) { print HTML "Direct link to the $sub_filename file.<br/>\n"; } print HTML "<br/><br/><br/>\n"; print HTML "</body></html>\n" ; close HTML ; So pretty basic. I run: ./example_watch_folder.py cce1b01926646d548f6ddc32ff01aa2e http://140.253.78.234/galaxy/api/ /home/galaxy/data_input /home/galaxy/data_output "My API Import" f2db41e1fa331b3e and get the following output: {'outputs': ['a799d38679e985db'], 'history': '33b43b4e7093c91f'} I can see the sample.txt file I placed in the /home/galaxy/data_input has been put into the database: /home/galaxy/galaxy-dist/database/files/000/dataset_8.dat and: http://140.253.78.234/galaxy/api/libraries/f2db41e1fa331b3e/contents/1cd8e2f... shows the file has been uploaded: { "data_type": "txt", "date_uploaded": "2013-05-09T04:41:10.579602", "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_8.dat", "file_size": 76, "genome_build": "?", "id": "1cd8e2f6b131e891", "ldda_id": "1cd8e2f6b131e891", "message": "", "metadata_data_lines": 8, "metadata_dbkey": "?", "misc_blurb": "8 lines", "misc_info": "uploaded txt file", "model_class": "LibraryDataset", "name": "sam2.txt", "template_data": {}, "uploaded_by": "test@test.com<mailto:test@test.com>", "uuid": null } However, looking in the histories: http://140.253.78.234/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38... The output file is zero as the input file can not be found as shown below in bold ... { "accessible": true, "api_type": "file", "data_type": "html", "deleted": false, "display_apps": [], "display_types": [], "download_url": "/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38679e985db/display", "file_ext": "html", "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_9.dat", "file_size": 0, "genome_build": "?", "hid": 1, "history_id": "33b43b4e7093c91f", "id": "a799d38679e985db", "metadata_data_lines": null, "metadata_dbkey": "?", "misc_blurb": "error", "misc_info": "Thu May 9 14:42:10 2013input_image is: None\nNo such file or directory at /home/galaxy/galaxy-dist/tools/copy/copy.pl line 27.\n", "model_class": "HistoryDatasetAssociation", "name": "Copy Tool on None", "peek": null, "purged": false, "state": "error", "uuid": null, "visible": true, "visualizations": [] } Does anyone have any idea why the input_file is not passed in/obtained in the perl script? as the script obviously copies it to the database so part of the script is working? the file sam2.txt looks like this Thanks again, sorry for swamping the list with this issue Neil
Hey Neil, A user browsing that history wouldn't automatically get the updates (unless the history was already refreshing for some reason), but they definitely would upon refresh. Glad you got the rest working! -Dannon On Thu, May 9, 2013 at 2:02 AM, <Neil.Burdett@csiro.au> wrote:
Hi all,****
I finally resolved the issue I’ll mention it here to help anyone else who has the same issue. In “Edit” mode on the workflow you add the “Workflow control->Inputs” to use the api.****
** **
Final question....****
Everything gets created and stored in the database, but the history column on the left of the browser doesn’t get updated. Can user see if the task has completed successfully and all files uploaded from the browser?****
** **
Thanks****
Neil****
** **
*From:* Burdett, Neil (ICT Centre, Herston - RBWH) *Sent:* Thursday, 9 May 2013 2:56 PM
*To:* galaxy-dev@lists.bx.psu.edu *Subject:* Getting example_watch_folder.py to work...****
** **
Hi,
I'm still struggling to get the example_watch_folder.py to work. So any help much appreciated.
I've created a simple workflow, which essentially opens a text file and then writes out the data to a html file.
My xml file: <tool id="CopyTool" name="Copy Tool"> <description>Test</description> <command interpreter="perl">$__root_dir__/tools/copy/copy.pl --input_image "$inputImage" --output_html "$output" </command>
<inputs> <param format="txt" name="inputImage" type="data" label="Input Image" /> </inputs> <outputs> <data format="html" name="output" label="${tool.name} on #echo os.path.basename( str ( $inputImage.name ) )#" /> </outputs> <help> Copy Tool
</help> </tool>
And copy.pl: #!/usr/bin/perl use strict; use warnings; use Getopt::Long; my $input_image ; my $output_html ;
# Get options from command line (i.e. galaxy) my $result = GetOptions ( "input_image=s" => \$input_image, "output_html=s" => \$output_html ) or die "Bad options";
print "input_image is: $input_image\n";
open FILE,"$input_image" or die $!; my @lines = <FILE>; close FILE; my $numOfLines = scalar @lines;
# Create the output HTML, with links to the files and the square gif. # open HTML, ">", $output_html or die "Failed to create output HTML file '$output_html': $! ";
print HTML<<EOF; <html> <head> <style> iframe { border: 0px; background: #eeeeee ; } </style> </head> <body> <h1>Copy Files tool</h1> <h2>Generated $numOfLines files</h2> EOF
# Put direct links to each output file foreach my $sub_filename ( @lines ) { print HTML "Direct link to the $sub_filename file.<br/>\n"; }
print HTML "<br/><br/><br/>\n";
print HTML "</body></html>\n" ;
close HTML ;
So pretty basic.
I run: ./example_watch_folder.py cce1b01926646d548f6ddc32ff01aa2e http://140.253.78.234/galaxy/api/ /home/galaxy/data_input /home/galaxy/data_output "My API Import" f2db41e1fa331b3e
and get the following output: {'outputs': ['a799d38679e985db'], 'history': '33b43b4e7093c91f'}
I can see the sample.txt file I placed in the /home/galaxy/data_input has been put into the database: /home/galaxy/galaxy-dist/database/files/000/dataset_8.dat
and:
http://140.253.78.234/galaxy/api/libraries/f2db41e1fa331b3e/contents/1cd8e2f... shows the file has been uploaded:****
{****
"data_type": "txt", ****
"date_uploaded": "2013-05-09T04:41:10.579602", ****
"file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_8.dat", ****
"file_size": 76, ****
"genome_build": "?", ****
"id": "1cd8e2f6b131e891", ****
"ldda_id": "1cd8e2f6b131e891", ****
"message": "", ****
"metadata_data_lines": 8, ****
"metadata_dbkey": "?", ****
"misc_blurb": "8 lines", ****
"misc_info": "uploaded txt file", ****
"model_class": "LibraryDataset", ****
"name": "sam2.txt", ****
"template_data": {}, ****
"uploaded_by": "test@test.com", ****
"uuid": null****
}
However, looking in the histories: http://140.253.78.234/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38...
The output file is zero as the input file can not be found as shown below in bold ...
{****
"accessible": true, ****
"api_type": "file", ****
"data_type": "html", ****
"deleted": false, ****
"display_apps": [], ****
"display_types": [], ****
"download_url": "/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38679e985db/display", ****
"file_ext": "html", ****
"file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_9.dat", ****
"file_size": 0, ****
"genome_build": "?", ****
"hid": 1, ****
"history_id": "33b43b4e7093c91f", ****
"id": "a799d38679e985db", ****
"metadata_data_lines": null, ****
"metadata_dbkey": "?", ****
"misc_blurb": "error", ****
*"misc_info": "Thu May 9 14:42:10 2013input_image is: None\nNo such file or directory at /home/galaxy/galaxy-dist/tools/copy/copy.pl line 27.\n", *****
"model_class": "HistoryDatasetAssociation", ****
"name": "Copy Tool on None", ****
"peek": null, ****
"purged": false, ****
"state": "error", ****
"uuid": null, ****
"visible": true, ****
"visualizations": []****
}
Does anyone have any idea why the input_file is not passed in/obtained in the perl script? as the script obviously copies it to the database so part of the script is working?
the file sam2.txt looks like this
Thanks again, sorry for swamping the list with this issue
Neil****
** **
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Sorry for bringing this back but I am having an issue with the example_watch_folder.pl script. After running in and put a fasta file, it gets copied over but it is empty. Any idea how to fix this issue. Uploaded by: galaxy@ Date uploaded: 2013-05-30 File size: 0 bytes Data type: txt Build: ? Miscellaneous information: Uploaded temporary file (/tmp/API/myseq.fasta) does not exist. empty Database/Build: ? Number of data lines: None Disk file: /usr/local/galaxy/galaxy-dist/database/files/000/dataset_299.dat Thank you for your help, Regards From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Dannon Baker Sent: Thursday, May 09, 2013 6:48 AM To: Neil.Burdett@csiro.au Cc: Galaxy Dev Subject: Re: [galaxy-dev] Getting example_watch_folder.py to work... Hey Neil, A user browsing that history wouldn't automatically get the updates (unless the history was already refreshing for some reason), but they definitely would upon refresh. Glad you got the rest working! -Dannon On Thu, May 9, 2013 at 2:02 AM, <Neil.Burdett@csiro.au<mailto:Neil.Burdett@csiro.au>> wrote: Hi all, I finally resolved the issue I'll mention it here to help anyone else who has the same issue. In "Edit" mode on the workflow you add the "Workflow control->Inputs" to use the api. Final question.... Everything gets created and stored in the database, but the history column on the left of the browser doesn't get updated. Can user see if the task has completed successfully and all files uploaded from the browser? Thanks Neil From: Burdett, Neil (ICT Centre, Herston - RBWH) Sent: Thursday, 9 May 2013 2:56 PM To: galaxy-dev@lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu> Subject: Getting example_watch_folder.py to work... Hi, I'm still struggling to get the example_watch_folder.py to work. So any help much appreciated. I've created a simple workflow, which essentially opens a text file and then writes out the data to a html file. My xml file: <tool id="CopyTool" name="Copy Tool"> <description>Test</description> <command interpreter="perl">$__root_dir__/tools/copy/copy.pl<http://copy.pl> --input_image "$inputImage" --output_html "$output" </command> <inputs> <param format="txt" name="inputImage" type="data" label="Input Image" /> </inputs> <outputs> <data format="html" name="output" label="${tool.name<http://tool.name>} on #echo os.path.basename( str ( $inputImage.name ) )#" /> </outputs> <help> Copy Tool </help> </tool> And copy.pl<http://copy.pl>: #!/usr/bin/perl use strict; use warnings; use Getopt::Long; my $input_image ; my $output_html ; # Get options from command line (i.e. galaxy) my $result = GetOptions ( "input_image=s" => \$input_image, "output_html=s" => \$output_html ) or die "Bad options"; print "input_image is: $input_image\n"; open FILE,"$input_image" or die $!; my @lines = <FILE>; close FILE; my $numOfLines = scalar @lines; # Create the output HTML, with links to the files and the square gif. # open HTML, ">", $output_html or die "Failed to create output HTML file '$output_html': $! "; print HTML<<EOF; <html> <head> <style> iframe { border: 0px; background: #eeeeee ; } </style> </head> <body> <h1>Copy Files tool</h1> <h2>Generated $numOfLines files</h2> EOF # Put direct links to each output file foreach my $sub_filename ( @lines ) { print HTML "Direct link to the $sub_filename file.<br/>\n"; } print HTML "<br/><br/><br/>\n"; print HTML "</body></html>\n" ; close HTML ; So pretty basic. I run: ./example_watch_folder.py cce1b01926646d548f6ddc32ff01aa2e http://140.253.78.234/galaxy/api/ /home/galaxy/data_input /home/galaxy/data_output "My API Import" f2db41e1fa331b3e and get the following output: {'outputs': ['a799d38679e985db'], 'history': '33b43b4e7093c91f'} I can see the sample.txt file I placed in the /home/galaxy/data_input has been put into the database: /home/galaxy/galaxy-dist/database/files/000/dataset_8.dat and: http://140.253.78.234/galaxy/api/libraries/f2db41e1fa331b3e/contents/1cd8e2f... shows the file has been uploaded: { "data_type": "txt", "date_uploaded": "2013-05-09T04:41:10.579602", "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_8.dat", "file_size": 76, "genome_build": "?", "id": "1cd8e2f6b131e891", "ldda_id": "1cd8e2f6b131e891", "message": "", "metadata_data_lines": 8, "metadata_dbkey": "?", "misc_blurb": "8 lines", "misc_info": "uploaded txt file", "model_class": "LibraryDataset", "name": "sam2.txt", "template_data": {}, "uploaded_by": "test@test.com<mailto:test@test.com>", "uuid": null } However, looking in the histories: http://140.253.78.234/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38... The output file is zero as the input file can not be found as shown below in bold ... { "accessible": true, "api_type": "file", "data_type": "html", "deleted": false, "display_apps": [], "display_types": [], "download_url": "/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38679e985db/display", "file_ext": "html", "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_9.dat", "file_size": 0, "genome_build": "?", "hid": 1, "history_id": "33b43b4e7093c91f", "id": "a799d38679e985db", "metadata_data_lines": null, "metadata_dbkey": "?", "misc_blurb": "error", "misc_info": "Thu May 9 14:42:10 2013input_image is: None\nNo such file or directory at /home/galaxy/galaxy-dist/tools/copy/copy.pl<http://copy.pl> line 27.\n", "model_class": "HistoryDatasetAssociation", "name": "Copy Tool on None", "peek": null, "purged": false, "state": "error", "uuid": null, "visible": true, "visualizations": [] } Does anyone have any idea why the input_file is not passed in/obtained in the perl script? as the script obviously copies it to the database so part of the script is working? the file sam2.txt looks like this Thanks again, sorry for swamping the list with this issue Neil ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
In case someone has the same issue, This is resolved by extending the sleep time found here http://gmod.827538.n3.nabble.com/Trouble-Shooting-example-watch-folder-py-td... From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Hakeem Almabrazi Sent: Thursday, May 30, 2013 4:47 PM To: Dannon Baker; Neil.Burdett@csiro.au Cc: Galaxy Dev Subject: Re: [galaxy-dev] Getting example_watch_folder.py to work... Sorry for bringing this back but I am having an issue with the example_watch_folder.pl script. After running in and put a fasta file, it gets copied over but it is empty. Any idea how to fix this issue. Uploaded by: galaxy@ Date uploaded: 2013-05-30 File size: 0 bytes Data type: txt Build: ? Miscellaneous information: Uploaded temporary file (/tmp/API/myseq.fasta) does not exist. empty Database/Build: ? Number of data lines: None Disk file: /usr/local/galaxy/galaxy-dist/database/files/000/dataset_299.dat Thank you for your help, Regards From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Dannon Baker Sent: Thursday, May 09, 2013 6:48 AM To: Neil.Burdett@csiro.au Cc: Galaxy Dev Subject: Re: [galaxy-dev] Getting example_watch_folder.py to work... Hey Neil, A user browsing that history wouldn't automatically get the updates (unless the history was already refreshing for some reason), but they definitely would upon refresh. Glad you got the rest working! -Dannon On Thu, May 9, 2013 at 2:02 AM, <Neil.Burdett@csiro.au<mailto:Neil.Burdett@csiro.au>> wrote: Hi all, I finally resolved the issue I'll mention it here to help anyone else who has the same issue. In "Edit" mode on the workflow you add the "Workflow control->Inputs" to use the api. Final question.... Everything gets created and stored in the database, but the history column on the left of the browser doesn't get updated. Can user see if the task has completed successfully and all files uploaded from the browser? Thanks Neil From: Burdett, Neil (ICT Centre, Herston - RBWH) Sent: Thursday, 9 May 2013 2:56 PM To: galaxy-dev@lists.bx.psu.edu<mailto:galaxy-dev@lists.bx.psu.edu> Subject: Getting example_watch_folder.py to work... Hi, I'm still struggling to get the example_watch_folder.py to work. So any help much appreciated. I've created a simple workflow, which essentially opens a text file and then writes out the data to a html file. My xml file: <tool id="CopyTool" name="Copy Tool"> <description>Test</description> <command interpreter="perl">$__root_dir__/tools/copy/copy.pl<http://copy.pl> --input_image "$inputImage" --output_html "$output" </command> <inputs> <param format="txt" name="inputImage" type="data" label="Input Image" /> </inputs> <outputs> <data format="html" name="output" label="${tool.name<http://tool.name>} on #echo os.path.basename( str ( $inputImage.name ) )#" /> </outputs> <help> Copy Tool </help> </tool> And copy.pl<http://copy.pl>: #!/usr/bin/perl use strict; use warnings; use Getopt::Long; my $input_image ; my $output_html ; # Get options from command line (i.e. galaxy) my $result = GetOptions ( "input_image=s" => \$input_image, "output_html=s" => \$output_html ) or die "Bad options"; print "input_image is: $input_image\n"; open FILE,"$input_image" or die $!; my @lines = <FILE>; close FILE; my $numOfLines = scalar @lines; # Create the output HTML, with links to the files and the square gif. # open HTML, ">", $output_html or die "Failed to create output HTML file '$output_html': $! "; print HTML<<EOF; <html> <head> <style> iframe { border: 0px; background: #eeeeee ; } </style> </head> <body> <h1>Copy Files tool</h1> <h2>Generated $numOfLines files</h2> EOF # Put direct links to each output file foreach my $sub_filename ( @lines ) { print HTML "Direct link to the $sub_filename file.<br/>\n"; } print HTML "<br/><br/><br/>\n"; print HTML "</body></html>\n" ; close HTML ; So pretty basic. I run: ./example_watch_folder.py cce1b01926646d548f6ddc32ff01aa2e http://140.253.78.234/galaxy/api/ /home/galaxy/data_input /home/galaxy/data_output "My API Import" f2db41e1fa331b3e and get the following output: {'outputs': ['a799d38679e985db'], 'history': '33b43b4e7093c91f'} I can see the sample.txt file I placed in the /home/galaxy/data_input has been put into the database: /home/galaxy/galaxy-dist/database/files/000/dataset_8.dat and: http://140.253.78.234/galaxy/api/libraries/f2db41e1fa331b3e/contents/1cd8e2f... shows the file has been uploaded: { "data_type": "txt", "date_uploaded": "2013-05-09T04:41:10.579602", "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_8.dat", "file_size": 76, "genome_build": "?", "id": "1cd8e2f6b131e891", "ldda_id": "1cd8e2f6b131e891", "message": "", "metadata_data_lines": 8, "metadata_dbkey": "?", "misc_blurb": "8 lines", "misc_info": "uploaded txt file", "model_class": "LibraryDataset", "name": "sam2.txt", "template_data": {}, "uploaded_by": "test@test.com<mailto:test@test.com>", "uuid": null } However, looking in the histories: http://140.253.78.234/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38... The output file is zero as the input file can not be found as shown below in bold ... { "accessible": true, "api_type": "file", "data_type": "html", "deleted": false, "display_apps": [], "display_types": [], "download_url": "/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38679e985db/display", "file_ext": "html", "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_9.dat", "file_size": 0, "genome_build": "?", "hid": 1, "history_id": "33b43b4e7093c91f", "id": "a799d38679e985db", "metadata_data_lines": null, "metadata_dbkey": "?", "misc_blurb": "error", "misc_info": "Thu May 9 14:42:10 2013input_image is: None\nNo such file or directory at /home/galaxy/galaxy-dist/tools/copy/copy.pl<http://copy.pl> line 27.\n", "model_class": "HistoryDatasetAssociation", "name": "Copy Tool on None", "peek": null, "purged": false, "state": "error", "uuid": null, "visible": true, "visualizations": [] } Does anyone have any idea why the input_file is not passed in/obtained in the perl script? as the script obviously copies it to the database so part of the script is working? the file sam2.txt looks like this Thanks again, sorry for swamping the list with this issue Neil ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (3)
-
Dannon Baker -
Hakeem Almabrazi -
Neil.Burdett＠csiro.au