Compatibility with Resque implementations ? #138

Activity

1. bluegod commented on Sep 1, 2014

   

   bluegod

   on Sep 1, 2014

   More actions

   Did anyone find out if Pyres is compatible?
2. binarymatt commented on Sep 1, 2014

   

   binarymatt

   on Sep 1, 2014

   Owner

   More actions

   @louisameline and @bluegod i guess the answer depends on the definition of compatible. Can you enqueue jobs with another implementation and pop them off with pyres? If the json that is push is the same, then yes. For instance, a pyres job looks for a json payload that has the following keys:
   class, args, enqueue_timestamp. From the tests you can see what a sample payload looks like
   {'class':'tests.Basic','args':['test1'],'enqueue_timestamp':job.enqueue_timestamp}
3. louisameline commented on Sep 2, 2014

   

   louisameline

   on Sep 2, 2014

   Author

   More actions

   Thanks for your answer.

   Ok, I see that class and args seem to be the only properties needed to enqueue a job (you add an extra enqueue_timestamp and php-resque adds an extra id but I guess this does not matter).

   It gets more complex when you deal with job failures though. Coffee-resque has this payload similar to Resque's one (https://github.com/resque/resque/blob/master/lib/resque/failure/redis.rb) that makes its errors seamlessly monitorable in Resque-web too :

   {
       "worker": "node:7236:node_contact_import",
       "queue":"node_contact_import",
       "payload": {
           "class":"import",
           "args":[....],
           "id":"716e8ac85c64e10286df04881c28f70d"
       },
       "exception":"error",
       "error":"my error name",
       "backtrace": [
           " at Connection.parseE",
           " at Connection.parseMessage,
           " at TCP.onread (net.js:526:21)"
       ],
       "failed_at":"Wed Aug 13 2014 11:04:12 GMT+0200 (Paris, Madrid (heure d�été))"
   }
   

   Have you done the same for Pyres ?
   I could check the source myself but I thought I'd ask you at the same time to include this information in the documentation if you're willing to.
   Thank you.
4. gkop commented on Oct 9, 2014

   

   gkop

   on Oct 9, 2014

   More actions

   I agree @louisameline that we should unambiguously document the level of compatibility between pyres and resque. I am about to wrap some python code in a worker job of some kind, and would prefer to use pyres, but want to know "Well is it a clone of resque, or not a clone of resque?"
5. binarymatt commented on Oct 9, 2014

   

   binarymatt

   on Oct 9, 2014

   Owner

   More actions

   @louisameline sorry about the long delay, i've not been seeing github notifications in my gmail. Pyres follows that same failure json. There may be some slight differences because it's not been a priority to track resque in the last few years.
6. louisameline commented on Sep 13, 2015

   

   louisameline

   on Sep 13, 2015

   Author

   More actions

   Hello again !

   Thank you, I am eventually implementing Pyres. Unfortunately with difficulty because the documentation falls short of thorough and explained examples for python total newbies like me.

   Anyway, I am trying to send a job from Python for Node to pick it up. I have created a parse class that has a queue = 'node' attribute and I call resque.enqueue(parse, arg). The problem is that the class property in the JSON payload is __main__.parse while Node only expects parse. What can I do ?

   Thank you.
7. louisameline commented on Sep 14, 2015

   

   louisameline

   on Sep 14, 2015

   Author

   More actions

   Looking at the source, I found a resq.enqueue_from_string(className, queue, *args, **kwargs) method that does just what I need. I think it should be documented.

   Next issue : the other way around, when enqueing from Node to Python. From the payload, Pyres seems to expect a class property in the form packageName.className. I guess I could include the package name when I enqueue in my Node app, but that does not really sound right. My Node app should probably not have to care for what package will be used by Python, that's outside of its scope.

   If pyres_worker and pyres_manager need the location/package of the class, could we pass it as an IMPORT_PATH option in the command line instead ? I don't know if it would be a very python-like way of doings things but it would :

   • solve the issue of the package name being present in the class property
   • save me the trouble of having to learn how import works and how I'm supposed to declare my app as a package. I know that sounds lazy :p ^^

   Thanks
8. louisameline commented on Sep 14, 2015

   

   louisameline

   on Sep 14, 2015

   Author

   More actions

   I confirm that everything works well when I package my python worker class under the name "workers" and give a "workers.parse" class when I enqueue in Coffee-Resque (Node). However I haven't tested to run ResqueWeb yet.

   For newbies like me : to package my parse class (included in a file called "workers.py") so that Pyres will find it, I followed the simple instructions given in this helpful discussion. Basically, you just have to execute this file :

   #!/usr/bin/python
   
   from distutils.core import setup
   
   setup(name='workers',
       version='1.0.0',
       py_modules=['workers'],
       )