Group items tagged

[IPython-user] ipython1 and farm tasking Brian Granger ellisonbg.net@gmail.... Wed Feb 27 16:29:03 CST 2008 Previous message: [IPython-user] ipython1 and farm tasking Next message: [IPython-user] yet another leopard/readline question Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] Alex, First, I would suggest updating your ipython1 install from our svn repository. We are about to push out a major new version and the documentation is _much_ better. Also, there are many new features that will hopefully help you. Here is a simple example (using the latest svn of ipython1): In [1]: from ipython1.kernel import client In [2]: mec = client.MultiEngineClient(('127.0.0.1',10105)) In [3]: tc = client.TaskClient(('127.0.0.1',10113)) In [4]: def fold_package(x): ...: return 2.0*x ...: In [5]: mec.push_function(dict(fold_package=fold_package)) Out[5]: [None, None, None, None] In [6]: tasks = [client.Task("y=fold_package(x)",push={'x':x},pull=('y',)) for x in range(128)] In [7]: task_ids = [tc.run(t) for t in tasks] In [8]: tc.barrier(task_ids) In [9]: task_results = [tc.get_task_result(tid) for tid in task_ids] In [10]: results = [tr.ns.y for tr in task_results] In [11]: print results [0.0, 2.0, 4.0, 6.0, 8.0, 10.0, 12.0, 14.0, 16.0, 18.0, 20.0, 22.0, 24.0, 26.0, 28.0, 30.0, 32.0, 34.0, 36.0, 38.0, 40.0, 42.0, 44.0, 46.0, 48.0, 50.0, 52.0, 54.0, 56.0, 58.0, 60.0, 62.0, 64.0, 66.0, 68.0, 70.0, 72.0, 74.0, 76.0, 78.0, 80.0, 82.0, 84.0, 86.0, 88.0, 90.0, 92.0, 94.0, 96.0, 98.0, 100.0, 102.0, 104.0, 106.0, 108.0, 110.0, 112.0, 114.0, 116.0, 118.0, 120.0, 122.0, 124.0, 126.0, 128.0, 130.0, 132.0, 134.0, 136.0, 138.0, 140.0, 142.0, 144.0, 146.0, 148.0, 150.0, 152.0, 154.0, 156.0, 158.0, 160.0, 162.0, 164.0, 166.0, 168.0, 170.0, 172.0, 174.0, 176.0, 178.0, 180.0, 182.0, 184.0, 186.0, 188.0, 190.0, 192.0, 194.0, 196.0, 198.0, 200.0, 202.0, 204.0, 206.0, 208.0, 210.0, 212.0, 214.0, 216.0, 218.0, 220.0, 222.0, 224.0, 226.0, 228.0, 230.0, 232.0, 234.0, 236.0, 238.0, 240.0, 242.0, 244.0, 246.0, 248.0, 250.0, 252.0, 254.0] Or if you don't need load balancing: # This sends the fold_package function for you! results = mec.map(fold_package, range(128)) Let us know if you run into other problems. Cheers, Brian

This script mimics rsync which is available for the unix platform and have been ported to win32 one. It is a sort of advanced version of xcopy. Its aim is to selectively synchronize folders. More precisely it copy selective parts of a folder to a destination folder and in addition can remove parts of the destination folder that do not correspond to parts of the original folder. I like its capability to avoid copying files through the .cvsignore mechanism or the pattern matching mechanism and its capability to delete files that are no longer relevant, not to mention that because it's a python script anyone can easily fix or improve it as he whish.