Group items tagged

[IPython-user] ipython1 and farm tasking Brian Granger ellisonbg.net@gmail.... Wed Feb 27 16:29:03 CST 2008 Previous message: [IPython-user] ipython1 and farm tasking Next message: [IPython-user] yet another leopard/readline question Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] Alex, First, I would suggest updating your ipython1 install from our svn repository. We are about to push out a major new version and the documentation is _much_ better. Also, there are many new features that will hopefully help you. Here is a simple example (using the latest svn of ipython1): In [1]: from ipython1.kernel import client In [2]: mec = client.MultiEngineClient(('127.0.0.1',10105)) In [3]: tc = client.TaskClient(('127.0.0.1',10113)) In [4]: def fold_package(x): ...: return 2.0*x ...: In [5]: mec.push_function(dict(fold_package=fold_package)) Out[5]: [None, None, None, None] In [6]: tasks = [client.Task("y=fold_package(x)",push={'x':x},pull=('y',)) for x in range(128)] In [7]: task_ids = [tc.run(t) for t in tasks] In [8]: tc.barrier(task_ids) In [9]: task_results = [tc.get_task_result(tid) for tid in task_ids] In [10]: results = [tr.ns.y for tr in task_results] In [11]: print results [0.0, 2.0, 4.0, 6.0, 8.0, 10.0, 12.0, 14.0, 16.0, 18.0, 20.0, 22.0, 24.0, 26.0, 28.0, 30.0, 32.0, 34.0, 36.0, 38.0, 40.0, 42.0, 44.0, 46.0, 48.0, 50.0, 52.0, 54.0, 56.0, 58.0, 60.0, 62.0, 64.0, 66.0, 68.0, 70.0, 72.0, 74.0, 76.0, 78.0, 80.0, 82.0, 84.0, 86.0, 88.0, 90.0, 92.0, 94.0, 96.0, 98.0, 100.0, 102.0, 104.0, 106.0, 108.0, 110.0, 112.0, 114.0, 116.0, 118.0, 120.0, 122.0, 124.0, 126.0, 128.0, 130.0, 132.0, 134.0, 136.0, 138.0, 140.0, 142.0, 144.0, 146.0, 148.0, 150.0, 152.0, 154.0, 156.0, 158.0, 160.0, 162.0, 164.0, 166.0, 168.0, 170.0, 172.0, 174.0, 176.0, 178.0, 180.0, 182.0, 184.0, 186.0, 188.0, 190.0, 192.0, 194.0, 196.0, 198.0, 200.0, 202.0, 204.0, 206.0, 208.0, 210.0, 212.0, 214.0, 216.0, 218.0, 220.0, 222.0, 224.0, 226.0, 228.0, 230.0, 232.0, 234.0, 236.0, 238.0, 240.0, 242.0, 244.0, 246.0, 248.0, 250.0, 252.0, 254.0] Or if you don't need load balancing: # This sends the fold_package function for you! results = mec.map(fold_package, range(128)) Let us know if you run into other problems. Cheers, Brian

One of the main difficulties I've had using parser generators has been the difficulty of figuring out why a grammar didn't work. Fixing shift-reduce and reduce-reduce conflicts seemed like voodoo to me, and though I slightly understand better how to fix such things now it's still a different mode of thinking that I don't want to try to get into when I just want to parse something simple. PyMeta uses a variation on the Parsing Expression Grammar (PEG) approach to parsing. The chief consequence of this is there's no possibility of ambiguity in a parse: a successful parse will yield exactly one result, and you can trace the control flow through the grammar to figure out how it got there.

PyInstaller is a program that converts (packages) Python programs into stand-alone executables, under Windows, Linux and Irix. Its main advantages over similar tools are that PyInstaller works with any version of Python since 1.5, it builds smaller executables thanks to transparent compression, it is multi-platform (so you can build one-file binaries also under Linux), and use the OS support to load the dynamic libraries, thus ensuring full compatibility. PyInstaller is an effort to rescue, maintain and further develop Gordon McMillan's Python Installer (now PyInstaller). Their official website is not longer available and the original package is not longer maintained. Believing that it is still far superior to py2exe, we have setup this site to continue its further development. Feel free to join us in the effort! Please consult our Roadmap to check our plans. Also usage reports are welcomed: let us know if PyInstaller works for you and how, or what problems you found in using it.

Document options Document options requiring JavaScript are not displayed Rate this pageHelp us improve this contentLevel: IntroductoryDavid Mertz (mertz@gnosis.cx), Applied Metaphysician, Gnosis Software, Inc. 01 Mar 2001Although users usually think of Python as a procedural and object-oriented language, it actually contains everything you need for a completely functional approach to programming. This article discusses general concepts of functional programming, and illustrates ways of implementing functional techniques in Python. We'd better start with the hardest question: "What is functional programming (FP), anyway?" One answer would be to say that FP is what you do when you program in languages like Lisp, Scheme, Haskell, ML, OCAML, Clean, Mercury, or Erlang (or a few others). That is a safe answer, but not one that clarifies very much. Unfortunately, it is hard to get a consistent opinion on just what FP is, even from functional programmers themselves. A story about elephants and blind men seems apropos here. It is also safe to contrast FP with "imperative programming" (what you do in languages like C, Pascal, C++, Java, Perl, Awk, TCL, and most others, at least for the most part).

What idioms should I use to make my code easier to read? Read "The Python Cookbook", especially the first few chapters. It's a great source of well-written Python code examples.

Use function factories to create utility functions. Often, especially if you're using map and filter a lot, you need utility functions that convert other functions or methods to taking a single parameter. In particular, you often want to bind some data to the function once, and then apply it repeatedly to different objects. In the above example, we needed a function that multiplied a particular field of an object by 3, but what we really want is a factory that's able to return for any field name and amount a multiplier function in that family:

Use zip and dict to map fields to names. zip turns a pair of sequences into a list of tuples containing the first, second, etc. values from each sequence. For example, zip('abc', [1,2,3]) == [('a',1),('b',2),('c',3)]. You can use this to save a lot of typing when you have fields in a known order that you want to map to names:

Sometimes you get an Excel spreadsheet (say, from the marketing departement) and you want to read tabular data from it (i.e. a line with column headers and lines of data). There are many ways to do this (including ODBC + mxODBC), but the easiest way I've found is this one : provide a file name and a sheet name, and read the data !

aspects.py library provides means to intercept function calls. Functions and methods (also in Python standard library and third party code) can be wrapped so that when they are called, the wrap is invoked first. Depending on the wrap, the execution of the original function can be omitted, or the function can be called arbitrarily many times. Wraps are able to modify the arguments and the return value of the original function. In the terminology of aspect-oriented programming, the library allows applying advices (wraps) to call join points of methods and functions in around fashion.

I love python for its beautiful code and practicality. But it’s not going to win a pure speed race with most languages. Most people think of speed and ease-of-use as polar opposites - probably because they remember the pain of writing C. Cython tries to eliminate that duality and lets you have python syntax with C data types and functions - the best of both worlds. Keeping in mind that I’m by no means an expert at this, here are my notes based on my first real experiment with Cython:

Given Python's goal to be a powerful and easy-to-use scripting language, its features for working with dates and times are not as user-friendly as they should be. The purpose of pyfdate is to remedy that situation by providing features for working with dates and times that are as powerful and easy-to-use as the rest of Python.

PyProtocols extends the PEP 246 adapt() function with a new "declaration API" that lets you easily define your own protocols and adapters, and declare what adapters should be used to adapt what types, objects, or protocols.  In addition to its own Interface type, PyProtocols can also use Twisted and Zope's Interface types too.  (Of course, since Twisted and Zope interfaces aren't as flexible, only a subset of the PyProtocols API works with them.  Specific limitations are listed in the documentation.)

RRDTool is a really good back-end for storing time-series data. If you are developing tools that need a data repository and graphing capabilities, this provides you both. You create an RRD and then you begin inserting data values at regular intervals. You then call the graphing API to have a graph displayed. The neat thing about this data storage is its “round robin” nature. You define various time spans, and the granularity at which you want them stored. A fixed binary file is created, which never grows in size over time. As you insert more data, it is inserted into each span. As results are collected, they are averaged and rolled into successive time spans. It makes a much more efficient system than using your own complex data structures, relational database, or file system storage.

 > According to the fact that all Thread run on the same CPU (if i didn't  > understand wrong), i'm asking if python will suffer from the future  > multicore CPU. Will not python use only one core, then a half or a  > quarter of CPU ? It could be a serious problem for the future of python... I agree that it could potentially be a serious hindrance for cpython if "multiple core" CPUs become commonplace. This is in contrast to jython and ironpython, both of which support multiple-cpu parallelism. Although I completely accept the usual arguments offered in defense of the GIL, i.e. that it isn't a problem in the great majority of use cases, I think that position will become more difficult to defend as desktop CPUs sprout more and more execution pipelines. I think that this also fits in with AM Kuchling's recent musing/thesis/prediction that the existing cpython VM may no longer be in use in 5 years, and that it may be superceded by python "interpreters" running on top of other VMs, namely the JVM, the CLR, Smalltalk VM, Parrot, etc, etc, etc. http://www.amk.ca/diary/archives/cat_python.html#003382 I too agree with Andrew's basic position: the Python language needs a period of library consolidation. There is so much duplication of functionality out there, with the situation only getting worse as people re-invent the wheel yet again using newer features such generators, gen-exps and decorators.