Group items tagged

Various Web surfing tasks that I regularly perform could be made much easier, and less tedious, if I could only use Python to fetch the HTML pages and to process them, yielding the information I really need. In this document I attempt to describe HTML processing in Python using readily available tools and libraries.

[IPython-user] ipython1 and farm tasking Brian Granger ellisonbg.net@gmail.... Wed Feb 27 16:29:03 CST 2008 Previous message: [IPython-user] ipython1 and farm tasking Next message: [IPython-user] yet another leopard/readline question Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] Alex, First, I would suggest updating your ipython1 install from our svn repository. We are about to push out a major new version and the documentation is _much_ better. Also, there are many new features that will hopefully help you. Here is a simple example (using the latest svn of ipython1): In [1]: from ipython1.kernel import client In [2]: mec = client.MultiEngineClient(('127.0.0.1',10105)) In [3]: tc = client.TaskClient(('127.0.0.1',10113)) In [4]: def fold_package(x): ...: return 2.0*x ...: In [5]: mec.push_function(dict(fold_package=fold_package)) Out[5]: [None, None, None, None] In [6]: tasks = [client.Task("y=fold_package(x)",push={'x':x},pull=('y',)) for x in range(128)] In [7]: task_ids = [tc.run(t) for t in tasks] In [8]: tc.barrier(task_ids) In [9]: task_results = [tc.get_task_result(tid) for tid in task_ids] In [10]: results = [tr.ns.y for tr in task_results] In [11]: print results [0.0, 2.0, 4.0, 6.0, 8.0, 10.0, 12.0, 14.0, 16.0, 18.0, 20.0, 22.0, 24.0, 26.0, 28.0, 30.0, 32.0, 34.0, 36.0, 38.0, 40.0, 42.0, 44.0, 46.0, 48.0, 50.0, 52.0, 54.0, 56.0, 58.0, 60.0, 62.0, 64.0, 66.0, 68.0, 70.0, 72.0, 74.0, 76.0, 78.0, 80.0, 82.0, 84.0, 86.0, 88.0, 90.0, 92.0, 94.0, 96.0, 98.0, 100.0, 102.0, 104.0, 106.0, 108.0, 110.0, 112.0, 114.0, 116.0, 118.0, 120.0, 122.0, 124.0, 126.0, 128.0, 130.0, 132.0, 134.0, 136.0, 138.0, 140.0, 142.0, 144.0, 146.0, 148.0, 150.0, 152.0, 154.0, 156.0, 158.0, 160.0, 162.0, 164.0, 166.0, 168.0, 170.0, 172.0, 174.0, 176.0, 178.0, 180.0, 182.0, 184.0, 186.0, 188.0, 190.0, 192.0, 194.0, 196.0, 198.0, 200.0, 202.0, 204.0, 206.0, 208.0, 210.0, 212.0, 214.0, 216.0, 218.0, 220.0, 222.0, 224.0, 226.0, 228.0, 230.0, 232.0, 234.0, 236.0, 238.0, 240.0, 242.0, 244.0, 246.0, 248.0, 250.0, 252.0, 254.0] Or if you don't need load balancing: # This sends the fold_package function for you! results = mec.map(fold_package, range(128)) Let us know if you run into other problems. Cheers, Brian

pylize is a Python script that makes the creation of on-screen presentations a matter of a few minutes. It generates a template master document, which you can edit with your favourite text or HTML editor. The master document is then processed by pylize to generate HTML files for every slide plus a file for the table of contents. You can view the presentation with any CSS-capable webbrowser.

What is Py++? Definition: Py++ is an object-oriented framework for creating a code generator for Boost.Python library. Py++ uses few different programming paradigms to help you to expose C++ declarations to Python. This code generator will not stand on your way. It will guide you through the whole process. It will raise warnings in the case you are doing something wrong with a link to the explanation. And the most important it will save your time - you will not have to update code generator script every time source code is changed.

One of the main difficulties I've had using parser generators has been the difficulty of figuring out why a grammar didn't work. Fixing shift-reduce and reduce-reduce conflicts seemed like voodoo to me, and though I slightly understand better how to fix such things now it's still a different mode of thinking that I don't want to try to get into when I just want to parse something simple. PyMeta uses a variation on the Parsing Expression Grammar (PEG) approach to parsing. The chief consequence of this is there's no possibility of ambiguity in a parse: a successful parse will yield exactly one result, and you can trace the control flow through the grammar to figure out how it got there.

Epydoc is a tool for generating API documentation for Python modules, based on their docstrings. For an example of epydoc's output, see the API documentation for epydoc itself (html, pdf). A lightweight markup language called epytext can be used to format docstrings, and to add information about specific fields, such as parameters and instance variables. Epydoc also understands docstrings written in reStructuredText, Javadoc, and plaintext. For a more extensive example of epydoc's output, see the API documentation for Python 2.5.

PyProtocols extends the PEP 246 adapt() function with a new "declaration API" that lets you easily define your own protocols and adapters, and declare what adapters should be used to adapt what types, objects, or protocols.  In addition to its own Interface type, PyProtocols can also use Twisted and Zope's Interface types too.  (Of course, since Twisted and Zope interfaces aren't as flexible, only a subset of the PyProtocols API works with them.  Specific limitations are listed in the documentation.)

Pypar does not require the Python interpreter to be modified or recompiled: Parallel python programs use the standard Python and need merely import the pypar module. This means for example that you can upgrade Python independently of your parallel codes.

Pypar is an efficient but easy-to-use module that allows programs/scripts written in the Python programming language to run in parallel on multiple processors and communicate using message passing. Pypar provides bindings to an important subset of the message passing interface standard MPI. Other Python MPI bindings available from other developers include: PyMPI, Scientific Python and pythonMPI.

 > According to the fact that all Thread run on the same CPU (if i didn't  > understand wrong), i'm asking if python will suffer from the future  > multicore CPU. Will not python use only one core, then a half or a  > quarter of CPU ? It could be a serious problem for the future of python... I agree that it could potentially be a serious hindrance for cpython if "multiple core" CPUs become commonplace. This is in contrast to jython and ironpython, both of which support multiple-cpu parallelism. Although I completely accept the usual arguments offered in defense of the GIL, i.e. that it isn't a problem in the great majority of use cases, I think that position will become more difficult to defend as desktop CPUs sprout more and more execution pipelines. I think that this also fits in with AM Kuchling's recent musing/thesis/prediction that the existing cpython VM may no longer be in use in 5 years, and that it may be superceded by python "interpreters" running on top of other VMs, namely the JVM, the CLR, Smalltalk VM, Parrot, etc, etc, etc. http://www.amk.ca/diary/archives/cat_python.html#003382 I too agree with Andrew's basic position: the Python language needs a period of library consolidation. There is so much duplication of functionality out there, with the situation only getting worse as people re-invent the wheel yet again using newer features such generators, gen-exps and decorators.

PyCHM - Python bindings for CHMLIB PyCHM is a package that provides bindings for Jed Wing's CHMLIB library. The chm package contains four modules, namely chm.chm, chm.chmlib, chm.extra and chm._chmlib. chm.chmlib is a low level wrapper module around the API provided by the C library chmlib. Quoted from Jed's README: chmlib is a small library designed for accessing MS ITSS files. The ITSS file format is used for Microsoft Html Help files (.chm), which have been the predominant medium for software documentation from Microsoft during the past several years, having superceded the previously used .hlp file format. chm.chm provides some high level functionality over chm.chmlib, such as access to the .chm file contents tree. chm.extra contains extra functionality to allow detection encodings in the CHM archives and to support full-text search.