Group items tagged

Various Web surfing tasks that I regularly perform could be made much easier, and less tedious, if I could only use Python to fetch the HTML pages and to process them, yielding the information I really need. In this document I attempt to describe HTML processing in Python using readily available tools and libraries.

What idioms should I use to make my code easier to read? Read "The Python Cookbook", especially the first few chapters. It's a great source of well-written Python code examples.

Use function factories to create utility functions. Often, especially if you're using map and filter a lot, you need utility functions that convert other functions or methods to taking a single parameter. In particular, you often want to bind some data to the function once, and then apply it repeatedly to different objects. In the above example, we needed a function that multiplied a particular field of an object by 3, but what we really want is a factory that's able to return for any field name and amount a multiplier function in that family:

Use zip and dict to map fields to names. zip turns a pair of sequences into a list of tuples containing the first, second, etc. values from each sequence. For example, zip('abc', [1,2,3]) == [('a',1),('b',2),('c',3)]. You can use this to save a lot of typing when you have fields in a known order that you want to map to names:

To attempt to explain it in the simplest terms, a continuation is a representation, at a particular point in a program, of everything the program is capable of doing subsequently. A continuation is a potential that depends on initial conditions. Rather than loop in a traditional way, it is possible to invoke the same continuation recursively with different initial conditions. One broad claim I have read is that continuations, in a theoretical sense, are more fundamental and underlie every other control structure. Don't worry if these ideas cause your brain to melt; that is a normal reaction.

Note that it's more rebust to run methods on the public IPython api.I.e. do ip = ipshell.api and then ip.magic('px import os')You can explore the api interactively by playing with _ip object.