PythonLanguage/HttpTechniques - Raymond Yee's Wiki

Problem to solve: how to programmatically push content to MyWikis. Some possible help in :

Http client to POST using multipart/form-data from the PythonCookbook
Maybe there is a web-services approach in MoinMoin too. For this problem look at

I will most likely end up using the httplib library; some examples of using httplib will be helpful.

HTML parsing

The example from MarkPilgrim's Extracting data from HTML documents uses SGMLParser, related to htmllib. An alternative interface is HTMLParser.

Which one to use? According to the discussion thread htmllib vs. HTMLParser, HTMLParser is "A much improved HTML parser -- a replacement for sgmllib". So I'll give HTMLParser a try.

Is the solution here?

mechanize: Stateful programmatic web browsing in Python, after Andy Lester's Perl module WWW::Mechanize

Crawling/Spidering/User Agent tool?

Stuart Langridge: Browser.py ["Browser.py is an automated web browser for Python. It's something like Perl's WWW:Mechanize - use it to navigate to a page, follow links, fill out forms, and the like."]

Misc Notes and Questions

A Python Web-SIG has been established. The mailing list archive is off to a good start.

Is there any Python equivalent of PerlLanguage's LWP (which I remember being really awesome? Last time I checked, I would say no, like this post on usenet. I hope I'm wrong.

Lots of links at Python Web programming wiki on lots of topics except for "Python-based Web Client Programming" -- maybe there is just a dearth of good modules in this area?

Is Python Web Programming any good?

Good to note a post I wrote in 2000 about using urllib and urllib2 in python -- one that got no response.

Another thread that shows how LWP makes life easy in Perl but there's a lot of low-level munging in Python to get the same effect. Sigh. (Time to pull out PerlLanguage again!)

Python and HTML Processing:

Various Web surfing tasks that I regularly perform could be made much easier, and less tedious, if I could only use Python to fetch the HTML pages and to process them, yielding the information I really need. In this document I attempt to describe HTML processing in Python using readily available tools and libraries.