Comments on: Summary of my Python optimization adventures

By: Top Posts « WordPress.com

Top Posts « WordPress.com — Mon, 18 Feb 2008 23:59:23 +0000

[...] Summary of my Python optimization adventures This is a follow up to two previous posts. In the first one I spoke about saving memory by reading line-by-line, […] [...]

By: Lorenzo E. Danielsson

Lorenzo E. Danielsson — Mon, 18 Feb 2008 20:43:32 +0000

@James: what is wrong about calling an external tool to perform a job that it was designed to do? It doesn’t make the program itself any less “python”. That’s the UNIX way, let each tool do what it’s best at.

By: Andrew Dalke

Andrew Dalke — Mon, 18 Feb 2008 17:39:07 +0000

Oops, missed the parens: execfile(“your_filename.py”)

By: Andrew Dalke

Andrew Dalke — Mon, 18 Feb 2008 17:23:43 +0000

While I find it annoying to use, try the “profile” module to get an idea of where time is spent in your program. Because your code isn’t written as a module, the easiest way to do the profiling is using the command “execfile ‘filename’”.

This should tell you which line consumes the most time.

By: Justin

Justin — Mon, 18 Feb 2008 15:07:52 +0000

If you are going to do this:
if search_cre(line): line = re.sub('>','<',line) aline = line.split('<') credit = float(aline[2])

you should just change the regular expression to to be
"total_credit >(?P[^<]+)<"

or such, and then just pull out the credit if it matched. The way you’re doing it, you are processing the same line 3 times. From the looks of it, you could change the program to use 1 regular expression for everything, instead of 4.

f = os.popen('zcat host.gz')

Will be a lot faster than the gzip module though.

By: isilanes

isilanes — Mon, 18 Feb 2008 13:34:05 +0000

Andrew, I am not doubting that the “in” construct is faster (I repeated your test, and here it’s 3 times faster, as well). The problem is that maybe that part of the script is not the bottleneck, and the uncertainty in measured time (I always measured walltime with /usr/bin/time -f %e command is of the order of the difference in using one or the other, so I can’t diferentiate. I’ll keep testing… (the faster the script, the more noticeable the subtle differences).

By: Andrew Dalke

Andrew Dalke — Mon, 18 Feb 2008 13:21:41 +0000

When I benchmark with

python -m timeit -s ‘import re’ -s ’search = re.compile(“Linux”).match’ ’search(“Uses Linux”)’

I get “0.413 usec per loop”. When I benchmark with

python -m timeit ‘”Linux” in “Uses Linux”‘

I get “0.141 usec per loop”. Not quite 3 times faster than using search. (My 20-fold case was when I tested re.search(), which has a couple extra function calls overhead and a cache check.)

Are you sure you timed what you think you timed? Every time I’ve done the comparison the “in” test is faster, and I know the underlying implementation well enough that I can’t think of how it can be slower than the re code.

Also, instead of writing to a temporary file and reading from that, use either the subprocess module or the older and harder to use os.popen call. (Harder to use because it’s harder to deal with errors.) That should also give you some performance increase because you aren’t doing a full read/write through the disk.

By: isilanes

isilanes — Mon, 18 Feb 2008 12:01:22 +0000

James, you’re absolutely right. What I meant was “optimizing a script”, by using the best tools I could get access to (plus my limited knowledge).

Anyway, my first attempt was to make it all in Python, and as effectively as possible, and I give some hints of what to do and what not to do… so this qualifies as “Python optimization”? :^)

By: James

James — Mon, 18 Feb 2008 11:41:00 +0000

Is this really Python optimization? Haven’t you just offloaded most of the program to grep, effectively “rewriting” it in C?

By: isilanes

isilanes — Mon, 18 Feb 2008 10:42:31 +0000

Thanks Paolo and Andrew for your recommendations. The fact is that I am by no means a hacker, just a lame hobbyist :^)

I post my experience so that others can benefit, but my ignorance is huuuuge. Thanks to interaction with others, such as your comments, I keep learning every day… thanks!

@Paolo: If you mean using match instead of search, I did it, and saw little or no gain (although I expected to see it).

@Andrew: I have read that “if pattern in line:” is faster than “if re.search(pattern, line):”, but I have tested it, and if “pattern” is pre-compiled, I see no real advantage (actually it was slightly slower).

The dictionary lookup avoidance thing is a good advice, I will try it. Anyway, I am reluctant to make much more optimization here, because most of the computation time is spent in the system call to zcat and grep, so it doesn’t make much sense make a big effort to reduce the time spent in the “Python part”. I will eventually do it, anyway, just for the fun :^)