Re: [nimh-dev] more format profiling

["Jon Stewart" <stew1@xxxxxxxxxxx>]

> Here's what the profiler tells me when I run my present version of scan.py:
> 
>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>         1    0.010    0.010    0.743    0.743 nscan.py:5(?)
>         1   -0.000   -0.000    0.743    0.743 profile:0(<code object ? at 0xf
> 6d0c420, file "nscan.py", line 5>)
>         1    0.017    0.017    0.638    0.638 test:1(?)
>         1   -0.000   -0.000    0.638    0.638 Format.py:23(inst_loop)
>       154    0.017    0.000    0.564    0.004 Folder.py:19(openmessage)
>       154    0.016    0.000    0.547    0.004 Message.py:4(message_from_file)
>       154    0.006    0.000    0.470    0.003 __init__.py:54(message_from_fil
> e)
>   177/154   -0.006   -0.000    0.457    0.003 Parser.py:53(parse)
>   177/154    0.167    0.001    0.356    0.002 Parser.py:143(_parsebody)
>        44    0.008    0.000    0.160    0.004 sre_compile.py:461(compile)
> 
> These are the top 10 functions by cumulative time. Of the 0.743s that
> nscan.py takes, most of that is spent in test, which is the format code.
> Unfortunately, 5/6 of the time used by test is for Folder.openmessage(),
> so my format code is within 0.074s of optimality. Folder.openmessage() wraps
> Message.message_from_file(), and from there execution goes off into the
> email module. So it looks like reducing execution time significantly would
> involve implementing some of the email module ourselves. Ick.


Nice profiling, Joel.

It looks like a fair amount of time is spent in Parser.py:_parsebody. If 
the name is any indication, that's largely unnecessary processing, since 
we only care about headers the vast majority of the time.

I'd be willing to bet that once we built something more substantial, we 
could hack the standard email module to be more efficient at processing 
headers only. Said hack could then be submitted as a patch, at which point 
we bring in other developers to make it better, via the magic of open 
source.

I don't think we should let performance concerns discourage us. Three 
quarters of a second is too long? Wait 2 years and you won't notice it, 
thanks to incremental optimization and Moore's law.

Make it run, make it right, make it fast.


Jon
-- 
Jon Stewart                                 Advanced Los Angeles C++
stew1@xxxxxxxxxxx                           http://www.alacpp.org
_______________________________________________
nimh-dev mailing list
nimh-dev@xxxxxxxxxxx
http://lists.ellipsis.cx/mailman/listinfo/nimh-dev