RCF's Software Thoughts

Thoughts On The Wide World Of Software Development

Zoom into Mandelbrot – YouTube video

leave a comment »

Used parallelism in creating the video, both in a generating program, and in using the ppmtompeg utility.

 

Written by Robert Field

August 18, 2016 at 5:59 AM

Posted in Uncategorized

jQuery In Action, Third Edition, is out

leave a comment »

This very useful guide has been updated: Publisher’s web site.

Written by Robert Field

January 7, 2016 at 12:58 PM

Posted in Uncategorized

Just found one more tool for parallel execution

leave a comment »

Recently I found about about another tool for parallel execution, GNU Parallel.  .

Parallel can be invoked with arguments on the command line or in xargs mode. Here’s a simple example using the command line:

parallel -k echo ::: A B C

which runs three echo invocations in parallel. The output will be A, B, C, in uncertain order, since they run in parallel.

More later, as I dive into this interesting new tool

Written by Robert Field

March 23, 2015 at 5:47 AM

Posted in Uncategorized

A joke that I believe I thought up

leave a comment »

Give a man a fish, and you feed him for a day.

Teach a man to fish, and you’ll be driving him to the lake all summer.

Written by Robert Field

October 1, 2014 at 6:36 PM

Posted in Uncategorized

John Wooden said: Be quick but don’t hurry.

leave a comment »

He was a great basketball coach who brought the UCLA team to success for many years.

When you hurry, you make mistakes.  And when you are programming, you are under great pressure and it’s easy to hurry.

 

Written by Robert Field

June 13, 2014 at 6:41 PM

Posted in Tips And Tricks

Tagged with

Django links: tutorial and advice on code organization

leave a comment »

Nice looking Effective Django tutorial

Suggestions on avoiding a code mess at Fat Models – A Django Code Organization Strategy. Basic idea is to move the business logics to the models, not the views, helping to put code in logical places.

Written by Robert Field

February 1, 2014 at 4:41 PM

Usings xargs: another technique for parallel execution

leave a comment »

A while ago, I posted about using GNU Make to support parallel execution, and discussed a few other ways to manage parallelism.

Here is another technique, using the xargs utility.  The basic idea of xargs is to use a sequence of lines as standard input, and using those lines as arguments for a command to run against each argument.

A classic use of xargs is processing all the results of a find command, here used to set all .pl files under a directory to be writable.

find . -name "*.pl" | xargs chmod +w

where xargs fills up chmod‘s command line and runs the command. If there are more input lines than a command line can hold then chmod will be invoked more than once. There are some details for odd input that you might need to know, especially processing text where the parameters have white space, such as files with spaces in the path.

For parallel execution, you can use two parameters to xargs. They are -P, telling how many instances of the command line to run in parallel, and -L, telling how many arguments to process invocation of the command.

Inspired by example above, you might have

find . -name "*.pl" | xargs -L 1 -P 8 long_processing --ultra-complicated

which would allow up to 8 parallel executions (because my current best computer has 8 cores, that’s why) and one file name to each invocation of the long_complicated_processing program.

So if one of the files from find is ./AA/BB/CC/DD/mnop.pl, the corresponding execution would be

long_processing --ultra-complicated ./AA/BB/CC/DD/mnop.pl

A nice advantage of this method is that it is very easy to set up. You could actually type it directly in the command line, or in a shell script. You don’t have to know how any arguments will get processed, for example.

One drawback is error detection. If one of the command executions exits with non-zero status, xargs will keep on executing commands, and will show the error with its own non-zero exit code.

This may or may not be acceptable.

This is one of the most lightweight techniques for managing parallel execution and a good one to have in your bag of tricks.

If you haven’t used xargs before, it is worth learning, even if you aren’t interested in parallelism.

Written by Robert Field

July 14, 2013 at 11:40 AM