Used parallelism in creating the video, both in a generating program, and in using the ppmtompeg utility.
This very useful guide has been updated: Publisher’s web site.
Recently I found about about another tool for parallel execution, GNU Parallel. .
Parallel can be invoked with arguments on the command line or in xargs mode. Here’s a simple example using the command line:
parallel -k echo ::: A B C
which runs three echo invocations in parallel. The output will be A, B, C, in uncertain order, since they run in parallel.
More later, as I dive into this interesting new tool
Give a man a fish, and you feed him for a day.
Teach a man to fish, and you’ll be driving him to the lake all summer.
He was a great basketball coach who brought the UCLA team to success for many years.
When you hurry, you make mistakes. And when you are programming, you are under great pressure and it’s easy to hurry.
A while ago, I posted about using GNU Make to support parallel execution, and discussed a few other ways to manage parallelism.
Here is another technique, using the
xargs utility. The basic idea of
xargs is to use a sequence of lines as standard input, and using those lines as arguments for a command to run against each argument.
A classic use of
xargs is processing all the results of a
find command, here used to set all
.pl files under a directory to be writable.
find . -name "*.pl" | xargs chmod +w
xargs fills up
chmod‘s command line and runs the command. If there are more input lines than a command line can hold then
chmod will be invoked more than once. There are some details for odd input that you might need to know, especially processing text where the parameters have white space, such as files with spaces in the path.
For parallel execution, you can use two parameters to
xargs. They are
-P, telling how many instances of the command line to run in parallel, and
-L, telling how many arguments to process invocation of the command.
Inspired by example above, you might have
find . -name "*.pl" | xargs -L 1 -P 8 long_processing --ultra-complicated
which would allow up to 8 parallel executions (because my current best computer has 8 cores, that’s why) and one file name to each invocation of the
So if one of the files from
./AA/BB/CC/DD/mnop.pl, the corresponding execution would be
long_processing --ultra-complicated ./AA/BB/CC/DD/mnop.pl
A nice advantage of this method is that it is very easy to set up. You could actually type it directly in the command line, or in a shell script. You don’t have to know how any arguments will get processed, for example.
One drawback is error detection. If one of the command executions exits with non-zero status, xargs will keep on executing commands, and will show the error with its own non-zero exit code.
This may or may not be acceptable.
This is one of the most lightweight techniques for managing parallel execution and a good one to have in your bag of tricks.
If you haven’t used
xargs before, it is worth learning, even if you aren’t interested in parallelism.