[SATLUG] xargs performance with python

Donald L Wilcox makiten at neonnightrider.com
Tue Feb 5 20:25:56 CST 2013


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey everyone,

I'm new to xargs and awk, and I've been trying to use it with wget to
download tens of thousands of XML files that I'll later have to parse
with lxml.

When I use xargs with wget and awk on my VM, I don't see any
performance changes. I was doing some research on xargs and apparently
it's designed for dealing with a lot of arguments and parameters, but
I don't know what constitutes "a lot."

My command is something like this:

awk -F: -v data_dir=$TMP_PATH '{print "xargs -n3 -P16 wget -q -O
"data_dir"/datafiles/"$0".xml
http://website.com/GetPlatform.php?id="$0}' $TMP_PATH/datafiles/ids.txt

Is there's a more efficient way to read a file line by line
and use wget and xargs outside of awk to get these xml files? I've
tried other approaches, but they don't work.

- -- 
__________________________________________________________________
Donald Wilcox        Web: http://www.neonnightrider.com
San Antonio, TX LinkedIn: http://www.linkedin.com/in/donaldwilcoxjr
__________________________________________________________________
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJREb8zAAoJEBq7PHS91y7ha7cIAIVohna3iFvKsJ53HuDiYwzU
fKnvXk04bougHxuzn6VsKVqmwQyd5w+9Ngaqvh4buYXtfUfE3Tj7z99aWiBZGrfO
Mu6J6hImbIQQmJOOyfylA3IQraRM15Srz+XkX2SiyKAOAq/kIJh0SI36cxN9qvKn
FQvmXc4gnYObb/FyLIn7dOxAInTfc5T6E0ULS1+CJ4VFhWuml9SpilrqmUJ1W7Ah
UR28MVzc7iJ1yEGv9ZTwptuLMfvGhYPDkvG2ljPv9SzWhAlfU1n273iR9uYpCWNA
CimWb6eCf4d32vw7X+YDy8zMkq6TzO5BbsRV93FK3XBqK29zJKQGsTRdSF3Tan4=
=+Xvh
-----END PGP SIGNATURE-----


More information about the SATLUG mailing list