finding unique ips in access log that have something in common

search for a string in access log, extract only ip address from matching line, sort the list of ip's and remove duplicates, output the [shortened] list...

for ip in `grep -i firefox /cygdrive/w/resin/log/access.log \
  | grep -o "^[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}" \
  | sort -u \
  | wc -l`; \
  do echo $ip; \
done

perl urlencode without libs

$url = "Foo bar com.dx?q=23";
$url =~ s/([^\w\-\.\@])/$1 eq " "?"+":
  sprintf("%%%2.2x",ord($1))/eg;
print $url;

I don't remember where I got this, but it must have come in handy for me at some point. Never hurts to have this kind of stuff laying around, just in case.

Generating a random fake name from the commandline

Today I needed to come up with a list of lots of fake names for test data. In the past, I either manually entered well-known fictional character names (e.g. Homer J. Simpson), or used strings of characters (like 'asdf g. hjkl' or 'aaaaaaaaaaaaaa').

I remembered seeing some sort of test data generator, somewhere, recently, so I googled for "fake name generator test data".

What I found was http://www.fakenamegenerator.com, which generates realistic test data based on country, name origin, and gender specifications. More info on the site at the end of this post. Here's a set of commands that I put together to retrieve one fake name from that site:

curl -s -b agreement=Yes "http://www.fakenamegenerator.com"
    | grep -o 'on Google">\([^<]\+\)'
    | sed -e "s/[^>]*>\([^<]\+\)<.*/\1/g"

When you use the site like a normal human being, the fake data that is generated includes full name, address, email address (usable, provided by an anonymous email service), phone number, mother's maiden name, date of birth, and credit card number (+ expiration date). Very cool! For a very small fee, you can also order a bulk batch of data, which also includes fake Social Security Numbers).

However, being the penny-pinching and geeky type, I wanted to be able to generate my own list of fake names (without all the other info), for free. The set of commands listed above work right now, from a cygwin bash shell, but will probably break sometime in the future, when the HTML structure of the page changes. Oh well.

Oh yeah, don't forget to read their terms of service* before using the service... Right now, I could not find anything prohibiting the use of automated tools to generate and retrieve names, but use the above set of commands at your own risk!

* The terms of service page only displays one time, unless you clear/disable your cookies.

Using ImageMagick and 4NT to find jpegs by quality level

Using ImageMagick and 4NT to find and modify all jpegs in the current dir who have quality level 94:

for %f in (*.jpg) do (echo %f
    & identify -verbose %f | grep Quality
    | grep -o "[0-9]*")

iff "%@execstr[identify -verbose %f
    | grep Quality | grep -o "[0-9][0-9]"]" == "94"
    then & echo %f & endiff

for %f in (*.jpg) do
    (iff "%@execstr[identify -verbose %f
    | grep Quality | grep -o "[0-9][0-9]"]" == "94"
    then & echo %f &
    mogrify -strip -quality 84 %f & endiff)

---

Unrelated side note:
This is one of several posts that I will be making in the near future, which come directly from my ever-growing toblog.txt file... No code cleanup or lengthly explanations, just commands I've run at some point that I thought were worth saving.

Bash history substitution

Anyone who has had an introductory unix course should know about the bash shell's "history" command, which gives you a numbered list of commands that you've run previously.

You can execute one of those commands again by doing:
$ ![number] (for a particular command you've seen on the history list)
or
$ !! (for the previous command/last command in the history list)


One thing that I didn't learn in any class, but did find out about from a co-worker, several years ago, was history substitution. It's easy to run the previous command with minor changes, by using a caret-delimited substitution expression. Here's a very simple example:

view a file:
$ cat spugbrap.txt

then, edit that same file:
$ ^cat^vim^

that gets expanded to (and executed as):
$ vim spugbrap.txt


I've used this feature countless times since learning about it, but it always suffered from a limitation: If the pattern you're trying to match occurs multiple times in the previous commandline, this subtitution method only replaces the first occurrance of it. So, I recently decided to find out how to substitute multiple occurances, since I was sure there had to be an easy way. Here's one way I found:

watch a couple of tomcat log files continuously:
$ tail -F ~/tomcat/logs/stdout_20060704.log ~/tomcat/logs/stderr_20060704.log

The next day, the log file names are different, because they're date-based. so I want to change 20060704 to 20060705, and it needs to happen twice because the date occurs twice in that commandline. No problem! Assuming the previously executed command was the "tail" commandline, above, simply enter this:
$ !!:gs/20060704/20060705/

that gets expanded to (and executed as):
$ tail -F ~/tomcat/logs/stdout_20060705.log ~/tomcat/logs/stderr_20060705.log


What if my previous command was long, and I need to make multiple substitutions with multiple strings?

previous command:
$ tail -F stdout_20060704.log stderr_20060704.log host-manager.2006-07-04.log catalina.2006-07-04.log admin.2006-07-04.log localhost.2006-07-04.log manager.2006-07-04.log jakarta_service_20060704.log

to change the dates, which occur in two formats:
$ !!:gs/20060704/20060705/:gs/2006-07-04/2006-07-05/


If the command I want to run (with substitutions) was not the previous command, but some other command that appears in the numbered list from running 'history', put the history line number between the exclamation point and the colon:
$ !123:gs/oldstr/replacementstr/


For more information about this, and other bash history manipulation capabilities, check out:
Bash Features - Using History Interactively