Analyze log files¶

Upload and analyze log files¶

1. This example assumes that logtest directory contains the following log files.

   2010-11-16-21_access.log
   2010-11-16-22_access.log
   2010-11-15-21_access.log
   2010-11-15-22_access.log
   

   Each file uses the following line format.

   Nov 15 21:53:52 lucid64 proxy-server - 127.0.0.1 15/Nov/2010/22/53/52 DELETE /v1/AUTH_cd4f57824deb4248a533f2c28bf156d3/2eefc05599d44df38a7f18b0b42ffedd HTTP/1.0 204 - \
    - test%3Atester%2CAUTH_tkcdab3c6296e249d7b7e2454ee57266ff - - - txaba5984c-aac7-460e-b04b-afc43f0c6571 - 0.0432
   

2. Change into the logtest directory:

   $ cd logtest
   

3. Upload the log files into the logtest container:

   $ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing upload logtest *.log
   

   2010-11-16-21_access.log
   2010-11-16-22_access.log
   2010-11-15-21_access.log
   2010-11-15-22_access.log
   

4. Get statistics for the account:

   $ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \
   -q stat
   

   Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
   Containers: 1
   Objects: 4
   Bytes: 5888268
   

5. Get statistics for the logtest container:

   $ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \
   stat logtest
   

   Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
   Container: logtest
   Objects: 4
   Bytes: 5864468
   Read ACL:
   Write ACL:
   

6. List all objects in the logtest container:

   $ swift -A http:///swift-auth.com:11000/v1.0 -U test:tester -K testing \
   list logtest
   

   2010-11-15-21_access.log
   2010-11-15-22_access.log
   2010-11-16-21_access.log
   2010-11-16-22_access.log
   

Download and analyze an object¶

This example uses the -o option and a hyphen (-) to get information about an object.

Use the swift download command to download the object. On this command, stream the output to awk to break down requests by return code and the date 2200 on November 16th, 2010.

Using the log line format, find the request type in column 9 and the return code in column 12.

After awk processes the output, it pipes it to sort and uniq -c to sum up the number of occurrences for each request type and return code combination.

1. Download an object:

   $ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \
     download -o - logtest 2010-11-16-22_access.log | \
     awk '{ print $9"-"$12}' | sort | uniq -c
   

   805 DELETE-204
   12 DELETE-404
   2 DELETE-409
   723 GET-200
   142 GET-204
   74 GET-206
   80 GET-304
   34 GET-401
   5 GET-403
   18 GET-404
   166 GET-412
   2 GET-416
   50 HEAD-200
   17 HEAD-204
   20 HEAD-401
   8 HEAD-404
   30 POST-202
   25 POST-204
   22 POST-400
   6 POST-404
   842 PUT-201
   2 PUT-202
   32 PUT-400
   4 PUT-403
   4 PUT-404
   2 PUT-411
   6 PUT-412
   6 PUT-413
   2 PUT-422
   8 PUT-499
   

2. Discover how many PUT requests are in each log file.

   Use a bash for loop with awk and swift with the -o or --output option and a hyphen (-) to discover how many PUT requests are in each log file.

   Run the swift list command to list objects in the logtest container. Then, for each item in the list, run the swift download -o - command. Pipe the output into grep to filter the PUT requests. Finally, pipe into wc -l to count the lines.

   $ for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester \
    -K testing list logtest` ; \
           do  echo -ne "PUTS - " ; swift -A \
           http://swift-auth.com:11000/v1.0 -U test:tester \
           -K testing download -o -  logtest $f | grep PUT | wc -l ; \
       done
   

   2010-11-15-21_access.log - PUTS - 402
   2010-11-15-22_access.log - PUTS - 1091
   2010-11-16-21_access.log - PUTS - 892
   2010-11-16-22_access.log - PUTS - 910
   

3. List the object names that begin with a specified string.

4. Run the swift list -p 2010-11-15 command to list objects in the logtest container that begin with the 2010-11-15 string.

5. For each item in the list, run the swift download -o - command.

6. Pipe the output to grep and wc. Use the echo command to display the object name.

   $ for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester \
    -K testing list -p 2010-11-15 logtest` ; \
           do  echo -ne "$f - PUTS - " ; swift -A \
           http://127.0.0.1:11000/v1.0 -U test:tester \
           -K testing download -o - logtest $f | grep PUT | wc -l ; \
         done
   

   2010-11-15-21_access.log - PUTS - 402
   2010-11-15-22_access.log - PUTS - 910