In: Computer Science
1. Create a Python module that reads Linux_syslog_ip_parser.py that reads syslog file and parses the IPV4 addresses and stores the result into a csv file. Your module should contain the following functions:
a. linux_logfile_reader(filename) function that reads the syslog file and returns all IPV4 addresses
b. count_ipfrequency(ip_list) function that accepts the list of IPV4 addresses retuned by linux_logfile_reader function and returns a dictionary list with ip:frequency pair.
c. Writeto_csv_file(ip_frequency) function that accepts a dictionary list containing ip:frequency pair returned from count_ipfrequeny function and stores the IpV4 addresses with its frequency value into a ‘ipv4.csv’ file.
i. Which internal IP address is the most frequency found in the ‘syslog’ file ( hint: internal ip addresses start with either 10, 172, or 192)
ii. Which external Ip address is the most frequently found in the ‘syslog’ file
Example From File.
83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET
/presentations/logstash-monitorama-2013/images/kibana-search.png
HTTP/1.1" 200 203023
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:43 +0000] "GET
/presentations/logstash-monitorama-2013/images/kibana-dashboard3.png
HTTP/1.1" 200 171717
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:47 +0000] "GET
/presentations/logstash-monitorama-2013/plugin/highlight/highlight.js
HTTP/1.1" 200 26185
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:12 +0000] "GET
/presentations/logstash-monitorama-2013/plugin/zoom-js/zoom.js
HTTP/1.1" 200 7697
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:07 +0000] "GET
/presentations/logstash-monitorama-2013/plugin/notes/notes.js
HTTP/1.1" 200 2892
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:34 +0000] "GET
/presentations/logstash-monitorama-2013/images/sad-medic.png
HTTP/1.1" 200 430406
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:57 +0000] "GET
/presentations/logstash-monitorama-2013/css/fonts/Roboto-Bold.ttf
HTTP/1.1" 200 38720
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:50 +0000] "GET
/presentations/logstash-monitorama-2013/css/fonts/Roboto-Regular.ttf
HTTP/1.1" 200 41820
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:24 +0000] "GET
/presentations/logstash-monitorama-2013/images/frontend-response-codes.png
HTTP/1.1" 200 52878
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:50 +0000] "GET
/presentations/logstash-monitorama-2013/images/kibana-dashboard.png
HTTP/1.1" 200 321631
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:46 +0000] "GET
/presentations/logstash-monitorama-2013/images/Dreamhost_logo.svg
HTTP/1.1" 200 2126
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:11 +0000] "GET
/presentations/logstash-monitorama-2013/images/kibana-dashboard2.png
HTTP/1.1" 200 394967
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:19 +0000] "GET
/presentations/logstash-monitorama-2013/images/apache-icon.gif
HTTP/1.1" 200 8095
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:33 +0000] "GET
/presentations/logstash-monitorama-2013/images/nagios-sms5.png
HTTP/1.1" 200 78075
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:00 +0000] "GET
/presentations/logstash-monitorama-2013/images/redis.png HTTP/1.1"
200 25230
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:25 +0000] "GET
/presentations/logstash-monitorama-2013/images/elasticsearch.png
HTTP/1.1" 200 8026
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:59 +0000] "GET
/presentations/logstash-monitorama-2013/images/logstashbook.png
HTTP/1.1" 200 54662
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:30 +0000] "GET
/presentations/logstash-monitorama-2013/images/github-contributions.png
HTTP/1.1" 200 34245
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:53 +0000] "GET
/presentations/logstash-monitorama-2013/css/print/paper.css
HTTP/1.1" 200 4254
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:24 +0000] "GET
/presentations/logstash-monitorama-2013/images/1983_delorean_dmc-12-pic-38289.jpeg
HTTP/1.1" 200 220562
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:54 +0000] "GET
/presentations/logstash-monitorama-2013/images/simple-inputs-filters-outputs.jpg
HTTP/1.1" 200 1168622
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:33 +0000] "GET
/presentations/logstash-monitorama-2013/images/tiered-outputs-to-inputs.jpg
HTTP/1.1" 200 1079983
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:56 +0000] "GET /favicon.ico
HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77
Safari/537.36"
24.236.252.67 - - [17/May/2015:10:05:40 +0000] "GET /favicon.ico
HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64;
rv:26.0) Gecko/20100101 Firefox/26.0"
The following is the complete code:-
import re
def linux_logfile_reader(path):
ip_list = []
with open(path, 'r') as f:
for line in f.readlines():
ip_list.append(
re.findall("\d+\.\d+\.\d+\.\d+", line)[0]
)
f.close()
return ip_list
def count_ipfrequency(ip_list):
count = {}
for i in ip_list:
count[i] = count.get(i, 0) + 1
return count
def Writeto_csv_file(ip_frequency_pair):
with open("ipv4.csv", "w") as f:
for key in ip_frequency_pair.keys():
f.write("%s,%s\n"%(key, ip_frequency_pair[key]))
f.close()
ip_list = linux_logfile_reader('sys_log.txt')
print(ip_list)
ip_frequency_pair = count_ipfrequency(ip_list)
print(ip_frequency_pair)
Writeto_csv_file(ip_frequency_pair)
1) linux_logfile_reader
This function is dependent on the re package. We open the path of the text file that has been passed into the function. We read the file line by line and append all the values that match the pattern digits.digits.digits.digits. Once we iterate through each line in the syslogs we return the list with the IP strings.
2) count_ipfrequency
This function takes the IP list we created in the last function and then converts it into a dictionary with IP addresses as the key and the occurrence of that IP as the value.
3) Writeto_csv_file
This function opens the file ipv4.csv but using the write method mentioned by w. We iterate through the dictionary and store the key and its count in the csv.
In order to answer the i and ii questions, you can do that by running the code on the actual data and then analysing the csv.
I hope this answered your queries. If any further doubts do let me know in the comments I will clear them out by updating the answer. Regards.