Archive for April, 2009

Easter log analysis

April 14, 2009

Here is a look at the logs for typical “holiday” day on a busy webserver. Here are the sections of the report created by FireGen:

1. Web traffic (HTTP/HTTPS) – Internal users (outbound connections) and Web traffic (HTTP/HTTPS) – visited sites

This is a webserver and normally there should be no users browsing the Internet from it and no websites visited. However, the report contained at least 30 connections, mostly towards IP addresses belonging to Microsoft. The IPs did not have a host name associated but the FireGen builtin Whois query identified them as being Microsoft IPs. So what happened? Well, in turned out that an administrator logged in remotely that day. Even though he did not make any updates or opened any websites, the Automatic Update option connected to Microsoft and looked for new hotfixes/updates for that version of Windows. All done in the background – if the automatic search/download of updates is a problem, it can be disabled.

2. Web traffic (HTTP/HTTPS) – incoming connections

This being a popular website, one can expect heavy traffic even during a holiday like Easter. A lack of traffic would indicate a problem. What should worry would be a large number of connections from the same IP. In this particular case, we could see many connections from various web crawlers (Yahoo, Google, Cuill, MSN) and this can be expected. One can control these web robots by configuring the proper robots files on the websites themselves (telling the robots what to search and what NOT to search). Given the emphasis on generating web traffic usually one would want to encourage this type of indexing. The number of connections would depend on the popularity of the website.

We found one IP, 84.14.247.2 (2.247-14-84.ripe.coltfrance.com) belonging to an ISP in France with a large number of connections. Usually this could indicate an abuse but IP itself, ending in .2 does not look like the typical IP assigned to an end user but, a network device. It is very possible that this is a proxy server trying to perform some proactive caching.

3. Email (SMTP) – Top 50 outbound connections

This webserver sends a certain number of emails on regular basis so seeing some traffic there is no surprise. Only a sudden increase in this type of traffic may indicate that somehow this server is used to relay spam but was not the case here.

4. Email (SMTP) – Top 50 inbound connections

This server is not accepting incoming SMTP traffic so there should be no connection here. However, two were listed in the report. The first one was from a server in Holland but it looked like an accidental connection (most probably our server sent an email to a recipient on that computer) and the second one from a server in Taiwan (an ISP). The IP was not dynamically assigned so chances are that again it was accidental. The question is why this SMTP traffic made past our firewall, so one question was sent to the firewall administrator to look into this.

5. Email clients (POP3/IMAP)

There were no entries under this section (a good thing!)

6. Custom protocol 1 – MS RPC (TCP/135)

We monitor this as the Microsoft RPC is a popular target for worms and other malware. It gives as an indication on how “hot” things are outside the firewall. Most of the connections listed under this section came from China. Were they just bots scanning the Internet for vulnerable hosts or attacks targeting directly our website? For most of them there was just one connection attempt so chances are it was just a scan. There were also several connections from US, probably just infected PCs.

7. Custom protocol 2 – NetBIOS (UDP/137)

Similar with MS RPC, NetBIOS is something one should keep his eyes on. Several connections were listed FROM our webserver against various IP addresses. We know actually that these were part of the FireGen attempt to reverse resolve the host name for various IP addresses. If there Whois does not return a host name, the reverse DNS module will attempt a NetBIOS query against that IP (sometimes the remote host responds with its name).

8. Custom protocol 3 – Whois (TCP/43)

This section shows the queries made by Firegen. Very few administrators would need this.

9. Other protocols

This is an interesting section as it shows all the other protocols that were not specifically entered in the Monitor Protocols FireGen configuration. We could see the following:

TCP/8010 – This is used by our Whois engine to determine the geographical location of various IP addresses

UDP ports higher than 1024 – Microsoft DNS sometimes uses these type of ports

UDP/123 – Network Time Protocol – used by the webserver to synchronize its clock

UDP/500 – Several probes from outside hosts, probably looking for VPN devices

So nothing really unusual in this section (and that is good).

10. Protocols – Top 50

This is the summary of protocols identified by FireGen. TCP/80 – http counted for 98.13%, as expected. In a normal day this would be even higher.

11. Internal IP addresses

This indicates the traffic generated from internal IP addresses. One should make sure that only the authorized IPs are listed there and that was the case in this report.

12. External IP addresses

This shows the traffic generated by external IPs. As expected, the web crawlers from Google, Yahoo and MSN are the top generators of traffic. Unless the bandwidth is limited, this is a good thing as the website gets published in the search engines. One should be worried if there is no traffic from such crawlers.

13. Total traffic by hour

This should indicate a smooth distribution of traffic, no sudden spikes.

14. Traffic and denials by hour

This graph can be used to see if there was a disproportionate number of denials compared with the regular traffic. They should be proportionate as the more traffic one gets, the more denials will be recorded. There will always be denials, some due to accidents, some due to attempts from bots and hackers.

15. Denied connections / Denied protocols / Denied IPs

This can be used to see the most persistent offenders. However, some connections are being denied due to the unreliable nature of Internet. if a legit but stray packet reaches the firewall too late, it will be denied – this type of denials are listed with “No service” as reason for denial.