Web/proxy Analysis

Web/proxy Analysis is a set of tools created by Richard Lees that automatically import web and proxy logs into a SQL database and processed to an OLAP cube.  The information is then available for fast and flexible analysis/reporting. 

Because all web servers log in w3c format, web/proxy Analysis supports all web servers. The only limitation for us, is what fields you are logging. For a reference case read the RugbyWorldCup web analysis case study.  This site had 500 Million hits over 5 weeks, with a busy hour having more than 10 million hits, which could be incrementally added to the OLAP cube in 15 minutes.  With web analysis the IRB (International Rugby Board) was able to quickly ascertain which countries were promoting the World Cup and determine where they needed to adjust their marketing initiatives.  They could do this from their head office in Ireland or locally in Sydney, viewing information within 15 minutes of the web servers cutting their logs.

For a live demonstration of web/proxy Analysis on real data, go to Demonstrations and follow the web analysis links. 

The Business Problem

Organisations are increasingly offering services, internally and externally, via Intranet and Internet architectures.  Important needs now include

· What activity is there on the site, by hits, users, visits etc?

· What pages are most popular?

· Which countries and cities are users coming from?

· Which pages have a slow processing or delivery time?

· Which pages are bloated (large in size and slow over a network)?

· What is the average response time for varying user groups or resources?

· Which users have seen a particular document?

· What are the trends in activity on your site?

· Where are users coming from?  (web referrals)

· What are the start and finish pages? 

· Which users consume the most proxy resources?

· What sites are most popular for our staff?

· Who is going to the naughty sites?

· Who is spending the most time surfing the internet?

 Ad Hoc Analysis Capability

Analysis is available by any combination of

· Client Agent

· Client OS

· Client Country and City

· Date (year, month and day)

· Day of Week

· Entry and/or Exit Page

· · Hour of Day

· HTTP Version

· IP Address of Client (by octet)

· IIS Directory

· IIS Operation

· IIS Resource (by type, eg. Company_Report.pdf)

· IIS Server

· IIS Status

· Referring Server and Page

· User (domain, user and session)

· Win32 Status

The report/query can measure

· Hits

· Request Count

· Bytes Transferred

· Average Bytes per Request

· Average Time per Request

· Maximum Request Time

· Visit Count

· Distinct Users

· Session count

 Performance

Typically, every request, ad hoc and standard reports, are satisfied in less than 2 seconds.

 Security

ThinSlicer works in the security context of the user on Internet Explorer, only your authenticated users are permitted to see your information.