R at Loyalty Matrix

Data Mining with R

Luís Torgo has posted a draft of the first three chapters of his book Data Mining with R: learning by case studies along with data and code files.

For those of us who learn best by doing, this is a great introduction to the potential use of R as a data mining tool kit.

Getting Started with R - Part 1

In these getting started notes, I'm assuming you are a business user in the US. Thus, I'll be directing you to the US CRAN mirror for downloads and I'm assuming you are using windows. If you are based elsewhere, please use your local CRAN mirror. R also runs under UNIX, Linux and MacOS. See your CRAN homepage for these other distributions.

Downloading R

First, download the current version of R for Windows (rw?????.exe) to a place you can find it. Your desktop works just fine. You may want to browse the Read Me whilst the setup file is downloading. Once downloaded, simply run the set-up file by double-clicking on it. A typical Windows setup dialog will start. Accept the default settings, except do check the option to create a short-cut on your desktop. I have never had any problems with the install, so hopefully all has gone smoothly. If it has, you may want to delete the setup download.

Trying R

Let's now give R a simple test. Start R by double clicking on the shortcut created by the setup process. R will now start. You should see an empty "RGui" window. After a moment the welcome text will appear followed by the R command prompt: ">".

OK, let's try your first R command! Type
1+1
followed by an Enter.

Now for something more exciting. Type the two lines:
x <- rnorm(1000)
hist(x)

The first line creates a 1000 element vector, x, filled with normally distributed random numbers. Then you asked R to plot the histogram of x.

Enough excitement!

Help with R

Now I want you to look the extensive help system that comes with R. R help is available in four forms:

1) Well formatted PDF files that you can read on-line or printed out. One of the great things about the R project is the completeness of the R documentation. (If you don't have an Acrobat reader on your PC, now would be a good time to install one from the Adobe site).

From the Windows menu bar look at: Help|Manuals|An Introduction to R

This 100 page manual is one place to start learning R. Read, at least, Chapter 1 and work through the sample session in Appendix A. Chapters 2 through 12 give a rather complete, if somewhat terse, overview of R.

2) The same content is also in HTML (browser) form: Help|Html Help
with some additional links.

3) FAQ's from Help or on CRAN (which will typically be more up to date than your local copy)

4) Interactive help by typing ?function. Try typing "?hist" at the ">" prompt.

There are actually a few more ways to get help, but this is enough for now.

Quiting R

Quitting R is easy. Just type "q()" or, in Windows, select File|Exit or click the top left "X" close icon. Yes, go ahead and quit. R will ask "Save workspace image?" Reply "No" for now. Often you would want to save your work!

Setting up your Workspace

R keeps its files in its working folder. By default, R uses the install folder in "Program Files." It's a good idea to keep your various projects separated by setting up different folders for each. My personal convention is to have a set of folders like: C:\Data\LearnR\R, where "LearnR" could be your first project.

In Windows, it's easy to set up a short-cut on your desktop for each project by

1) Create your working folder. Mine would be: "C:\Data\LearnR\R"
2) Copy the default R short cut on your desktop and paste a copy of it back on the desktop.
3) Rename the copy, for example, to "Learn R"
4) Right click on your new shortcut and open the Properties dialog.
5) The "Start in" entry will be something like: "C:\Program Files\R\rw1090". Change it to your working folder. Mine is: "C:\Data\LearnR\R"

Test your new short cut!

R should open as before. Open the Change Directory dialog with the menu bar: File|Change dir... to verify that R is starting where you intended. If it is not, quit R and check your short cut properties.

Type "ls()" to list the objects in your workspace. R should reply "character(0)", which is an empty character vector. Now type "x <- rnorm(1000)" as above. Now when you type "ls()", R should know about your newly defined vector x.

Quit R and take a look in your working folder. You should see two files .RData and .Rhistory. The next time you start R as above, R will load these so you can start again just where you left off!

Next Steps

Try doing some R. Try Appendix A in the Introduction to R. CRAN's Contributed Documentation page has a number of short getting started guides which I found helpful in the beginning.

I would also recommend starting with one, or both, of the following books:

Peter Dalgaard's Introductory Statistics with R which was my first R book. Peter starts with a 40 page introduction to R which I found more novice friendly than the R docs. This 250 page book covers a lot of ground going beyond the typical introductory stats text. Very practical with a lot of R examples.

John Maindonald and John Braun's Data Analysis and Graphics Using R - An Example-based Approach The title, almost, says it all. Somewhat longer (350 pages) with more graphics emphasis but less on R as a language. Also has a lot of R examples - interestingly in footnotes so the code does not "disrupt" the flow of the text. Solid data analysis, including multi-level models, tree-based classification/regression and multivariate exploration & discrimination.

(Remember books purchased through this blog benefit the R Foundation.)

Have Fun!

useR! Photos

Fritz Leisch has put his photos of the conerence up on the useR! site.

Achim Zeileis & I talking at the opening reception. Achim (on left) did a great job organizing the program.

Jim's Presentation at useR!

Doing Customer Intelligence with R

Here is the PDF of the slides I presented Thursday at the useR! conference. This is the verion I "talked" through so it is not self explanitory.

I'll work on an annotated version on the plane which should be more freestanding.

1st Real Test of R Server

I put our dedicated R Server (AKA Redtail) to a real test over the weekend doing some sizable randomForest runs. Problem was supervised learning with 2 categories. Vehicle purchase data + rich demographic info. For ~ 480 cases Redtail was ~2x faster that my desktop. (15 vrs 30 min)

Then I ran ~ 8k cases in 4 hours so time is ~ linear with # cases. This certainly would be a no-go on the desktop because of RAM lmitations.

Redtail (Loyalty Matrix names all servers after beer. Red Tail is a very nice ale from Mendocino in California.) is a fast 64-bit AMD box with 2G RAM. (I'll add real specs when I get back to the office)

Redtail will be the engine for R sessions within our MatrixOptimizer application.

Off to Vienna!

I’m heading out to the airport. I’ll be landing in Vienna early afternoon on Wednesday in time for the reception at the University.

Hopefully, there will be some time to post updates during the conference. Stay tuned!

Help the R Foundation by Ordering Books Here

When you order books through the links on this blog to Cody's Books online, Loyalty Matrix will send all referral fees to the R Foundation immediately upon receipt.

Loyalty Matrix has been a supporting member of the R Foundation since 2003. We depend on R to help us deliver Customer Intelligence insights to our clients. Hosting this weblog is one way we can give something back to the R community.

Read Me 1st

The very least you need to know about this weblog...

Purpose
To promote and support the use of R, the open source data analysis & graphics software system, as the tool of choice for marketing analytics in general and for customer intelligence in particular.

We have purposely chosen the weblog format to encourage the free exchange of ideas by informal and timely postings. Please contribute. Ask questions. Add comments.

See the Welcome and Open Source Customer Intelligence posts for more detail.

Navigation Hints
Left sidebar will remain fairly static. New R books will be added, of course, as they are published.

Posts, like the one you are reading now, are in the center column and are sorted most recent first.

Within each post you may find:
1) Clickable links to other posts in the blog or elsewhere on the web.
2) Clickable links to files uploaded to the blog.
3) Clickable thumbnails for diagrams and/or pictures

The footer below each post shows:
1) Author & Date Time stamp
2) Category to which post is assigned. Click on it to see posts all in the same category.
3) Permalink is the static link (URL) for the post. Use it when setting up links back to this post.
4) Comments(n) shows number of comments on this post. Click on it to see the comments and the form add your own comments

Right sidebar is your navigation friend!
1) The calendar shows days with posts. Click on a day to see only postings on that day.
2) "Categories" show four categories I've defined (thus far). Click on a category to see only the posts on that topic.
3) "Recent Posts" and "Recent Comments" each list 10 most recent of each.
4) "Archives" are set up by month. Click on a month link to see only posts done in that month.

Etiquette
All visitors are welcome to comment on any post as long as comments are relevant and observe the usual code of behavior. The blog administrator reserves the right to edit or delete any posting. Habitual offenders will be blocked from commenting.

Guest Authors
If you would like actively participate in this blog, especially the “Open CI” initiative, please email me and asked to be added to the author list.

Legal stuff
Posts on this weblog are the individual contributions and opinions of the posters. They do not speak for Loyalty Matrix, Inc., even when the poster may be a Loyalty Matrix employee. The only official Loyalty Matrix policy stated here is: Loyalty Matrix, Inc. will forward all referral fees received for book sales through the links on this site to the R Foundation immediately upon receipt.

When suggestions, methods, and actual code is posted it is done so as an informal contribution to the community. Neither Loyalty Matrix, Inc. nor the individual contributors assume any responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.

Welcome

Welcome to Loyalty Matrix's weblog devoted to R. Here is a brief introduction to R and how we use it.

Overview
The statistical computing, data analysis & graphics software package R has become an important part of our analytic toolkit. The purpose of this memo is to introduce R to those of you who are not familiar with this powerful resource.

R Basics
R is the free open source version of S – a statistical language developed by John Chambers at Bell Labs in the 1980’s. The ACM System Software Award received by John Chambers in 1998 stated that S “… has forever altered the way people analyze, visualize and manipulate data…”

R is an open and cross platform system. It has pre-compiled binaries for easy installation on Windows, MacOS and various Linux implementations. The sources can be easily compiled, I’m told, for various flavors of UNIX. Since R is (mostly) written in R, it is easily extended.

R source, executables, documentation, newsletters, and more is available at www.r-project.org/ and at various mirror sites around the world. Here, in the USA, the preferred site is cran.us.r-project.org/.

Loyalty Matrix and R
Loyalty Matrix discovered R early in 2003 while doing analysis work for several of our clients who were looking for a high value solution with low upfront costs. Since the analytical needs of our clients were very specific to customer data analysis, sponsoring a large budget for one of the conventional commercial packages like SAS or SPSS was not feasible. As a result, we were tasked with finding a viable alternative, for which R definitely fit the bill, literally.

While R has been around since the early 1990’s (and released under GNU GPL in 1995), it has really become popular in the last couple of years. Version 1.0.0 was officially released in February 2000. Major revisions generally happen twice a year. The current version is 1.9.0.

Just like SAS and SPSS, R is essentially a programming language fine-tuned for statistical analysis, data mining, and data visualization. However, since it is open-source, it has an extremely fast speed of optimization because of the size and caliber of its contributor community. All the required functionality for customer data analysis (such as scatter plots, histograms, cluster analysis, decision trees, time series analysis, CHAID, CART, ANOVA, to name a few) are available in R. Furthermore, with its advanced data visualization capabilities, we found a lot of our traditional SAS and SPSS analysts opting for R.

Since R is essentially S-language compatible, the extensive S code-base, providing all the basic analytic and graphic methods, gave the R project a significant head start. Nowadays, many new packages are released first on R and then picked up by the S community. We were immediately attracted to R by its strong and flexible graphic tools, wealth of exploratory data analysis methods, and general activity around the R project.

The R movement is continuing to expand. The R Foundation was created in April, 2003 to give a formal focal point for the project (modeled after the Apache and GNOME Foundations). Founding organizations include the Institute of Mathematical Statistics (IMS), MedAnalytics, Baxter AG, Johns Hopkins University and the University of Wisconsin. Loyalty Matrix became a “Supporting Institution” of the R Foundation in 2003.

Cutting edge institutions, such as Stanford, now use R in their data mining courses rather than traditional commercial packages such as SAS or SPSS.

The first useR! (annual user conference) will be held in Vienna in on May 20-22, 2004. “Data Mining and Large Databases” will be one featured topic.

R Resources
The CRAN link is the place to start. In particular, the Windows install file is there. Note PDF’s of the extensive standard documentation set are part of the standard installs and available separately on CRAN. PDF’s of the all issues R News, the R newsletter, various lecture notes/handouts, and documentation for the nearly 300 add-on packages are also on CRAN. The R-Help mail list is extremely active with 50 + messages on many days.

Conclusion
Loyalty Matrix uses R as a core element of our analytic toolkit leveraging its wide range of methods for exploratory data analysis (EDA), presentation plots, and rigorous statistical algorithms.

Over the next few months, we will be coding our models for customer intelligence analysis in a set of formal R packages. These will become part of our MatrixOptimizerTM suite, which is our product framework to derive marketing insights from customer data. The overall goal is to utilize R to improve the efficiency and effectiveness of our processes and insight development techniques. We will also release these models into R’s open-source community so that we can invite leading world-class statisticians and data mining experts to critique and, hopefully, contribute to these customer intelligence packages. Our goal is to share our expertise and spearhead a cooperative “Open Source Customer Intelligence” project within the R community.