STAT 592/CSE 590, Winter 2004

Homework 1

• 1. The purpose of this Problem is to implement and experiment with linear discriminant analysis.
  • (a) Write functions within.class.ssp(X, y) and between.class.ssp(X, y) computing within-class and between-class covariance matrices. X is the predictor matrix and y is the response vector containing the class identities of the populations.
  • (b) Write functions lda.fit(X.train, y.train) and lda.predict(lda.model, X.test, q = min(p, K-1), priors = rep(1/K, K)) The idea is that lda.fit takes the training data and returns a S list lda.model containing such things as the eigenvectors of W^(-1)B and other quantities needed for prediction. The function lda.predict takes a model produced by lda.fit, the predictor matrix for a test sample, the number q of discriminant functions to be used for prediction, and the class priors, and returns a vector containing the predicted class identities for the test sample.
  • (c)
    (S) Here we use the vowel data (a file created using the S command data.dump). Documentation is in vowel.doc.txt. Using data.restore gives two lists, vowel.train and vowel.test. Each of these has an x component (as in vowel.train$x), and a y component.

    (ASCII) Or download ascii files with the vowel training and testing data.

    • What is the resubstitution error rate (APER) using all the discriminant functions and assuming equal priors?
    • Plot the eigenvalues. How many discriminant functions do you think should be used?
    • What is the resubstitution error rate for your chosen value of q?
    • What is the error rate for the test data using all discriminant functions?
    • What is the error rate for the test data using your value of q?

  • 2. The purpose of this Problem is to familiarize you with nearest neighbor classifiers.
    • (a) Write an S function

      knn.classifier(X.train, y.train, X.test, k.try = 1, pi = rep(1/K, K), CV = F)
      

      where

             X.train, y.train, X.test have the obvious meanings;
             k.try is a vector of neighborhood sizes
             pi is a vector of prior probabilities
             CV = T if cross-validation is to be used.
      
             CV = T only makes sense if X.train = X.test
      

      The function should return a (n.test x length(k.try)) matrix of predicted class identities for the n.test test observations and the different values of $k$ provided in k.try.
    • (b) Run the function on the vowel training data with

       k.try = c(1, 3, 7, 11, 15, 21, 27, 35, 43)
      

      Which of the spans (i.e. values of k) appears to be best? What is the resubstitution estimate of the risk for this span? What is the cross-validated estimate?
    • (c) Classify the vowel test data using the span that gave the best result on the training data. What is the estimated risk?

  • 3. The purpose of this Problem is to familiarize you with the S-implementation of Classification Trees.
    • (a) Build a classification tree for the vowel training data. Summarize and plot the tree.
    • (b) Estimate the optimal complexity parameter for the vowel data, using cross-validation. Remember to leave out one speaker at a time. What is the estimated optimal tree size? Comments? What is the resubstitution estimate of risk for the optimal tree?
    • (c) Use the optimal tree to predict the test sample. What is the test sample estimate of risk?

  • 4. This Problem completes the proof the Claim in page 65 of the notes, and is for those of you more inclined to theoretical reasoning.
    • Prove the Claim in page 66 of the notes:

            APER( c_n ) <= APER( f ) + ( d / n )
      

      for any linear classifier f.
      
      Hint: show that for each linear classifier f there exists an f_i, (data dependent linear classifier defined in pages 63-64 of the notes) that coincides on all data points, except for perhaps those defining the hyperplane associated to f_i, with f.