Question

The Tokenizer.java file should contain:

A class called: Tokenizer

Tokenizer should have a private variable that is an ArrayList of Token objects.

Tokenizer should have a private variable that is an int, and keeps track of the number of keywords encountered when parsing through a files content.

In this case there will only be one keyword: public.

Tokenizer should have a default constructor that initializes the ArrayList of Token objects Tokenizer should have a public method called: tokenizeFile

tokenizeFile should have a string parameter that will be a file path tokenizeFile should return void tokenizeFile should throw an IOException tokenizeFile should take the contents of the file and tokenize it using a space character as a delimiter

If there are multiple spaces between tokens then ignore those spaces and only retrieve the words, so given the string: “The cat” should still only produce two tokens: “The”, and “cat”

A newline should also serve to separate tokens, for example:

If the input contains multiple lines such as:

The cat in the hat Red fish blue fish There is no space between “hat” and “Red”, just a newline, which means the resulting tokens are “hat” and “Red”, NOT a single token: “hatRed”

Each token should be added to the private ArrayList variable as a Token object If the value of a token is the keyword: public , then increment the private int counter that is keeping track of the number of keywords encountered when parsing the content of a file.

- Remember that “public” is the only keyword you have to worry about - Any letter in the word: public , can be capitalized and you must still increment the counter -

If public is part of another token, then it should NOT count as a keyword -

For example if a file contained: - Blahblahpublic -

The above public does not count as a keyword since public in this case is not its own token Every call to tokenizeFile should first clear the previous tokens from the ArrayList of Tokens, as well as reset the private int counter of keywords (public) For example: if a user calls tokenizeFile(“someFile.txt”) and it generates these tokens:

“The” , “cat”, “in” , and then the user calls tokenizeFile again but with a different input file, such as: tokenizeFile(“somenewfile.txt”), the ArrayList should no longer hold the previous tokens, and should instead hold only the new tokens generated by the new file.

Tokenizer should have only a getter for the ArrayList of Token objects (not a setter) The getter should be called: getTokens and should return an Array of Token objects, NOT an ArrayList of Token objects Tokenizer should have a method called: writeTokens writeTokens should have no input parameter writeTokens should have a return type of void writeTokens should throw an IOException writeTokens should write the tokens out to a file called: “output.txt”, and should reside in your current working directory. Each token should be written on a newline, for example - Given tokens: “The”, “cat”, “in”, “the”,”hat” The result in output.txt should be: The cat in the hat Notice the order the tokens should be written to file are in order from which they are read from the input file, which should be from left to right and from top to bottom Tokenizer should have a public method called: getKeywordCount , that has no input arguments and returns the private int keyword counter field Override the toString method inherited from the Object class, have it return the same value as calling the toString method on the ArrayList of Token objects

Answer 1

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.StringTokenizer;

//Tokenizer class
public class Tokenizer {
   private ArrayList<String> tokens;
   private int counter;
   /**
   * Constructor
   */
   public Tokenizer() {
       tokens=new ArrayList<String>();
       counter=0;
   }

   //tokenizeFile() method
   public void tokenizeFile(String filepath) throws IOException {
       try {
           File file = new File(filepath);
           FileReader reader = new FileReader(filepath);
           BufferedReader br = new BufferedReader(reader);
           String line;
           while ((line = br.readLine())!= null) {
               StringTokenizer stringTokenizer = new StringTokenizer(line," ");
               while (stringTokenizer.hasMoreTokens()) {
                   counter++;
                   tokens.add(stringTokenizer.nextToken());
               }
           }
           br.close();
           reader.close();
       }catch(IOException ioe) {
           ioe.printStackTrace();
       }
   }

   //getKeywordCount() method
   public int getKeywordCount() {
       return counter;
   }

   //writeTokens() method
   public void writeTokens() {
       try {
           FileWriter writer = new FileWriter("output.txt", true);
           for(String token : tokens) {
               writer.write(token);
               writer.write("\n");
           }
           writer.close();
       }catch(IOException ioe) {
           ioe.printStackTrace();
       }
   }
}

Screenshot: