Tutorial 05 - Keeping local landing reports in sync with the Elandings Database

For the completed project for this section see ElandingsClient5.zip
A web service client could be used by third-party developers to keep their local database in sync with the eLandings database. The client could be added to a scheduled task. Each time it wakes up and runs, it could be configured to pull any landing reports from eLandings that have been modified since the last time the client ran. In order to do this, we need to have the Client keep track of when it last ran.
To begin with, we will add a utililty class called CalendarUtility
Right click on the elandingsclient package > New >Java Class

Enter the class name of LastRunTimestamp

Click Finish
We now have an empty class called LastRunTimestamp.

We will use this class to help us keep track of when we last ran time we ran the client. To do this, we will need some tools to work with Date objects and the ability to store and retrieve a Date object. There are lots of different ways that this might be accomplished. In this simple example, we will store the date in a local file and reference it on subsequent runs.

Add the following code to the file:

private static String LAST_RUN_TIMESTAMP_FILE_NAME = "lastRunDate.dat";
private static String TIMESTAMP_DATE_FORMAT = "yyyy-MM-dd-HH.mm.ss.mmm";
private SimpleDateFormat dateFormatter;
Date lastRunDate;
Date currentDate;

We will store our last run in a file called lastRunDate.dat

Next we will create a constructor for the class:

public LastRunTimestamp() {
        currentDate = Calendar.getInstance().getTime();
        dateFormatter = new SimpleDateFormat(TIMESTAMP_DATE_FORMAT);
}

We need a method to write the currentDate to a file. Add the following code:

public void writeCurrentDateToFile() throws FileNotFoundException, IOException{
        String timestamp = dateFormatter.format(currentDate);
        BufferedWriter out = new BufferedWriter(new FileWriter(LAST_RUN_TIMESTAMP_FILE_NAME));
        out.write(timestamp);
        out.close();
    }

Notice that the writeCurrentDateToFile() uses the dateFormatter object to convert the currentDate object into a human readable string. This string is then written to the file lastRunDate.dat

Test the method. Go to the main method of ELandingsClient.java. Edit the main method to just run:

LastRunTimestamp lastRun = new LastRunTimestamp();
lastRun.writeCurrentDateToFile();

Run the application.
Look in the netbeans project for the lastRunDate.dat file

Open the lastRunDate.dat file with a text editor. You should see a single line like "2011-08-17-10.09.34.009"

We now have a way to record when the client ran last. Now we need a way to retrieve this date from the file.
Go back to the LastRunTimestamp.java file
Add the following code to the end of the class:

public void readDateFromFile() throws IOException, ParseException{
        BufferedReader in = null;
        String lastRunDateText = "";
        try {
            in = new BufferedReader(new FileReader(LAST_RUN_TIMESTAMP_FILE_NAME));
            lastRunDateText = in.readLine();
            if (null != lastRunDateText) {
                lastRunDate = dateFormatter.parse(lastRunDateText);
            }

        } catch (FileNotFoundException ex) {
           //DO NOTHING
        } finally {
            if (null != in) {
                try {
                    in.close();
                } catch (IOException ex) {
                    //do nothing
                }
            }
        }
    }

The readDateFromFile() will open up the file lastRunDate.dat and read in the date string stored in the file. It will then convert the string into a Date object and store this value in the Date lastRunDate. Notice that if the method cannot find the file, it captures the error and does nothing. The file will not exist if the application is running for the first time. This handles this situation for us.

In order to see the results of the readDateFromFile(), we will add two print methods:

public void printCurrentDateString() {
        if (null == currentDate) {
            System.out.println("The Current Date is not defined");
        } else {
            String timestamp = dateFormatter.format(currentDate);
            System.out.println("The current date is " + timestamp);
        }
    }

    public void printLastRunDateString() {
        if (null == lastRunDate) {
            System.out.println("The Last Run Date is not defined");
        } else {
            String timestamp = dateFormatter.format(lastRunDate);
            System.out.println("The Last Run Date was " + timestamp);
        }
    }

Now test the LastRunTimestamp methods by modifying ElandingsClient.java main method:

LastRunTimestamp lastRun = new LastRunTimestamp();
            lastRun.readDateFromFile();
            lastRun.printCurrentDateString();
            lastRun.printLastRunDateString();
            lastRun.writeCurrentDateToFile();

Run the application to test the behavior.
If the lastRunDate.dat file does not exist on the first run, you should see something like:

The current date is 2011-08-17-11.03.26.003
The Last Run Date is not defined

Run the application to test the behavior.
If the lastRunDate.dat file does not exist on the first run, you should see something like:

The current date is 2011-08-17-11.04.13.004
The Last Run Date was 2011-08-17-11.03.26.003

We now have a means of recording and loading the last time the elandingsClient application was executed.

We want a means of using this date help us get all landing reports that have changed in the eLandings database, since the last time we ran the eLandingsClient application.

In ElandingsClient.java, find the getLandingReportList()

In the last tutorial, we had to pass the web service method findUserLandingReports001() an XMLGregorianCalendar object called fromLastUpdateDateArg. The fromLastUpdateDateArg tells the eLandings database to get all landing reports that the current user can see that have changed from the fromLastUpdateDateArg to the present.

We want to pass the LastRunDate into this argument so that we can get all the landing reports that have been updated in eLandings since the last time we ran eLandingsClient.

Open the LastRunTimestamp.java file and add the following code to create an XMLGregorianCalendar object containing the lastRunDate

At the top of the LastRunTimestamp.java class, add a DatetypeFactory object and instanciate it in the constructor:

private DatatypeFactory datatypeFactory;

    public LastRunTimestamp() throws DatatypeConfigurationException {
        currentDate = Calendar.getInstance().getTime();
        dateFormatter = new SimpleDateFormat(TIMESTAMP_DATE_FORMAT);
        datatypeFactory = DatatypeFactory.newInstance();
    }

At the end of the LastRunTimestamp.java, create a new method called getLastRunXmlGregorianCalendar():

public XMLGregorianCalendar getLastRunXmlGregorianCalendar(){
if(null == lastRunDate){
            return null;
        }
Calendar calendar = Calendar.getInstance();
calendar.setTime(lastRunDate);

XMLGregorianCalendar calArg = (null == calendar) ? null : datatypeFactory.newXMLGregorianCalendar((GregorianCalendar) calendar);
return calArg;
}

Go back to the ElandingsClient.java file and find the getLandingReportList(). Modify the method to be:

public void getLandingReportList(LastRunTimestamp lastRun) throws JAXBException {
       XMLGregorianCalendar fromLastUpdateDateArg = lastRun.getLastRunXmlGregorianCalendar();
       if (null == fromLastUpdateDateArg) {
            System.out.println("The fromDate is null.");
            System.out.println("Calling the webservice without a fromDate, would result in attempting to download all eLandings landing reports.");
            System.out.println("This may result in an out of memory error and is not recommended.");
            System.out.println("Please enter a more manageable date range and try again.");

            System.out.println("The getLandingReportList() request has been aborted");
            return;
        }
        String xml = svc.findUserLandingReports001("amarx", "A_marx", "2.1", null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null,
        null, null, null, null, fromLastUpdateDateArg, null);
        System.out.println(xml);

        LandingReportInfoWrapper infoWrapper = new LandingReportInfoWrapper(xml, unmarshaller);
        ArrayList<LandingReportWrapper> landingReports = new ArrayList<LandingReportWrapper>();
        int size = infoWrapper.getInfo().getLandingReportSummary().size();
        for (int x = 0; x < size; x++) {
            String landingReportId = infoWrapper.getInfo().getLandingReportSummary().get(x).getLandingReportId().getValue().toString();
            System.out.println((x + 1) + " " + landingReportId);

            String landingReportXml = svc.getLandingReport("amarx", "A_marx", "2.1", landingReportId);
            System.out.println(landingReportXml);
            LandingReportWrapper landingReportWrapper = new LandingReportWrapper(landingReportXml, unmarshaller);
            landingReports.add(landingReportWrapper);
        }

        System.out.println("The landingReports arrayList contains " + landingReports.size() + " reports ");
}

Notice that we are now passing in the LastRunTimestamp object into the getLandingReportList() method. Within the getLandingReportList() method, we are calling the LastRunTimestamp. getLastRunXmlGregorianCalendar() method to get an XMLGregorianCalendar instance of our lastRunDate. This xml calendar object is then being passed into the webservice call.

Notice that we have some basic logic to cancel the webservice call if the XMLGregorianCalendar object is null. Imagine that you have been working with eLandings since it's inseption in 2005. If you were to call the findUserLandingReports() without a date range of any kind, eLandings would attempt to send you all landing reports available to you. This could result in the download of tens of thousands of records depending on the user making the request. This method might run for hours before it completes the requests or more likely eLandings may terminate the request or your local java heap would run out of memory and crash. The eLandings team recommends working with smaller time ranges to minimize impact on the eLandnigs system and your own system.

Modify the ElandingsClient.java main() method to test these changes

public static void main(String\[\] args) {
       System.out.println("Hello ELandings Client");
       try {
            ElandingsClient client = new ElandingsClient();
            //client.connectToWebService();
            LastRunTimestamp lastRun = new LastRunTimestamp();
            lastRun.readDateFromFile();
            lastRun.printCurrentDateString();
            lastRun.printLastRunDateString();
            client.getLandingReportList(lastRun);
            lastRun.writeCurrentDateToFile();
        } catch (Exception e) {
            e.printStackTrace();
        }
}

Run the ElandingsClient application
If you don't have a lastRunDate.dat file, you may get the following results.

Hello ELandings Client
The current date is 2011-08-17-11.48.52.048
The Last Run Date is not defined
The fromDate is null.
Calling the webservice without a fromDate, would result in attempting to download all eLandngs landing reports.
This may result in an out of memory error and is not recommended.
Please enter a more manageable date range and try again.
getLandingReportList request has been aborted

If you have a lastRunDate.dat of today, and no landing reports have been changed in the database after that time you may see an xml result containing an error message 'No Landing Reports found that match search criteria'
If you have a lastRunDate.dat of today, and there were landing reports modified since the last run, you might see a list of landing_report objects displayed in the output.

To play around with this, try the following.
Find the lastRunDate.dat file
Look in the netbeans project for the lastRunDate.dat file in your project folder.

Open the lastRunDate.dat file with a text editor.

Change the date and or time in the file. Save your changes and run the ElandingsClient application again. Repeat this until you get back a few landing reports or understand the process.

Alternatively, log into the eLandings test system and enter a new landing report and save it. Then run the ELandingsClient application again. If your lastRunDate.dat file contains a timestamp before you created the new landing report, ElandingsClient should retrieve the landing report for you.

We will demonstrate creating an landing report in eLandings now. We first run the ElandingsClient and get no results.

Log into eLandings test website. We are using test because our ElandingsClient is pointed to the test webservices. If our ElandingsClient was pointed at eLandings training or production webservices, then we would use the associated web page.

http://elandingstest.alaska.gov/elandings/Login
Login with the same credentials that you are using in your ElandingsClient webservice calls. In our case we will log in as amarx/A_marx.

We could create any kind of landing report. We will make a salmon landing report because they are the simplest to create. Click on the Salmon Landing Report button.

Enter some data and click the Save button when done.

We now have a unsubmitted landing report. We still need to add some line item information

Click on the Add/Edit Itemized Catch button

Enter some data and click the save button

We now have a basic landing report. Click the Submit Initial Report button

The landing report has now been initially submitted but not finally submitted.

Back in Netbeans, if we run the ElandingsClient now, we will get this report returned to us:

Our landing report ID = 15498992, just like in the eLandings web page. The report was last modified at 2011-08-17T12:35:07.000. You can see it was modified after the Last Run Date of 2011-08-17-12:20:55.020

Go back to the eLandings web page report and click the submit final report:

Back in Netbeans, run the ElandingsClient application again. We get the landing report again because the landing report has been modified in the eLandings database since the last time we ran ElandingsClient. The landing report was changed from a submission status of 'Initial Submit' to 'Final Submit'. ElandingsClient was able to retrieve the most current version of the landing report from the eLandings database. In this way we could keep a remote copy of the data up-to-date with the eLandings database.

We now can get the data but we have no means of storing the data yet. There are lots of ways to store the data. As a partial list of potential options, we could store the data using:

• Java serialized data objects. For example, the LandingReportWrapper object could be serialized and saved to a data file.
• XML files. For example, each landing report to an xml file.
• EXCEL file. For example, some elements from each landing report could be extracted and stored to an EXCEL spreadsheet.
• Database. For example, each landing report could be broken into a header record, line item records, and grading and pricing records and inserted into a header table, line item table and grading table.

There is lots of latitude for how the data can be stored. In this excersise, we will store each landing report to its own xml file. To do this we need a mechanism to write xml string data to a file.

Open up the LandingReportWrapper.java file and add the following code:

public static void writeListToFile(ArrayList<LandingReportWrapper> reportsList, File targetFolder, Marshaller marshaller) throws FileNotFoundException, JAXBException, IOException{
        if(null == reportsList || 0==reportsList.size() || null == marshaller){
            return;
        }
        LandingReportWrapper report = null;
        int size = reportsList.size();
        for(int x = 0; x<size; x++){
            report = reportsList.get(x);
            writeReportToFile(report, targetFolder, marshaller);
        }
    }

    public static void writeReportToFile(LandingReportWrapper rpt, File targetFolder, Marshaller marshaller) throws FileNotFoundException, JAXBException, IOException{
        if(!targetFolder.isDirectory() || !targetFolder.exists()){
            targetFolder.mkdir();
        }
        File fileOut = new File(targetFolder, rpt.getRpt().getLandingReportId().getValue().toString()+".xml");
        String xml = rpt.toXmlString(marshaller);
        BufferedWriter outWriter = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(fileOut)));
        try {
            outWriter.write(xml);
        }
        finally {
            outWriter.close();
        }
    }

We have added two methods
The first method writeListToFile() takes a list of LandingReportWrappers and iterates over the list. For each LandingReportWrapper in the list we call the second method writeReportToFile().

The method writeReportToFile() checks to see if the target folder exists. It then writes the landing report xml data to the xml file in the target folder. The method names the xml file to <LandingReportID>.xml

Save your changes

Open the ElandingsClient.java file
Find the method getLandingReportList().
Add the following line of code after the for loop:

LandingReportWrapper.writeListToFile(landingReports, new File("LandingReports"), marshaller);

The method getLandingReportList() should now:

1. Get a list of landing report summaries since the last time we have run the ElandingsClient application
2. Get each of the landing reports listed in the landing report summary
3. Store each of the landing reports to an XML file in the LandingReports folder.
   

Modify the lastRunDate.dat file so that you will get a few landing report records.

To do this, look in the netbeans project for the lastRunDate.dat file

Open the lastRunDate.dat file with a text editor.

Change the date and or time in the file to a little while before you modified your last eLandings record.
Save your changes.
Run the ElandingsClient applications

You should see something like:

In this case, only one report was downloaded from eLandings.
Look in the Netbeans project and find the LandingReport folder that was created when you ran the ElandingsClient program.

Open the folder

Open the landing report xml file in a text editor

Notice that this is the same data that was returned by the webservice call to svc.getLandingReport("amarx", "A_marx", "2.1", landingReportId);

We are now able to get landing reports that have been modified since the last time we ran the ElandingsClient application. We are also able to store the data locally. Now we need to be able to run the application as a scheduled task. There are several ways to do this. We could:

1. Add a timer to the ElandingsClient that executes the getLandingReportList() every so often.
2. Run ElandingsClient as a Linux CronJob or Windows Scheduled Task.

Either way can be made to work. For this example, we will create a Windows Scheduled Task.

In order to do this, we have to be able to run ElandingsClient outside of Netbeans from the command line.

Before we do this, let's look at the project properties. Right click on the project > Properties

Click on the Build > Packaging on the left hand side of the Project Properties dialog.

By default, Netbeans will create a Jar File to the folder dist when we build the project. This is what we want. Click ok.

Click on Run > Build Main Project to compile the project and create the jar file after compiling.
You should see the console say that the project jar was created.

Open the ELandingsClient folder outside Netbeans.
Open the dist sub-folder

Netbeans created a ElandingsClient.jar file for us. We want to run this jar file.
Create a new file in dist called Run.bat – We will use this file to run the jar

Open the run.bat file in a text editor and add the following lines:

cd "%~dp0"
java -jar ElandingsClient.jar elandingsclient.ElandingsClient
pause

Save the run.bat file
Note: the dist folder gets deleted and recreated by Netbeans every time you run the clean and build command. When this happens, the run.bat file will be deleted in the dist folder as well. It is good practice to place a copy of the run.bat in the Netbeans project root directory.

Before we forget, copy and paste the run.bat file from the dist folder to the root directory of ElandingsClient

Double-click on the run.bat file to execute the application.
The first execution fails because we do not have a lastRunDate.dat file

Close the windows console by clicking any key

Look back in the dist folder. Notice that we now have a lastRunDate.dat file

Double click on the run.bat file again to run it once more.
We now have a lastRunDate.dat file but there have been no eLandings changes since we ran the application less than a minute ago. We get a message saying there were no landing records found.

In the dist folder, edit the lastRunDate.dat file so that the ElandingsClient will get some landing reports.
For example, I had:

2011-08-17-14.22.07.022

I changed it to:

2011-08-17-09.22.07.022

Save your changes.
Now run the application again by double-clicking on run.bat
This time ElandingsClient gets back 18 landing reports from the web services.

Look back in the dist folder

Notice that we now have a LandingReports folder

We can see our 18 landing reports stored as XML files:

We now can run the application from the command line. To speed up the execution, we will comment out the System.out.println statements that print out the XML strings in the code and rebuild the project.
Click Run > Build Main Project

You should see the console reporting that the ElandingsClient.jar has been recreated.

Note: We could have clicked Clean and Build Main Project to generate all the JAXB files from the XSD definition files and regenerate the webservice method objects. If we did click on the Clean and Build Main Project be sure to copy and past the run.bat file back into the dist folder. The lastRunDate.dat file would also be deleted.

Look at the dist folder now

A scheduled task cannot have a pause statement in the executable file. Before we create a scheduledTask, we must remove the pause from the run.bat file.
Open up the run.bat file and comment out the pause statement with the line comment rem.

cd "%~dp0"
java -jar ElandingsClient.jar elandingsclient.ElandingsClient
rem pause

Save your work.

Click on the Start > Control Panel

Click on Scheduled Tasks

Click on Add Scheduled Task

Right click in the name category and create new > Scheduled task

Double click on the new task

Click browse and select the rub.bat in the dist folder

For me, I will run only if logged on but on a server, you would have it run 24x7.
Click on the Schedule tab

Click on Advanced button
Check the Repeat task button and set the duration

Click OK
Click Apply
Right click on the new Task and rename to runElandingsClient
You may need to run the task once manually before it will run automatically.

We now have the ElandingsClient set up to run as a scheduled task on my local machine. Every 10 minutes, the application will run and ask for any landing reports that have been created or modified in the last 10 minutes. Any records returned from the webservices will be stored locally in the LandingReport folder in xml files.

This demonstrates the basic process of keeping a local copy of landing reports in sync with the eLandings database. The same principles could be used to keep a database or EXCEL spreadsheet up-to-date.