com.gisgraphy.importer

## Class AbstractSimpleImporterProcessor

- java.lang.Object
- - com.gisgraphy.importer.AbstractSimpleImporterProcessor

- All Implemented Interfaces:
    IImporterProcessor
  Direct Known Subclasses:
    AdmExtracter, GeonamesAdmExtracter, GeonamesAlternateNamesExtracter, GeonamesAlternateNamesSimpleImporter, GeonamesCountryImporter, GeonamesFeatureSimpleImporter, GeonamesLanguageImporter, GeonamesZipCodeSimpleImporter, OpenStreetMapCitiesSimpleImporter, OpenStreetMapHouseNumberSimpleImporter, OpenStreetMapPoisSimpleImporter, OpenStreetMapSimpleImporter, QuattroshapesSimpleImporter

      public abstract class AbstractSimpleImporterProcessor
      extends Object
      implements IImporterProcessor

  Base class for all geonames processor. it provides session management and the ability to process one or more CSV file
  Author:
    David Masclet

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected String` | `COMMENT_START` Lines starting with this prefix are considered as comments
    `protected File` | `currentFile` The current processed file
    `protected ImporterConfig` | `importerConfig`
    `protected BufferedReader` | `in` The bufferReader for the current read Geonames file
    `protected IInternationalisationService` | `internationalisationService`
    `protected static org.slf4j.Logger` | `logger` The logger
    `protected int` | `readFileLine`
    `protected ImporterStatus` | `status`
    `protected String` | `statusMessage`
    `protected int` | `totalReadLine`
    `protected org.springframework.transaction.PlatformTransactionManager` | `transactionManager` The transaction manager
    `protected org.springframework.transaction.support.DefaultTransactionDefinition` | `txDefinition`
    `protected org.springframework.transaction.TransactionStatus` | `txStatus`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `AbstractSimpleImporterProcessor()` Default constructor

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `protected void` | `checkNumberOfColumn(String[] fields)` Utility method which throw an exception if the number of fields is not the one expected (retrieved by `getNumberOfColumns()`)
    `protected void` | `commit()`
    `protected int` | `countLines(File[] files)`
    `protected int` | `decrementReadedFileLine(int decrement)`
    `protected static String` | `dumpFields(String[] fields)`
    `protected abstract void` | `flushAndClear()` Should flush and clear all the Daos that are used by the processor.
    `String` | `getCurrentFileName()`
    `protected abstract File[]` | `getFiles()`
    `protected int` | `getMaxInsertsBeforeFlush()`
    `protected abstract int` | `getNumberOfColumns()`
    `long` | `getNumberOfLinesToProcess()`
    `long` | `getReadFileLine()` The number of read line for the current processed file
    `ImporterStatus` | `getStatus()`
    `String` | `getStatusMessage()`
    `long` | `getTotalReadLine()` The number of read line for all the processed file
    `protected int` | `incrementReadedFileLine(int increment)`
    `protected static boolean` | `isEmptyField(String[] fields, int position, boolean required)` Check that the array is not null, and the fields of the specified position is not empty (after been trimed)
    `protected boolean` | `isEndOfDocument()`
    `protected boolean` | `needCommit()`
    `protected void` | `onFileProcessed(File file)` hook to do when the file has been processed without error
    `void` | `process()` Manage the transaction, flush Daos, and process all files to be processed
    `protected abstract void` | `processData(String line)` Process a read line of the geonames file, must be implemented by the concrete class
    `protected void` | `processError(Exception e)` Method called when there is an exception.
    `int` | `readLineAndProcessData()` Process the line if needed (is not a comment, should ignore first line, is end of document,...)
    `void` | `resetStatus()` Reset status fields, it should be done when the import has been canceled
    `protected void` | `rollbackTransaction()`
    `protected abstract void` | `setCommitFlushMode()` Will flush after every commit
    `void` | `setImporterConfig(ImporterConfig importerConfig)`
    `void` | `setInternationalisationService(IInternationalisationService internationalisationService)`
    `void` | `setTransactionManager(org.springframework.transaction.PlatformTransactionManager transactionManager)`
    `protected void` | `setup()` Template method that can be override.
    `boolean` | `shouldBeSkipped()`
    `protected abstract boolean` | `shouldIgnoreComments()` Whether the filter should ignore the comments (i.e. lines starting with #)
    `protected abstract boolean` | `shouldIgnoreFirstLine()` Template Method : Whether the processor should ignore the first line of the input
    `protected void` | `startTransaction()`
    `protected void` | `tearDown()` Template method that can be override.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface com.gisgraphy.importer.IImporterProcessor

      `rollback`

- - ### Field Detail

    - #### totalReadLine

          protected int totalReadLine

    - #### readFileLine

          protected int readFileLine

    - #### statusMessage

          protected String statusMessage

    - #### status

          protected ImporterStatus status

    - #### internationalisationService

          @Autowired
          protected IInternationalisationService internationalisationService

    - #### importerConfig

          protected ImporterConfig importerConfig

    - #### logger

          protected static final org.slf4j.Logger logger

      The logger

    - #### COMMENT_START

          protected String COMMENT_START

      Lines starting with this prefix are considered as comments

    - #### in

          protected BufferedReader in

      The bufferReader for the current read Geonames file

    - #### transactionManager

          protected org.springframework.transaction.PlatformTransactionManager transactionManager

      The transaction manager

    - #### txStatus

          protected org.springframework.transaction.TransactionStatus txStatus

    - #### txDefinition

          protected org.springframework.transaction.support.DefaultTransactionDefinition txDefinition

    - #### currentFile

          protected File currentFile

      The current processed file

  - ### Constructor Detail

    - #### AbstractSimpleImporterProcessor

          public AbstractSimpleImporterProcessor()

      Default constructor

  - ### Method Detail

    - #### shouldIgnoreFirstLine

          protected abstract boolean shouldIgnoreFirstLine()

      Template Method : Whether the processor should ignore the first line of the input
      Returns:
        true if the processor should ignore first line

    - #### flushAndClear

          protected abstract void flushAndClear()

      Should flush and clear all the Daos that are used by the processor. This avoid memory leak

    - #### setCommitFlushMode

          protected abstract void setCommitFlushMode()

      Will flush after every commit
      See Also:
        `flushAndClear()`

    - #### getNumberOfColumns

          protected abstract int getNumberOfColumns()

      Returns:
        the number of fields the processed Geonames file should have

    - #### shouldIgnoreComments

          protected abstract boolean shouldIgnoreComments()

      Whether the filter should ignore the comments (i.e. lines starting with #)
      See Also:
        `COMMENT_START`

    - #### setup

          protected void setup()

      Template method that can be override. This method is called before the process start. it is not called for each file processed.

    - #### getFiles

          protected abstract File[] getFiles()

      Returns:
        The files to be process
      See Also:
        `ImporterHelper`

    - #### getCurrentFileName

          public String getCurrentFileName()

      Specified by:
        `getCurrentFileName` in interface `IImporterProcessor`
      Returns:
        The name of the file currently processed or null if no file is processed

    - #### readLineAndProcessData

          public int readLineAndProcessData()
                                     throws ImporterException

      Process the line if needed (is not a comment, should ignore first line, is end of document,...)
      Returns:
        The number of lines that have been processed for the current processed file
      Throws:
        `ImporterException` - if an error occurred

    - #### processData

          protected abstract void processData(String line)
                                       throws ImporterException

      Process a read line of the geonames file, must be implemented by the concrete class
      Parameters:
        `line` - the line to process
      Throws:
        `ImporterException`

    - #### process

          public void process()

      Manage the transaction, flush Daos, and process all files to be processed
      Specified by:
        `process` in interface `IImporterProcessor`

    - #### processError

          protected void processError(Exception e)

      Method called when there is an exception. the teardown method will be call after this
      Parameters:
        `e` -

    - #### shouldBeSkipped

          public boolean shouldBeSkipped()

      Specified by:
        `shouldBeSkipped` in interface `IImporterProcessor`
      Returns:
        true if the processor should Not be executed

    - #### incrementReadedFileLine

          protected int incrementReadedFileLine(int increment)

    - #### decrementReadedFileLine

          protected int decrementReadedFileLine(int decrement)

    - #### rollbackTransaction

          protected void rollbackTransaction()

    - #### needCommit

          protected boolean needCommit()

    - #### startTransaction

          protected void startTransaction()

    - #### tearDown

          protected void tearDown()

      Template method that can be override. This method is called after the end of the process. it is not called for each file processed. You should always call super.tearDown() when you overide this method

    - #### onFileProcessed

          protected void onFileProcessed(File file)

      hook to do when the file has been processed without error
      Parameters:
        `file` -

    - #### commit

          protected void commit()

    - #### isEmptyField

          protected static boolean isEmptyField(String[] fields,
                             int position,
                             boolean required)

      Check that the array is not null, and the fields of the specified position is not empty (after been trimed)
      Parameters:
        `fields` - The array to test
        `position` - the position of the field to test in the array
        `required` - if an exception should be thrown if the field is empty
      Returns:
        true is the field of the specifed position is empty
      Throws:
        `MissingRequiredFieldException` - if the fields is empty and required is true

    - #### dumpFields

          protected static String dumpFields(String[] fields)

      Parameters:
        `fields` - The array to process
      Returns:
        a string which represent a human readable string of the Array

    - #### checkNumberOfColumn

          protected void checkNumberOfColumn(String[] fields)

      Utility method which throw an exception if the number of fields is not the one expected (retrieved by `getNumberOfColumns()`)
      Parameters:
        `fields` - The array to check
      See Also:
        `getNumberOfColumns()`

    - #### isEndOfDocument

          protected boolean isEndOfDocument()

      Returns:
        true if the end of the document for the current processed file is reached

    - #### getReadFileLine

          public long getReadFileLine()

      Description copied from interface: `IImporterProcessor`
      The number of read line for the current processed file
      Specified by:
        `getReadFileLine` in interface `IImporterProcessor`
      See Also:
        `IImporterProcessor.getTotalReadLine()`

    - #### getTotalReadLine

          public long getTotalReadLine()

      Description copied from interface: `IImporterProcessor`
      The number of read line for all the processed file
      Specified by:
        `getTotalReadLine` in interface `IImporterProcessor`
      See Also:
        `IImporterProcessor.getReadFileLine()`

    - #### setTransactionManager

          public void setTransactionManager(org.springframework.transaction.PlatformTransactionManager transactionManager)

    - #### setImporterConfig

          public void setImporterConfig(ImporterConfig importerConfig)

    - #### countLines

          protected int countLines(File[] files)

      Returns:
        the number of line to process

    - #### getNumberOfLinesToProcess

          public long getNumberOfLinesToProcess()

      Specified by:
        `getNumberOfLinesToProcess` in interface `IImporterProcessor`
      Returns:
        The number of line the processor will process. (it is not the number of lines left!)

    - #### getStatus

          public ImporterStatus getStatus()

      Specified by:
        `getStatus` in interface `IImporterProcessor`
      Returns:
        The current status of the importer

    - #### getMaxInsertsBeforeFlush

          protected int getMaxInsertsBeforeFlush()

      Returns:
        The option
      See Also:
        `ImporterConfig.setMaxInsertsBeforeFlush(int)`

    - #### resetStatus

          public void resetStatus()

      Description copied from interface: `IImporterProcessor`
      Reset status fields, it should be done when the import has been canceled
      Specified by:
        `resetStatus` in interface `IImporterProcessor`

    - #### getStatusMessage

          public String getStatusMessage()

      Specified by:
        `getStatusMessage` in interface `IImporterProcessor`
      Returns:
        A text Message for the importer

    - #### setInternationalisationService

          public void setInternationalisationService(IInternationalisationService internationalisationService)

Copyright © 2015. All rights reserved.