com.gisgraphy.importer

Class AbstractSimpleImporterProcessor

java.lang.Object

com.gisgraphy.importer.AbstractSimpleImporterProcessor

All Implemented Interfaces:

IImporterProcessor

Direct Known Subclasses:

AdmExtracter, GeonamesAdmExtracter, GeonamesAlternateNamesExtracter, GeonamesAlternateNamesSimpleImporter, GeonamesCountryImporter, GeonamesFeatureSimpleImporter, GeonamesLanguageImporter, GeonamesZipCodeSimpleImporter, OpenStreetMapCitiesSimpleImporter, OpenStreetMapHouseNumberSimpleImporter, OpenStreetMapPoisSimpleImporter, OpenStreetMapSimpleImporter, QuattroshapesSimpleImporter

public abstract class AbstractSimpleImporterProcessor
extends Object
implements IImporterProcessor

Base class for all geonames processor. it provides session management and the ability to process one or more CSV file

Author:

David Masclet

Field Summary

Fields

Modifier and Type	Field and Description
protected String	COMMENT_START Lines starting with this prefix are considered as comments
protected File	currentFile The current processed file
protected ImporterConfig	importerConfig
protected BufferedReader	in The bufferReader for the current read Geonames file
protected IInternationalisationService	internationalisationService
protected static org.slf4j.Logger	logger The logger
protected int	readFileLine
protected ImporterStatus	status
protected String	statusMessage
protected int	totalReadLine
protected org.springframework.transaction.PlatformTransactionManager	transactionManager The transaction manager
protected org.springframework.transaction.support.DefaultTransactionDefinition	txDefinition
protected org.springframework.transaction.TransactionStatus	txStatus

Constructor Summary

Constructors

Constructor and Description
AbstractSimpleImporterProcessor() Default constructor

Method Summary

Methods

Modifier and Type	Method and Description
protected void	checkNumberOfColumn(String[] fields) Utility method which throw an exception if the number of fields is not the one expected (retrieved by getNumberOfColumns())
protected void	commit()
protected int	countLines(File[] files)
protected int	decrementReadedFileLine(int decrement)
protected static String	dumpFields(String[] fields)
protected abstract void	flushAndClear() Should flush and clear all the Daos that are used by the processor.
String	getCurrentFileName()
protected abstract File[]	getFiles()
protected int	getMaxInsertsBeforeFlush()
protected abstract int	getNumberOfColumns()
long	getNumberOfLinesToProcess()
long	getReadFileLine() The number of read line for the current processed file
ImporterStatus	getStatus()
String	getStatusMessage()
long	getTotalReadLine() The number of read line for all the processed file
protected int	incrementReadedFileLine(int increment)
protected static boolean	isEmptyField(String[] fields, int position, boolean required) Check that the array is not null, and the fields of the specified position is not empty (after been trimed)
protected boolean	isEndOfDocument()
protected boolean	needCommit()
protected void	onFileProcessed(File file) hook to do when the file has been processed without error
void	process() Manage the transaction, flush Daos, and process all files to be processed
protected abstract void	processData(String line) Process a read line of the geonames file, must be implemented by the concrete class
protected void	processError(Exception e) Method called when there is an exception.
int	readLineAndProcessData() Process the line if needed (is not a comment, should ignore first line, is end of document,...)
void	resetStatus() Reset status fields, it should be done when the import has been canceled
protected void	rollbackTransaction()
protected abstract void	setCommitFlushMode() Will flush after every commit
void	setImporterConfig(ImporterConfig importerConfig)
void	setInternationalisationService(IInternationalisationService internationalisationService)
void	setTransactionManager(org.springframework.transaction.PlatformTransactionManager transactionManager)
protected void	setup() Template method that can be override.
boolean	shouldBeSkipped()
protected abstract boolean	shouldIgnoreComments() Whether the filter should ignore the comments (i.e. lines starting with #)
protected abstract boolean	shouldIgnoreFirstLine() Template Method : Whether the processor should ignore the first line of the input
protected void	startTransaction()
protected void	tearDown() Template method that can be override.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.gisgraphy.importer.IImporterProcessor

rollback

Field Detail

totalReadLine

protected int totalReadLine

readFileLine

protected int readFileLine

statusMessage

protected String statusMessage

status

protected ImporterStatus status

internationalisationService

@Autowired
protected IInternationalisationService internationalisationService

importerConfig

protected ImporterConfig importerConfig

logger

protected static final org.slf4j.Logger logger

The logger

COMMENT_START

protected String COMMENT_START

Lines starting with this prefix are considered as comments

in

protected BufferedReader in

The bufferReader for the current read Geonames file

transactionManager

protected org.springframework.transaction.PlatformTransactionManager transactionManager

The transaction manager

txStatus

protected org.springframework.transaction.TransactionStatus txStatus

txDefinition

protected org.springframework.transaction.support.DefaultTransactionDefinition txDefinition

currentFile

protected File currentFile

The current processed file

Constructor Detail

AbstractSimpleImporterProcessor

public AbstractSimpleImporterProcessor()

Default constructor

Method Detail

shouldIgnoreFirstLine

protected abstract boolean shouldIgnoreFirstLine()

Template Method : Whether the processor should ignore the first line of the input

Returns:

true if the processor should ignore first line

flushAndClear

protected abstract void flushAndClear()

Should flush and clear all the Daos that are used by the processor. This avoid memory leak

setCommitFlushMode

protected abstract void setCommitFlushMode()

Will flush after every commit

See Also:

flushAndClear()

getNumberOfColumns

protected abstract int getNumberOfColumns()

Returns:

the number of fields the processed Geonames file should have

shouldIgnoreComments

protected abstract boolean shouldIgnoreComments()

Whether the filter should ignore the comments (i.e. lines starting with #)

See Also:

COMMENT_START

setup

protected void setup()

Template method that can be override. This method is called before the process start. it is not called for each file processed.

getFiles

protected abstract File[] getFiles()

Returns:

The files to be process

See Also:

ImporterHelper

getCurrentFileName

public String getCurrentFileName()

Specified by:

getCurrentFileName in interface IImporterProcessor

Returns:

The name of the file currently processed or null if no file is processed

readLineAndProcessData

public int readLineAndProcessData()
                           throws ImporterException

Process the line if needed (is not a comment, should ignore first line, is end of document,...)

Returns:

The number of lines that have been processed for the current processed file

Throws:

ImporterException - if an error occurred

processData

protected abstract void processData(String line)
                             throws ImporterException

Process a read line of the geonames file, must be implemented by the concrete class

Parameters:

line - the line to process

Throws:

ImporterException

process

public void process()

Manage the transaction, flush Daos, and process all files to be processed

Specified by:

process in interface IImporterProcessor

processError

protected void processError(Exception e)

Method called when there is an exception. the teardown method will be call after this

Parameters:

e -

shouldBeSkipped

public boolean shouldBeSkipped()

Specified by:

shouldBeSkipped in interface IImporterProcessor

Returns:

true if the processor should Not be executed

incrementReadedFileLine

protected int incrementReadedFileLine(int increment)

decrementReadedFileLine

protected int decrementReadedFileLine(int decrement)

rollbackTransaction

protected void rollbackTransaction()

needCommit

protected boolean needCommit()

startTransaction

protected void startTransaction()

tearDown

protected void tearDown()

Template method that can be override. This method is called after the end of the process. it is not called for each file processed. You should always call super.tearDown() when you overide this method

onFileProcessed

protected void onFileProcessed(File file)

hook to do when the file has been processed without error

Parameters:

file -

commit

protected void commit()

isEmptyField

protected static boolean isEmptyField(String[] fields,
                   int position,
                   boolean required)

Check that the array is not null, and the fields of the specified position is not empty (after been trimed)

Parameters:

fields - The array to test

position - the position of the field to test in the array

required - if an exception should be thrown if the field is empty

Returns:

true is the field of the specifed position is empty

Throws:

MissingRequiredFieldException - if the fields is empty and required is true

dumpFields

protected static String dumpFields(String[] fields)

Parameters:

fields - The array to process

Returns:

a string which represent a human readable string of the Array

checkNumberOfColumn

protected void checkNumberOfColumn(String[] fields)

Utility method which throw an exception if the number of fields is not the one expected (retrieved by getNumberOfColumns())

Parameters:

fields - The array to check

See Also:

getNumberOfColumns()

isEndOfDocument

protected boolean isEndOfDocument()

Returns:

true if the end of the document for the current processed file is reached

getReadFileLine

public long getReadFileLine()

Description copied from interface: IImporterProcessor

The number of read line for the current processed file

Specified by:

getReadFileLine in interface IImporterProcessor

See Also:

IImporterProcessor.getTotalReadLine()

getTotalReadLine

public long getTotalReadLine()

Description copied from interface: IImporterProcessor

The number of read line for all the processed file

Specified by:

getTotalReadLine in interface IImporterProcessor

See Also:

IImporterProcessor.getReadFileLine()

setTransactionManager

public void setTransactionManager(org.springframework.transaction.PlatformTransactionManager transactionManager)

setImporterConfig

public void setImporterConfig(ImporterConfig importerConfig)

countLines

protected int countLines(File[] files)

Returns:

the number of line to process

getNumberOfLinesToProcess

public long getNumberOfLinesToProcess()

Specified by:

getNumberOfLinesToProcess in interface IImporterProcessor

Returns:

The number of line the processor will process. (it is not the number of lines left!)

getStatus

public ImporterStatus getStatus()

Specified by:

getStatus in interface IImporterProcessor

Returns:

The current status of the importer

getMaxInsertsBeforeFlush

protected int getMaxInsertsBeforeFlush()

Returns:

The option

See Also:

ImporterConfig.setMaxInsertsBeforeFlush(int)

resetStatus

public void resetStatus()

Description copied from interface: IImporterProcessor

Reset status fields, it should be done when the import has been canceled

Specified by:

resetStatus in interface IImporterProcessor

getStatusMessage

public String getStatusMessage()

Specified by:

getStatusMessage in interface IImporterProcessor

Returns:

A text Message for the importer

setInternationalisationService

public void setInternationalisationService(IInternationalisationService internationalisationService)