com.gisgraphy.importer
Class AbstractAdvancedImporterProcessor

java.lang.Object
  com.gisgraphy.importer.AbstractSimpleImporterProcessor
      com.gisgraphy.importer.AbstractAdvancedImporterProcessor

All Implemented Interfaces:

IImporterProcessor

public abstract class AbstractAdvancedImporterProcessor
extends AbstractSimpleImporterProcessor

Import the street from an (pre-processed) openStreet map data file in with all the processor capacity.

Author:

David Masclet

Field Summary

protected ImporterException	exception
protected FileSpliter	fileSpliter
protected IIdGenerator	idGenerator
protected static org.slf4j.Logger	logger The logger
protected List<File>	splitedFiles
protected org.springframework.scheduling.backportconcurrent.ThreadPoolTaskExecutor	threadPoolTaskExecutor
protected AtomicInteger	totalReadLineMultithreaded

Fields inherited from class com.gisgraphy.importer.AbstractSimpleImporterProcessor

COMMENT_START, currentFile, importerConfig, in, internationalisationService, readFileLine, status, statusMessage, totalReadLine, transactionManager, txDefinition, txStatus

Constructor Summary

AbstractAdvancedImporterProcessor()

Method Summary

abstract Runnable	createTask(File file)
protected void	flushAndClear() Should flush and clear all the Daos that are used by the processor.
String	getCurrentFileName()
protected int	getNumberOfColumns()
long	getNumberOfLinesToProcess()
int	getNumberOfProcessors()
int	getNumberOfTaskExecutors()
long	getReadFileLine() The number of read line for the current processed file
protected abstract List<File>	getSplitedFiles()
protected int	getSplitLength(int nblines)
long	getTotalReadLine() The number of read line for all the processed file
int	incrementReadedFileLine(int increment)
void	process() Manage the transaction, flush Daos, and process all files to be processed
protected void	processData(String line) Process a read line of the geonames file, must be implemented by the concrete class
void	setIdGenerator(IIdGenerator idGenerator)
protected void	setup() Template method that can be override.
protected boolean	shouldIgnoreComments() Whether the filter should ignore the comments (i.e. lines starting with #)
protected boolean	shouldIgnoreFirstLine() Template Method : Whether the processor should ignore the first line of the input
protected void	splitFiles()

Methods inherited from class com.gisgraphy.importer.AbstractSimpleImporterProcessor

checkNumberOfColumn, commit, countLines, decrementReadedFileLine, dumpFields, getFiles, getMaxInsertsBeforeFlush, getStatus, getStatusMessage, isEmptyField, isEndOfDocument, needCommit, onFileProcessed, processError, readLineAndProcessData, resetStatus, rollbackTransaction, setCommitFlushMode, setImporterConfig, setInternationalisationService, setTransactionManager, shouldBeSkipped, startTransaction, tearDown

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.gisgraphy.importer.IImporterProcessor

rollback

Field Detail

idGenerator

protected IIdGenerator idGenerator

totalReadLineMultithreaded

protected AtomicInteger totalReadLineMultithreaded

threadPoolTaskExecutor

protected org.springframework.scheduling.backportconcurrent.ThreadPoolTaskExecutor threadPoolTaskExecutor

fileSpliter

protected FileSpliter fileSpliter

splitedFiles

protected List<File> splitedFiles

logger

protected static final org.slf4j.Logger logger

The logger

exception

protected ImporterException exception

Constructor Detail

AbstractAdvancedImporterProcessor

public AbstractAdvancedImporterProcessor()

Method Detail

setup

protected void setup()

Description copied from class: AbstractSimpleImporterProcessor

Template method that can be override. This method is called before the process start. it is not called for each file processed.

Overrides:

setup in class AbstractSimpleImporterProcessor

getSplitedFiles

protected abstract List<File> getSplitedFiles()

splitFiles

protected void splitFiles()

getSplitLength

protected int getSplitLength(int nblines)

process

public void process()

Description copied from class: AbstractSimpleImporterProcessor

Manage the transaction, flush Daos, and process all files to be processed

Specified by:

process in interface IImporterProcessor

Overrides:

process in class AbstractSimpleImporterProcessor

createTask

public abstract Runnable createTask(File file)

incrementReadedFileLine

public int incrementReadedFileLine(int increment)

Overrides:

incrementReadedFileLine in class AbstractSimpleImporterProcessor

Parameters:

increment -

Returns:

the incremented value

processData

protected void processData(String line)
                    throws ImporterException

Description copied from class: AbstractSimpleImporterProcessor

Process a read line of the geonames file, must be implemented by the concrete class

Specified by:

processData in class AbstractSimpleImporterProcessor

Parameters:

line - the line to process

Throws:

ImporterException

getCurrentFileName

public String getCurrentFileName()

Specified by:

getCurrentFileName in interface IImporterProcessor

Overrides:

getCurrentFileName in class AbstractSimpleImporterProcessor

Returns:

The name of the file currently processed or null if no file is processed

getReadFileLine

public long getReadFileLine()

Description copied from interface: IImporterProcessor

The number of read line for the current processed file

Specified by:

getReadFileLine in interface IImporterProcessor

Overrides:

getReadFileLine in class AbstractSimpleImporterProcessor

See Also:

IImporterProcessor.getTotalReadLine()

getTotalReadLine

public long getTotalReadLine()

Description copied from interface: IImporterProcessor

The number of read line for all the processed file

Specified by:

getTotalReadLine in interface IImporterProcessor

Overrides:

getTotalReadLine in class AbstractSimpleImporterProcessor

See Also:

IImporterProcessor.getReadFileLine()

shouldIgnoreComments

protected boolean shouldIgnoreComments()

Description copied from class: AbstractSimpleImporterProcessor

Whether the filter should ignore the comments (i.e. lines starting with #)

Specified by:

shouldIgnoreComments in class AbstractSimpleImporterProcessor

See Also:

AbstractSimpleImporterProcessor.COMMENT_START

shouldIgnoreFirstLine

protected boolean shouldIgnoreFirstLine()

Description copied from class: AbstractSimpleImporterProcessor

Template Method : Whether the processor should ignore the first line of the input

Specified by:

shouldIgnoreFirstLine in class AbstractSimpleImporterProcessor

Returns:

true if the processor should ignore first line

getNumberOfColumns

protected int getNumberOfColumns()

Specified by:

getNumberOfColumns in class AbstractSimpleImporterProcessor

Returns:

the number of fields the processed Geonames file should have

getNumberOfProcessors

public int getNumberOfProcessors()

getNumberOfTaskExecutors

public int getNumberOfTaskExecutors()

setIdGenerator

public void setIdGenerator(IIdGenerator idGenerator)

flushAndClear

protected void flushAndClear()

Description copied from class: AbstractSimpleImporterProcessor

Should flush and clear all the Daos that are used by the processor. This avoid memory leak

Specified by:

flushAndClear in class AbstractSimpleImporterProcessor

getNumberOfLinesToProcess

public long getNumberOfLinesToProcess()

Specified by:

getNumberOfLinesToProcess in interface IImporterProcessor

Overrides:

getNumberOfLinesToProcess in class AbstractSimpleImporterProcessor

Returns:

The number of line the processor will process. (it is not the number of lines left!)