Autopsy  4.7.0
Graphical digital forensics platform for The Sleuth Kit and other tools.
Classes | Private Member Functions | Static Private Member Functions | Private Attributes | Static Private Attributes | List of all members
org.sleuthkit.autopsy.ingest.DataSourceIngestJob Class Reference

Classes

class  Snapshot
 
enum  Stages
 

Private Member Functions

void addIngestModules (List< IngestModuleTemplate > templates, IngestModuleType type, SleuthkitCase skCase) throws TskCoreException
 
void checkForStageCompleted ()
 
void createIngestPipelines ()
 
void finish ()
 
void finishFirstStage ()
 
boolean hasFileIngestPipeline ()
 
boolean hasFirstStageDataSourceIngestPipeline ()
 
boolean hasSecondStageDataSourceIngestPipeline ()
 
void logIngestModuleErrors (List< IngestModuleError > errors)
 
void startDataSourceIngestProgressBar ()
 
void startFileIngestProgressBar ()
 
void startFirstStage ()
 
void startSecondStage ()
 
List< IngestModuleError > startUpIngestPipelines ()
 

Static Private Member Functions

static List< IngestModuleTemplate > getConfiguredIngestModuleTemplates (Map< String, IngestModuleTemplate > ingestModuleTemplates, List< String > pipelineConfig)
 

Private Attributes

volatile IngestJob.CancellationReason cancellationReason = IngestJob.CancellationReason.NOT_CANCELLED
 
volatile boolean cancelled
 
final List< String > cancelledDataSourceIngestModules = new CopyOnWriteArrayList<>()
 
final long createTime
 
volatile boolean currentDataSourceIngestModuleCancelled
 
DataSourceIngestPipeline currentDataSourceIngestPipeline
 
String currentFileIngestModule = ""
 
String currentFileIngestTask = ""
 
final Content dataSource
 
final Object dataSourceIngestPipelineLock = new Object()
 
ProgressHandle dataSourceIngestProgress
 
final Object dataSourceIngestProgressLock = new Object()
 
final boolean doUI
 
long estimatedFilesToProcess
 
final List< FileIngestPipeline > fileIngestPipelines = new ArrayList<>()
 
final LinkedBlockingQueue< FileIngestPipeline > fileIngestPipelinesQueue = new LinkedBlockingQueue<>()
 
ProgressHandle fileIngestProgress
 
final Object fileIngestProgressLock = new Object()
 
final List< AbstractFile > files = new ArrayList<>()
 
final List< String > filesInProgress = new ArrayList<>()
 
DataSourceIngestPipeline firstStageDataSourceIngestPipeline
 
final long id
 
volatile IngestJobInfo ingestJob
 
final List< IngestModuleInfo > ingestModules = new ArrayList<>()
 
final IngestJob parentJob
 
long processedFiles
 
DataSourceIngestPipeline secondStageDataSourceIngestPipeline
 
final IngestJobSettings settings
 
volatile Stages stage = DataSourceIngestJob.Stages.INITIALIZATION
 
final Object stageCompletionCheckLock = new Object()
 

Static Private Attributes

static final Logger logger = Logger.getLogger(DataSourceIngestJob.class.getName())
 
static final AtomicLong nextJobId = new AtomicLong(0L)
 
static final IngestTasksScheduler taskScheduler = IngestTasksScheduler.getInstance()
 

Detailed Description

Encapsulates a data source and the ingest module pipelines used to process it.

Definition at line 58 of file DataSourceIngestJob.java.

Member Function Documentation

void org.sleuthkit.autopsy.ingest.DataSourceIngestJob.addIngestModules ( List< IngestModuleTemplate >  templates,
IngestModuleType  type,
SleuthkitCase  skCase 
) throws TskCoreException
private

Definition at line 293 of file DataSourceIngestJob.java.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.createIngestPipelines().

void org.sleuthkit.autopsy.ingest.DataSourceIngestJob.checkForStageCompleted ( )
private

Checks to see if the ingest tasks for the current stage of this job are completed and does a stage transition if they are.

Definition at line 626 of file DataSourceIngestJob.java.

References org.sleuthkit.autopsy.ingest.DataSourceIngestJob.finish(), org.sleuthkit.autopsy.ingest.DataSourceIngestJob.finishFirstStage(), org.sleuthkit.autopsy.ingest.DataSourceIngestJob.stageCompletionCheckLock, and org.sleuthkit.autopsy.ingest.DataSourceIngestJob.taskScheduler.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startFirstStage().

void org.sleuthkit.autopsy.ingest.DataSourceIngestJob.createIngestPipelines ( )
private

Creates the file and data source ingest pipelines.

Make mappings of ingest module factory class names to templates.

Use the mappings and the ingest pipelines configuration to create ordered lists of ingest module templates for each ingest pipeline.

Add any module templates that were not specified in the pipelines configuration to an appropriate pipeline - either the first stage data source ingest pipeline or the file ingest pipeline.

Construct the data source ingest pipelines.

Construct the file ingest pipelines, one per file ingest thread.

The current thread was interrupted while blocked on a full queue. Blocking should actually never happen here, but reset the interrupted flag rather than just swallowing the exception.

Definition at line 221 of file DataSourceIngestJob.java.

References org.sleuthkit.autopsy.ingest.DataSourceIngestJob.addIngestModules(), org.sleuthkit.autopsy.ingest.DataSourceIngestJob.getConfiguredIngestModuleTemplates(), org.sleuthkit.autopsy.casemodule.Case.getCurrentCaseThrows(), org.sleuthkit.autopsy.ingest.IngestJobSettings.getEnabledIngestModuleTemplates(), org.sleuthkit.autopsy.ingest.IngestManager.getInstance(), org.sleuthkit.autopsy.ingest.IngestManager.getNumberOfFileIngestThreads(), and org.sleuthkit.autopsy.casemodule.Case.getSleuthkitCase().

void org.sleuthkit.autopsy.ingest.DataSourceIngestJob.finish ( )
private

Shuts down the ingest pipelines and progress bars for this job.

Definition at line 695 of file DataSourceIngestJob.java.

References org.sleuthkit.autopsy.ingest.DataSourceIngestJob.dataSourceIngestProgressLock, and org.sleuthkit.autopsy.ingest.DataSourceIngestJob.Stages.FINALIZATION.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.checkForStageCompleted(), and org.sleuthkit.autopsy.ingest.DataSourceIngestJob.finishFirstStage().

void org.sleuthkit.autopsy.ingest.DataSourceIngestJob.finishFirstStage ( )
private

Shuts down the first stage ingest pipelines and progress bars for this job and starts the second stage, if appropriate.

Start the second stage, if appropriate.

Definition at line 645 of file DataSourceIngestJob.java.

References org.sleuthkit.autopsy.ingest.DataSourceIngestJob.dataSourceIngestProgressLock, org.sleuthkit.autopsy.ingest.DataSourceIngestJob.fileIngestProgressLock, org.sleuthkit.autopsy.ingest.DataSourceIngestJob.finish(), org.sleuthkit.autopsy.ingest.DataSourceIngestJob.hasSecondStageDataSourceIngestPipeline(), org.sleuthkit.autopsy.ingest.DataSourceIngestJob.logIngestModuleErrors(), and org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startSecondStage().

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.checkForStageCompleted().

static List<IngestModuleTemplate> org.sleuthkit.autopsy.ingest.DataSourceIngestJob.getConfiguredIngestModuleTemplates ( Map< String, IngestModuleTemplate >  ingestModuleTemplates,
List< String >  pipelineConfig 
)
staticprivate

Uses an input collection of ingest module templates and a pipeline configuration, i.e., an ordered list of ingest module factory class names, to create an ordered output list of ingest module templates for an ingest pipeline. The ingest module templates are removed from the input collection as they are added to the output collection.

Parameters
ingestModuleTemplatesA mapping of ingest module factory class names to ingest module templates.
pipelineConfigAn ordered list of ingest module factory class names representing an ingest pipeline.
Returns
An ordered list of ingest module templates, i.e., an uninstantiated pipeline.

Definition at line 314 of file DataSourceIngestJob.java.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.createIngestPipelines().

boolean org.sleuthkit.autopsy.ingest.DataSourceIngestJob.hasFileIngestPipeline ( )
private

Checks to see if this job has a file level ingest pipeline.

Returns
True or false.

Definition at line 406 of file DataSourceIngestJob.java.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startFirstStage().

boolean org.sleuthkit.autopsy.ingest.DataSourceIngestJob.hasFirstStageDataSourceIngestPipeline ( )
private

Checks to see if this job has a first stage data source level ingest pipeline.

Returns
True or false.

Definition at line 387 of file DataSourceIngestJob.java.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startFirstStage().

boolean org.sleuthkit.autopsy.ingest.DataSourceIngestJob.hasSecondStageDataSourceIngestPipeline ( )
private

Checks to see if this job has a second stage data source level ingest pipeline.

Returns
True or false.

Definition at line 397 of file DataSourceIngestJob.java.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.finishFirstStage().

void org.sleuthkit.autopsy.ingest.DataSourceIngestJob.logIngestModuleErrors ( List< IngestModuleError >  errors)
private

Write ingest module errors to the log.

Parameters
errorsThe errors.

Definition at line 1074 of file DataSourceIngestJob.java.

References org.sleuthkit.autopsy.ingest.DataSourceIngestJob.id, and org.sleuthkit.autopsy.ingest.DataSourceIngestJob.logger.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.finishFirstStage(), and org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startUpIngestPipelines().

void org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startDataSourceIngestProgressBar ( )
private

Starts a data source level ingest progress bar for this job.

Definition at line 564 of file DataSourceIngestJob.java.

References org.sleuthkit.autopsy.ingest.DataSourceIngestJob.dataSourceIngestProgressLock, and org.sleuthkit.autopsy.ingest.IngestJob.CancellationReason.USER_CANCELLED.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startFirstStage(), and org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startSecondStage().

void org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startFileIngestProgressBar ( )
private

Starts the file level ingest progress bar for this job.

Definition at line 599 of file DataSourceIngestJob.java.

References org.sleuthkit.autopsy.ingest.DataSourceIngestJob.fileIngestProgressLock, and org.sleuthkit.autopsy.ingest.IngestJob.CancellationReason.USER_CANCELLED.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startFirstStage().

void org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startFirstStage ( )
private

Starts the first stage of this job.

Start one or both of the first stage ingest progress bars.

Make the first stage data source level ingest pipeline the current data source level pipeline.

Schedule the first stage tasks.

No data source ingest task has been scheduled for this stage, and it is possible, if unlikely, that no file ingest tasks were actually scheduled since there are files that get filtered out by the tasks scheduler. In this special case, an ingest thread will never get to check for completion of this stage of the job, so do it now.

Definition at line 491 of file DataSourceIngestJob.java.

References org.sleuthkit.autopsy.ingest.DataSourceIngestJob.checkForStageCompleted(), org.sleuthkit.autopsy.ingest.DataSourceIngestJob.dataSourceIngestPipelineLock, org.sleuthkit.autopsy.ingest.DataSourceIngestJob.fileIngestProgressLock, org.sleuthkit.autopsy.ingest.DataSourceIngestJob.Stages.FIRST, org.sleuthkit.autopsy.ingest.DataSourceIngestJob.firstStageDataSourceIngestPipeline, org.sleuthkit.autopsy.ingest.DataSourceIngestJob.hasFileIngestPipeline(), org.sleuthkit.autopsy.ingest.DataSourceIngestJob.hasFirstStageDataSourceIngestPipeline(), org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startDataSourceIngestProgressBar(), org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startFileIngestProgressBar(), and org.sleuthkit.autopsy.ingest.DataSourceIngestJob.taskScheduler.

void org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startSecondStage ( )
private

Starts the second stage of this ingest job.

Definition at line 548 of file DataSourceIngestJob.java.

References org.sleuthkit.autopsy.ingest.DataSourceIngestJob.dataSourceIngestPipelineLock, org.sleuthkit.autopsy.ingest.DataSourceIngestJob.Stages.SECOND, org.sleuthkit.autopsy.ingest.DataSourceIngestJob.secondStageDataSourceIngestPipeline, org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startDataSourceIngestProgressBar(), and org.sleuthkit.autopsy.ingest.DataSourceIngestJob.taskScheduler.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.finishFirstStage().

List<IngestModuleError> org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startUpIngestPipelines ( )
private

Starts up each of the ingest pipelines for this job to collect any file and data source level ingest modules errors that might occur.

Returns
A collection of ingest module startup errors, empty on success.

Definition at line 443 of file DataSourceIngestJob.java.

References org.sleuthkit.autopsy.ingest.DataSourceIngestJob.logIngestModuleErrors().

Member Data Documentation

volatile IngestJob.CancellationReason org.sleuthkit.autopsy.ingest.DataSourceIngestJob.cancellationReason = IngestJob.CancellationReason.NOT_CANCELLED
private

Definition at line 139 of file DataSourceIngestJob.java.

volatile boolean org.sleuthkit.autopsy.ingest.DataSourceIngestJob.cancelled
private

Definition at line 138 of file DataSourceIngestJob.java.

final List<String> org.sleuthkit.autopsy.ingest.DataSourceIngestJob.cancelledDataSourceIngestModules = new CopyOnWriteArrayList<>()
private

Definition at line 137 of file DataSourceIngestJob.java.

final long org.sleuthkit.autopsy.ingest.DataSourceIngestJob.createTime
private

A data source ingest job uses this field to report its creation time.

Definition at line 177 of file DataSourceIngestJob.java.

volatile boolean org.sleuthkit.autopsy.ingest.DataSourceIngestJob.currentDataSourceIngestModuleCancelled
private

A data source ingest job supports cancellation of either the currently running data source level ingest module or the entire ingest job.

TODO: The currentDataSourceIngestModuleCancelled field and all of the code concerned with it is a hack to avoid an API change. The next time an API change is legal, a cancel() method needs to be added to the IngestModule interface and this field should be removed. The "ingest job is canceled" queries should also be removed from the IngestJobContext class.

Definition at line 136 of file DataSourceIngestJob.java.

DataSourceIngestPipeline org.sleuthkit.autopsy.ingest.DataSourceIngestJob.currentDataSourceIngestPipeline
private

Definition at line 113 of file DataSourceIngestJob.java.

String org.sleuthkit.autopsy.ingest.DataSourceIngestJob.currentFileIngestModule = ""
private

Definition at line 169 of file DataSourceIngestJob.java.

String org.sleuthkit.autopsy.ingest.DataSourceIngestJob.currentFileIngestTask = ""
private

Definition at line 170 of file DataSourceIngestJob.java.

final Content org.sleuthkit.autopsy.ingest.DataSourceIngestJob.dataSource
private

Definition at line 72 of file DataSourceIngestJob.java.

final Object org.sleuthkit.autopsy.ingest.DataSourceIngestJob.dataSourceIngestPipelineLock = new Object()
private

A data source ingest job has separate data source level ingest module pipelines for the first and second processing stages. Longer running, lower priority modules belong in the second stage pipeline, although this cannot be enforced. Note that the pipelines for both stages are created at job start up to allow for verification that they both can be started up without errors.

Definition at line 110 of file DataSourceIngestJob.java.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startFirstStage(), and org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startSecondStage().

ProgressHandle org.sleuthkit.autopsy.ingest.DataSourceIngestJob.dataSourceIngestProgress
private

Definition at line 158 of file DataSourceIngestJob.java.

final Object org.sleuthkit.autopsy.ingest.DataSourceIngestJob.dataSourceIngestProgressLock = new Object()
private

A data source ingest job uses these fields to report data source level ingest progress.

Definition at line 157 of file DataSourceIngestJob.java.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.finish(), org.sleuthkit.autopsy.ingest.DataSourceIngestJob.finishFirstStage(), and org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startDataSourceIngestProgressBar().

final boolean org.sleuthkit.autopsy.ingest.DataSourceIngestJob.doUI
private

A data source ingest job can run interactively using NetBeans progress handles.

Definition at line 151 of file DataSourceIngestJob.java.

long org.sleuthkit.autopsy.ingest.DataSourceIngestJob.estimatedFilesToProcess
private

Definition at line 166 of file DataSourceIngestJob.java.

final List<FileIngestPipeline> org.sleuthkit.autopsy.ingest.DataSourceIngestJob.fileIngestPipelines = new ArrayList<>()
private

Definition at line 123 of file DataSourceIngestJob.java.

final LinkedBlockingQueue<FileIngestPipeline> org.sleuthkit.autopsy.ingest.DataSourceIngestJob.fileIngestPipelinesQueue = new LinkedBlockingQueue<>()
private

A data source ingest job has a collection of identical file level ingest module pipelines, one for each file level ingest thread in the ingest manager. A blocking queue is used to dole out the pipelines to the threads and an ordinary list is used when the ingest job needs to access the pipelines to query their status.

Definition at line 122 of file DataSourceIngestJob.java.

ProgressHandle org.sleuthkit.autopsy.ingest.DataSourceIngestJob.fileIngestProgress
private

Definition at line 168 of file DataSourceIngestJob.java.

final Object org.sleuthkit.autopsy.ingest.DataSourceIngestJob.fileIngestProgressLock = new Object()
private

A data source ingest job uses these fields to report file level ingest progress.

Definition at line 164 of file DataSourceIngestJob.java.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.finishFirstStage(), org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startFileIngestProgressBar(), and org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startFirstStage().

final List<AbstractFile> org.sleuthkit.autopsy.ingest.DataSourceIngestJob.files = new ArrayList<>()
private

Definition at line 73 of file DataSourceIngestJob.java.

final List<String> org.sleuthkit.autopsy.ingest.DataSourceIngestJob.filesInProgress = new ArrayList<>()
private

Definition at line 165 of file DataSourceIngestJob.java.

DataSourceIngestPipeline org.sleuthkit.autopsy.ingest.DataSourceIngestJob.firstStageDataSourceIngestPipeline
private

Definition at line 111 of file DataSourceIngestJob.java.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startFirstStage().

final long org.sleuthkit.autopsy.ingest.DataSourceIngestJob.id
private

Definition at line 70 of file DataSourceIngestJob.java.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.logIngestModuleErrors().

volatile IngestJobInfo org.sleuthkit.autopsy.ingest.DataSourceIngestJob.ingestJob
private

Definition at line 172 of file DataSourceIngestJob.java.

final List<IngestModuleInfo> org.sleuthkit.autopsy.ingest.DataSourceIngestJob.ingestModules = new ArrayList<>()
private

Definition at line 171 of file DataSourceIngestJob.java.

final Logger org.sleuthkit.autopsy.ingest.DataSourceIngestJob.logger = Logger.getLogger(DataSourceIngestJob.class.getName())
staticprivate

Definition at line 60 of file DataSourceIngestJob.java.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.logIngestModuleErrors().

final AtomicLong org.sleuthkit.autopsy.ingest.DataSourceIngestJob.nextJobId = new AtomicLong(0L)
staticprivate

Definition at line 69 of file DataSourceIngestJob.java.

final IngestJob org.sleuthkit.autopsy.ingest.DataSourceIngestJob.parentJob
private

These fields define a data source ingest job: the parent ingest job, an ID, the user's ingest job settings, and the data source to be analyzed. Optionally, there is a set of files to be analyzed instead of analyzing all of the files in the data source.

Definition at line 68 of file DataSourceIngestJob.java.

long org.sleuthkit.autopsy.ingest.DataSourceIngestJob.processedFiles
private

Definition at line 167 of file DataSourceIngestJob.java.

DataSourceIngestPipeline org.sleuthkit.autopsy.ingest.DataSourceIngestJob.secondStageDataSourceIngestPipeline
private

Definition at line 112 of file DataSourceIngestJob.java.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startSecondStage().

final IngestJobSettings org.sleuthkit.autopsy.ingest.DataSourceIngestJob.settings
private

Definition at line 71 of file DataSourceIngestJob.java.

volatile Stages org.sleuthkit.autopsy.ingest.DataSourceIngestJob.stage = DataSourceIngestJob.Stages.INITIALIZATION
private

Definition at line 99 of file DataSourceIngestJob.java.

final Object org.sleuthkit.autopsy.ingest.DataSourceIngestJob.stageCompletionCheckLock = new Object()
private

Definition at line 100 of file DataSourceIngestJob.java.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.checkForStageCompleted().

final IngestTasksScheduler org.sleuthkit.autopsy.ingest.DataSourceIngestJob.taskScheduler = IngestTasksScheduler.getInstance()
staticprivate

A data source ingest job uses the task scheduler singleton to create and queue the ingest tasks that make up the job.

Definition at line 145 of file DataSourceIngestJob.java.

Referenced by org.sleuthkit.autopsy.ingest.DataSourceIngestJob.checkForStageCompleted(), org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startFirstStage(), and org.sleuthkit.autopsy.ingest.DataSourceIngestJob.startSecondStage().

The documentation for this class was generated from the following file:

Copyright © 2012-2016 Basis Technology. Generated on: Mon Jun 18 2018
This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.