cascading.tap.local

Class DirTap

java.lang.Object

cascading.tap.Tap<java.util.Properties,java.io.InputStream,java.io.OutputStream>

cascading.tap.local.FileTap

cascading.tap.local.DirTap

All Implemented Interfaces:

FlowElement, ScopedElement, FileType<java.util.Properties>, TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream>, Traceable, java.io.Serializable

public class DirTap
extends FileTap

Class DirTap processes all files in the given directory that match the given glob or regex.

A DirTap can be used as a sink or a source.

When used as a source, the given pattern and depth are used to identify input files from the filesystem.

When used as a sink, a single file is created in the directory for all the output data. The file is name is returned by FileTap.getOutputIdentifier(FlowProcess) and can be overridden (see getOutputFilename(FlowProcess) and getOutputFileBasename()).

When deleting the resource identified by this Tap, the value of FileTap.getOutputIdentifier(FlowProcess) will be deleted, if it exists.

DirTap must be used with the LocalFlowConnector to create Flow instances that run in "local" mode.

The given pattern must adhere to the syntax supported by FileSystem.getPathMatcher(String).

Or a sub-class may override getPathMatcher() and return a custom matcher.

The maxDepth parameter is the maximum number of levels of directories to visit. A value of 0 means that only the starting file is visited, unless denied by the security manager.

A value of MAX_VALUE (the default) may be used to indicate that all levels should be visited.

See Also:

Serialized Form

Field Summary

Fields inherited from interface cascading.tap.type.FileType

CASCADING_SOURCE_PATH

Constructor Summary

Constructors

Constructor and Description
DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.nio.file.Path directory) Constructor DirTap creates a new DirTap instance using the given Scheme and file directory.
DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.nio.file.Path directory, SinkMode sinkMode) Constructor DirTap creates a new DirTap instance using the given Scheme and file directory.
DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.nio.file.Path directory, java.lang.String pattern) Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, and pattern.
DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.nio.file.Path directory, java.lang.String pattern, int maxDepth) Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, pattern, and maxDepth
DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.nio.file.Path directory, java.lang.String pattern, int maxDepth, SinkMode sinkMode) Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, pattern, and maxDepth
DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.nio.file.Path directory, java.lang.String pattern, SinkMode sinkMode) Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, and pattern.
DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String directory) Constructor DirTap creates a new DirTap instance using the given Scheme and file directory.
DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String directory, SinkMode sinkMode) Constructor DirTap creates a new DirTap instance using the given Scheme and file directory.
DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String directory, java.lang.String pattern) Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, and pattern.
DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String directory, java.lang.String pattern, int maxDepth) Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, pattern, and maxDepth
DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String directory, java.lang.String pattern, int maxDepth, SinkMode sinkMode) Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, pattern, and maxDepth
DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String directory, java.lang.String pattern, SinkMode sinkMode) Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, and pattern.

Method Summary

Modifier and Type	Method and Description
protected static void	deleteChildren(java.nio.file.Path parentPath, java.lang.String[] childIdentifiers) Deletes the child files and their directories.
static boolean	deleteDirTap(DirTap dirTap, java.util.Properties conf) Method deleteDirTap will recursively delete all files referenced by the given DirTap.
boolean	deleteResource(java.util.Properties conf)
int	getMaxDepth()
protected java.lang.String	getOutputFileBasename()
protected java.lang.String	getOutputFilename(FlowProcess<? extends java.util.Properties> flowProcess)
protected java.lang.String	getOutputIdentifier(FlowProcess<? extends java.util.Properties> flowProcess) Only used with FileTap.openForWrite(FlowProcess, OutputStream) calls.
protected java.nio.file.PathMatcher	getPathMatcher()
java.lang.String	getPattern()
TupleEntryIterator	openForRead(FlowProcess<? extends java.util.Properties> flowProcess, java.io.InputStream input)
protected void	verify()

Methods inherited from class cascading.tap.local.FileTap

commitResource, create, createResource, getChildIdentifiers, getChildIdentifiers, getChildIdentifiers, getChildIdentifiers, getFullIdentifier, getIdentifier, getModifiedTime, getPath, getSize, getSize, isDirectory, isDirectory, openForWrite, resourceExists, withChildIdentifier, withScheme, withSinkMode

Methods inherited from class cascading.tap.Tap

createResource, deleteResource, entryStream, entryStream, entryStreamCopy, entryStreamCopy, equals, flowConfInit, getConfigDef, getFullIdentifier, getModifiedTime, getNodeConfigDef, getScheme, getSinkFields, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasNodeConfigDef, hasStepConfigDef, id, isKeep, isReplace, isSink, isSource, isTemporary, isUpdate, openForRead, openForReadUnchecked, openForWrite, outgoingScopeFor, prepareResourceForRead, prepareResourceForWrite, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, retrieveSinkFields, retrieveSourceFields, rollbackResource, setScheme, sinkConfInit, sourceConfInit, spliterator, splititerator, taps, toString, tupleStream, tupleStream, tupleStreamCopy, tupleStreamCopy

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface cascading.tap.type.TapWith

asTap

Constructor Detail

DirTap

public DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
              java.lang.String directory)

Constructor DirTap creates a new DirTap instance using the given Scheme and file directory.

Parameters:

scheme - of type Scheme

directory - of type String

DirTap

public DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
              java.lang.String directory,
              java.lang.String pattern)

Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, and pattern.

Parameters:

scheme - of type Scheme

directory - of type String

pattern - of type String

DirTap

public DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
              java.lang.String directory,
              java.lang.String pattern,
              int maxDepth)

Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, pattern, and maxDepth

Parameters:

scheme - of type Scheme

directory - of type String

pattern - of type String

maxDepth - of type int

DirTap

public DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
              java.lang.String directory,
              SinkMode sinkMode)

Constructor DirTap creates a new DirTap instance using the given Scheme and file directory.

Parameters:

scheme - of type Scheme

directory - of type String

sinkMode - of type SinkMode

DirTap

public DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
              java.lang.String directory,
              java.lang.String pattern,
              SinkMode sinkMode)

Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, and pattern.

Parameters:

scheme - of type Scheme

directory - of type String

pattern - of type String

sinkMode - of type SinkMode

DirTap

public DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
              java.lang.String directory,
              java.lang.String pattern,
              int maxDepth,
              SinkMode sinkMode)

Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, pattern, and maxDepth

Parameters:

scheme - of type Scheme

directory - of type String

pattern - of type String

maxDepth - of type int

sinkMode - of type SinkMode

DirTap

public DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
              java.nio.file.Path directory)

Constructor DirTap creates a new DirTap instance using the given Scheme and file directory.

Parameters:

scheme - of type Scheme

directory - of type Path

DirTap

public DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
              java.nio.file.Path directory,
              java.lang.String pattern)

Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, and pattern.

Parameters:

scheme - of type Scheme

directory - of type Path

pattern - of type String

DirTap

public DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
              java.nio.file.Path directory,
              java.lang.String pattern,
              int maxDepth)

Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, pattern, and maxDepth

Parameters:

scheme - of type Scheme

directory - of type Path

pattern - of type String

maxDepth - of type int

DirTap

public DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
              java.nio.file.Path directory,
              SinkMode sinkMode)

Constructor DirTap creates a new DirTap instance using the given Scheme and file directory.

Parameters:

scheme - of type Scheme

directory - of type Path

sinkMode - of type SinkMode

DirTap

public DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
              java.nio.file.Path directory,
              java.lang.String pattern,
              SinkMode sinkMode)

Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, and pattern.

Parameters:

scheme - of type Scheme

directory - of type Path

pattern - of type String

sinkMode - of type SinkMode

DirTap

public DirTap(Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
              java.nio.file.Path directory,
              java.lang.String pattern,
              int maxDepth,
              SinkMode sinkMode)

Constructor DirTap creates a new DirTap instance using the given Scheme, file directory, pattern, and maxDepth

Parameters:

scheme - of type Scheme

directory - of type Path

pattern - of type String

maxDepth - of type int

sinkMode - of type SinkMode

Method Detail

verify

protected void verify()

Overrides:

verify in class FileTap

getOutputIdentifier

protected java.lang.String getOutputIdentifier(FlowProcess<? extends java.util.Properties> flowProcess)

Description copied from class: FileTap

Only used with FileTap.openForWrite(FlowProcess, OutputStream) calls.

Overrides:

getOutputIdentifier in class FileTap

getOutputFilename

protected java.lang.String getOutputFilename(FlowProcess<? extends java.util.Properties> flowProcess)

getOutputFileBasename

protected java.lang.String getOutputFileBasename()

getPattern

public java.lang.String getPattern()

getMaxDepth

public int getMaxDepth()

deleteResource

public boolean deleteResource(java.util.Properties conf)
                       throws java.io.IOException

Overrides:

deleteResource in class FileTap

Throws:

java.io.IOException

openForRead

public TupleEntryIterator openForRead(FlowProcess<? extends java.util.Properties> flowProcess,
                                      java.io.InputStream input)
                               throws java.io.IOException

Overrides:

openForRead in class FileTap

Throws:

java.io.IOException

getPathMatcher

protected java.nio.file.PathMatcher getPathMatcher()

Overrides:

getPathMatcher in class FileTap

deleteDirTap

public static boolean deleteDirTap(DirTap dirTap,
                                   java.util.Properties conf)
                            throws java.io.IOException

Method deleteDirTap will recursively delete all files referenced by the given DirTap.

Parameters:

dirTap - the directory to delete

Throws:

java.io.IOException

deleteChildren

protected static void deleteChildren(java.nio.file.Path parentPath,
                                     java.lang.String[] childIdentifiers)
                              throws java.io.IOException

Deletes the child files and their directories. Does not delete the parent path.

Parameters:

parentPath -

childIdentifiers -

Throws:

java.io.IOException