cascading.scheme.hadoop

Class TextLine

java.lang.Object

cascading.scheme.Scheme<Configuration,RecordReader,OutputCollector,java.lang.Object[],java.lang.Object[]>

cascading.scheme.hadoop.TextLine

All Implemented Interfaces:

FileFormat, Traceable, java.io.Serializable

Direct Known Subclasses:

TextDelimited

public class TextLine
extends Scheme<Configuration,RecordReader,OutputCollector,java.lang.Object[],java.lang.Object[]>
implements FileFormat

A TextLine is a type of Scheme for plain text files. Files are broken into lines. Either line-feed or carriage-return are used to signal end of line.

By default, this scheme returns a Tuple with two fields, "offset" and "line".

Many of the constructors take both "sourceFields" and "sinkFields". sourceFields denote the field names to be used instead of the names "offset" and "line". sinkFields is a selector and is by default Fields.ALL. Any available field names can be given if only a subset of the incoming fields should be used.

If a Fields instance is passed on the constructor as sourceFields having only one field, the return tuples will simply be the "line" value using the given field name.

Note that TextLine will concatenate all the Tuple values for the selected fields with a TAB delimiter before writing out the line.

Note sink compression is TextLine.Compress.DISABLE by default. If null is passed to the constructor for the compression value, it will remain disabled.

If any of the input files end with ".zip", an error will be thrown. *

By default, all text is encoded/decoded as UTF-8. This can be changed via the charsetName constructor argument.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TextLine.Compress

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_CHARSET
static Fields	DEFAULT_SOURCE_FIELDS Field DEFAULT_SOURCE_FIELDS

Constructor Summary

Constructors

Constructor and Description
TextLine() Creates a new TextLine instance that sources "offset" and "line" fields, and sinks all incoming fields, where "offset" is the byte offset in the input file.
TextLine(Fields sourceFields) Creates a new TextLine instance.
TextLine(Fields sourceFields, Fields sinkFields) Creates a new TextLine instance.
TextLine(Fields sourceFields, Fields sinkFields, int numSinkParts) Creates a new TextLine instance.
TextLine(Fields sourceFields, Fields sinkFields, java.lang.String charsetName) Creates a new TextLine instance.
TextLine(Fields sourceFields, Fields sinkFields, TextLine.Compress sinkCompression) Constructor TextLine creates a new TextLine instance.
TextLine(Fields sourceFields, Fields sinkFields, TextLine.Compress sinkCompression, int numSinkParts) Constructor TextLine creates a new TextLine instance.
TextLine(Fields sourceFields, Fields sinkFields, TextLine.Compress sinkCompression, int numSinkParts, java.lang.String charsetName) Constructor TextLine creates a new TextLine instance.
TextLine(Fields sourceFields, Fields sinkFields, TextLine.Compress sinkCompression, java.lang.String charsetName) Constructor TextLine creates a new TextLine instance.
TextLine(Fields sourceFields, int numSinkParts) Creates a new TextLine instance.
TextLine(Fields sourceFields, java.lang.String charsetName) Creates a new TextLine instance.
TextLine(int numSinkParts) Creates a new TextLine instance that sources "offset" and "line" fields, and sinks all incoming fields, where "offset" is the byte offset in the input file.
TextLine(TextLine.Compress sinkCompression) Creates a new TextLine instance that sources "offset" and "line" fields, and sinks all incoming fields, where "offset" is the byte offset in the input file.

Method Summary

Modifier and Type	Method and Description
java.lang.String	getCharsetName()
java.lang.String	getExtension()
TextLine.Compress	getSinkCompression() Method getSinkCompression returns the sinkCompression of this TextLine object.
protected java.lang.String	makeEncodedString(java.lang.Object[] context)
void	presentSinkFields(FlowProcess<? extends Configuration> flowProcess, Tap tap, Fields fields)
void	presentSourceFields(FlowProcess<? extends Configuration> flowProcess, Tap tap, Fields fields)
protected void	setCharsetName(java.lang.String charsetName)
void	setSinkCompression(TextLine.Compress sinkCompression) Method setSinkCompression sets the sinkCompression of this TextLine object.
void	sink(FlowProcess<? extends Configuration> flowProcess, SinkCall<java.lang.Object[],OutputCollector> sinkCall)
void	sinkConfInit(FlowProcess<? extends Configuration> flowProcess, Tap<Configuration,RecordReader,OutputCollector> tap, Configuration conf)
void	sinkPrepare(FlowProcess<? extends Configuration> flowProcess, SinkCall<java.lang.Object[],OutputCollector> sinkCall)
boolean	source(FlowProcess<? extends Configuration> flowProcess, SourceCall<java.lang.Object[],RecordReader> sourceCall)
void	sourceCleanup(FlowProcess<? extends Configuration> flowProcess, SourceCall<java.lang.Object[],RecordReader> sourceCall)
void	sourceConfInit(FlowProcess<? extends Configuration> flowProcess, Tap<Configuration,RecordReader,OutputCollector> tap, Configuration conf)
protected void	sourceHandleInput(SourceCall<java.lang.Object[],RecordReader> sourceCall)
void	sourcePrepare(FlowProcess<? extends Configuration> flowProcess, SourceCall<java.lang.Object[],RecordReader> sourceCall)
protected void	verify(Fields sourceFields)

Methods inherited from class cascading.scheme.Scheme

equals, getNumSinkParts, getSinkFields, getSourceFields, getTrace, hashCode, isSink, isSource, isSymmetrical, presentSinkFieldsInternal, presentSourceFieldsInternal, retrieveSinkFields, retrieveSourceFields, setNumSinkParts, setSinkFields, setSourceFields, sinkCleanup, sinkWrap, sourceRePrepare, sourceWrap, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_CHARSET

public static final java.lang.String DEFAULT_CHARSET

See Also:

Constant Field Values

DEFAULT_SOURCE_FIELDS

public static final Fields DEFAULT_SOURCE_FIELDS

Field DEFAULT_SOURCE_FIELDS

Constructor Detail

TextLine

public TextLine()

Creates a new TextLine instance that sources "offset" and "line" fields, and sinks all incoming fields, where "offset" is the byte offset in the input file.

TextLine

@ConstructorProperties(value="numSinkParts")
public TextLine(int numSinkParts)

Creates a new TextLine instance that sources "offset" and "line" fields, and sinks all incoming fields, where "offset" is the byte offset in the input file.

Parameters:

numSinkParts - of type int

TextLine

@ConstructorProperties(value="sinkCompression")
public TextLine(TextLine.Compress sinkCompression)

Creates a new TextLine instance that sources "offset" and "line" fields, and sinks all incoming fields, where "offset" is the byte offset in the input file.

Parameters:

sinkCompression - of type Compress

TextLine

@ConstructorProperties(value={"sourceFields","sinkFields"})
public TextLine(Fields sourceFields,
                                                                            Fields sinkFields)

Creates a new TextLine instance. If sourceFields has one field, only the text line will be returned in the subsequent tuples.

Parameters:

sourceFields - the source fields for this scheme

sinkFields - the sink fields for this scheme

TextLine

@ConstructorProperties(value={"sourceFields","sinkFields","charsetName"})
public TextLine(Fields sourceFields,
                                                                                          Fields sinkFields,
                                                                                          java.lang.String charsetName)

Creates a new TextLine instance. If sourceFields has one field, only the text line will be returned in the subsequent tuples.

Parameters:

sourceFields - the source fields for this scheme

sinkFields - the sink fields for this scheme

charsetName - of type String

TextLine

@ConstructorProperties(value={"sourceFields","sinkFields","numSinkParts"})
public TextLine(Fields sourceFields,
                                                                                           Fields sinkFields,
                                                                                           int numSinkParts)

Creates a new TextLine instance. If sourceFields has one field, only the text line will be returned in the subsequent tuples.

Parameters:

sourceFields - the source fields for this scheme

sinkFields - the sink fields for this scheme

numSinkParts - of type int

TextLine

@ConstructorProperties(value={"sourceFields","sinkFields","sinkCompression"})
public TextLine(Fields sourceFields,
                                                                                              Fields sinkFields,
                                                                                              TextLine.Compress sinkCompression)

Constructor TextLine creates a new TextLine instance. If sourceFields has one field, only the text line will be returned in the subsequent tuples.

Parameters:

sourceFields - of type Fields

sinkFields - of type Fields

sinkCompression - of type Compress

TextLine

@ConstructorProperties(value={"sourceFields","sinkFields","sinkCompression","charsetName"})
public TextLine(Fields sourceFields,
                                                                                                            Fields sinkFields,
                                                                                                            TextLine.Compress sinkCompression,
                                                                                                            java.lang.String charsetName)

Constructor TextLine creates a new TextLine instance. If sourceFields has one field, only the text line will be returned in the subsequent tuples.

Parameters:

sourceFields - of type Fields

sinkFields - of type Fields

sinkCompression - of type Compress

charsetName - of type String

TextLine

@ConstructorProperties(value={"sourceFields","sinkFields","sinkCompression","numSinkParts"})
public TextLine(Fields sourceFields,
                                                                                                             Fields sinkFields,
                                                                                                             TextLine.Compress sinkCompression,
                                                                                                             int numSinkParts)

Constructor TextLine creates a new TextLine instance. If sourceFields has one field, only the text line will be returned in the subsequent tuples.

Parameters:

sourceFields - of type Fields

sinkFields - of type Fields

sinkCompression - of type Compress

numSinkParts - of type int

TextLine

@ConstructorProperties(value={"sourceFields","sinkFields","sinkCompression","numSinkParts","charsetName"})
public TextLine(Fields sourceFields,
                                                                                                                           Fields sinkFields,
                                                                                                                           TextLine.Compress sinkCompression,
                                                                                                                           int numSinkParts,
                                                                                                                           java.lang.String charsetName)

Constructor TextLine creates a new TextLine instance. If sourceFields has one field, only the text line will be returned in the subsequent tuples.

Parameters:

sourceFields - of type Fields

sinkFields - of type Fields

sinkCompression - of type Compress

numSinkParts - of type int

charsetName - of type String

TextLine

@ConstructorProperties(value="sourceFields")
public TextLine(Fields sourceFields)

Creates a new TextLine instance. If sourceFields has one field, only the text line will be returned in the subsequent tuples.

Parameters:

sourceFields - the source fields for this scheme

TextLine

@ConstructorProperties(value={"sourceFields","charsetName"})
public TextLine(Fields sourceFields,
                                                                             java.lang.String charsetName)

Creates a new TextLine instance. If sourceFields has one field, only the text line will be returned in the subsequent tuples.

Parameters:

sourceFields - the source fields for this scheme

charsetName - of type String

TextLine

@ConstructorProperties(value={"sourceFields","numSinkParts"})
public TextLine(Fields sourceFields,
                                                                              int numSinkParts)

Creates a new TextLine instance. If sourceFields has one field, only the text line will be returned in the subsequent tuples. The resulting data set will have numSinkParts.

Parameters:

sourceFields - the source fields for this scheme

numSinkParts - of type int

Method Detail

setCharsetName

protected void setCharsetName(java.lang.String charsetName)

getCharsetName

public java.lang.String getCharsetName()

verify

protected void verify(Fields sourceFields)

getSinkCompression

public TextLine.Compress getSinkCompression()

Method getSinkCompression returns the sinkCompression of this TextLine object.

Returns:

the sinkCompression (type Compress) of this TextLine object.

setSinkCompression

public void setSinkCompression(TextLine.Compress sinkCompression)

Method setSinkCompression sets the sinkCompression of this TextLine object. If null, compression will remain disabled.

Parameters:

sinkCompression - the sinkCompression of this TextLine object.

sourceConfInit

public void sourceConfInit(FlowProcess<? extends Configuration> flowProcess,
                           Tap<Configuration,RecordReader,OutputCollector> tap,
                           Configuration conf)

Specified by:

sourceConfInit in class Scheme<Configuration,RecordReader,OutputCollector,java.lang.Object[],java.lang.Object[]>

presentSourceFields

public void presentSourceFields(FlowProcess<? extends Configuration> flowProcess,
                                Tap tap,
                                Fields fields)

Overrides:

presentSourceFields in class Scheme<Configuration,RecordReader,OutputCollector,java.lang.Object[],java.lang.Object[]>

presentSinkFields

public void presentSinkFields(FlowProcess<? extends Configuration> flowProcess,
                              Tap tap,
                              Fields fields)

Overrides:

presentSinkFields in class Scheme<Configuration,RecordReader,OutputCollector,java.lang.Object[],java.lang.Object[]>

sinkConfInit

public void sinkConfInit(FlowProcess<? extends Configuration> flowProcess,
                         Tap<Configuration,RecordReader,OutputCollector> tap,
                         Configuration conf)

Specified by:

sinkConfInit in class Scheme<Configuration,RecordReader,OutputCollector,java.lang.Object[],java.lang.Object[]>

sourcePrepare

public void sourcePrepare(FlowProcess<? extends Configuration> flowProcess,
                          SourceCall<java.lang.Object[],RecordReader> sourceCall)

Overrides:

sourcePrepare in class Scheme<Configuration,RecordReader,OutputCollector,java.lang.Object[],java.lang.Object[]>

source

public boolean source(FlowProcess<? extends Configuration> flowProcess,
                      SourceCall<java.lang.Object[],RecordReader> sourceCall)
               throws java.io.IOException

Specified by:

source in class Scheme<Configuration,RecordReader,OutputCollector,java.lang.Object[],java.lang.Object[]>

Throws:

java.io.IOException

sourceHandleInput

protected void sourceHandleInput(SourceCall<java.lang.Object[],RecordReader> sourceCall)
                          throws java.io.IOException

Throws:

java.io.IOException

makeEncodedString

protected java.lang.String makeEncodedString(java.lang.Object[] context)

sourceCleanup

public void sourceCleanup(FlowProcess<? extends Configuration> flowProcess,
                          SourceCall<java.lang.Object[],RecordReader> sourceCall)

Overrides:

sourceCleanup in class Scheme<Configuration,RecordReader,OutputCollector,java.lang.Object[],java.lang.Object[]>

sinkPrepare

public void sinkPrepare(FlowProcess<? extends Configuration> flowProcess,
                        SinkCall<java.lang.Object[],OutputCollector> sinkCall)
                 throws java.io.IOException

Overrides:

sinkPrepare in class Scheme<Configuration,RecordReader,OutputCollector,java.lang.Object[],java.lang.Object[]>

Throws:

java.io.IOException

sink

public void sink(FlowProcess<? extends Configuration> flowProcess,
                 SinkCall<java.lang.Object[],OutputCollector> sinkCall)
          throws java.io.IOException

Specified by:

sink in class Scheme<Configuration,RecordReader,OutputCollector,java.lang.Object[],java.lang.Object[]>

Throws:

java.io.IOException

getExtension

public java.lang.String getExtension()

Specified by:

getExtension in interface FileFormat