cascading.scheme.util

Class DelimitedParser

java.lang.Object

cascading.scheme.util.DelimitedParser

All Implemented Interfaces:

java.io.Serializable

public class DelimitedParser
extends java.lang.Object
implements java.io.Serializable

Class DelimitedParser is a base class for parsing text delimited files.

It maybe sub-classed to change its behavior.

The interface FieldTypeResolver maybe used to clean and prepare field names for data columns, and to infer type information from column names.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.regex.Pattern	cleanPattern Field cleanPattern
protected CoercibleType[]	coercibles Fields coercibles
protected java.lang.String	delimiter Field delimiter *
protected boolean	enforceStrict Field enforceStrict
protected java.util.regex.Pattern	escapePattern Field escapePattern
protected FieldTypeResolver	fieldTypeResolver fieldTypeResolver
protected int	numValues Field numValues
protected java.lang.String	quote Field quote
protected boolean	safe Field safe
protected Fields	sourceFields Field sourceFields
protected java.util.regex.Pattern	splitPattern Field splitPattern
protected boolean	strict Field strict
protected java.lang.reflect.Type[]	types Field types

Constructor Summary

Constructors

Constructor and Description
DelimitedParser(java.lang.String delimiter, java.lang.String quote, java.lang.Class[] types)
DelimitedParser(java.lang.String delimiter, java.lang.String quote, java.lang.Class[] types, boolean strict, boolean safe)
DelimitedParser(java.lang.String delimiter, java.lang.String quote, java.lang.Class[] types, boolean strict, boolean safe, Fields sourceFields, Fields sinkFields)
DelimitedParser(java.lang.String delimiter, java.lang.String quote, java.lang.Class[] types, boolean strict, boolean safe, Fields sourceFields, Fields sinkFields, FieldTypeResolver fieldTypeResolver)
DelimitedParser(java.lang.String delimiter, java.lang.String quote, java.lang.Class[] types, boolean strict, boolean safe, FieldTypeResolver fieldTypeResolver)
DelimitedParser(java.lang.String delimiter, java.lang.String quote, FieldTypeResolver fieldTypeResolver)

Method Summary

Modifier and Type	Method and Description
protected java.lang.Object[]	cleanFields(java.lang.Object[] result)
protected java.lang.Object[]	cleanParsedLine(java.lang.Object[] split)
java.lang.Object[]	cleanSplit(java.lang.Object[] split, java.util.regex.Pattern cleanPattern, java.util.regex.Pattern escapePattern, java.lang.String quote) Method cleanSplit will return a quote free array of String values, the given split array will be updated in place.
protected java.lang.Object[]	coerceParsedLine(java.lang.String line, java.lang.Object[] split)
java.util.regex.Pattern	createCleanPatternFor(java.lang.String quote) Method createCleanPatternFor creates a regex Pattern for removing quote characters from a String.
java.util.regex.Pattern	createEscapePatternFor(java.lang.String quote) Method createEscapePatternFor creates a regex Pattern cleaning quote escapes from a String.
java.lang.String[]	createSplit(java.lang.String value, java.util.regex.Pattern splitPattern, int numValues) Method createSplit will split the given value with the given splitPattern.
java.util.regex.Pattern	createSplitPatternFor(java.lang.String delimiter, java.lang.String quote) Method createSplitPatternFor creates a regex Pattern for splitting a line of text into its component parts using the given delimiter and quote Strings.
java.lang.String	getDelimiter()
java.lang.String	getQuote()
protected java.lang.reflect.Type[]	inferTypes(java.lang.Object[] result)
java.lang.Appendable	joinFirstLine(java.lang.Iterable iterable, java.lang.Appendable buffer)
java.lang.Appendable	joinLine(java.lang.Iterable iterable, java.lang.Appendable buffer)
protected java.lang.Appendable	joinNoQuote(java.lang.Iterable tuple, java.lang.Appendable buffer)
protected java.lang.Appendable	joinWithQuote(java.lang.Iterable tuple, java.lang.Appendable buffer)
protected java.lang.Object[]	onlyParseLine(java.lang.String line)
Fields	parseFirstLine(FlowProcess flowProcess, Tap tap)
java.lang.Object[]	parseLine(java.lang.String line)
protected java.lang.Iterable	prepareFields(java.lang.Iterable fields)
void	reset(Fields sourceFields, Fields sinkFields)
void	reset(java.lang.String delimiter, java.lang.String quote, java.lang.reflect.Type[] types, boolean strict, boolean safe, Fields sourceFields, Fields sinkFields, FieldTypeResolver fieldTypeResolver)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

sourceFields

protected Fields sourceFields

Field sourceFields

splitPattern

protected java.util.regex.Pattern splitPattern

Field splitPattern

cleanPattern

protected java.util.regex.Pattern cleanPattern

Field cleanPattern

escapePattern

protected java.util.regex.Pattern escapePattern

Field escapePattern

delimiter

protected java.lang.String delimiter

Field delimiter *

quote

protected java.lang.String quote

Field quote

strict

protected boolean strict

Field strict

enforceStrict

protected boolean enforceStrict

Field enforceStrict

numValues

protected int numValues

Field numValues

types

protected java.lang.reflect.Type[] types

Field types

coercibles

protected CoercibleType[] coercibles

Fields coercibles

safe

protected boolean safe

Field safe

fieldTypeResolver

protected FieldTypeResolver fieldTypeResolver

fieldTypeResolver

Constructor Detail

DelimitedParser

public DelimitedParser(java.lang.String delimiter,
                       java.lang.String quote,
                       java.lang.Class[] types)

DelimitedParser

public DelimitedParser(java.lang.String delimiter,
                       java.lang.String quote,
                       java.lang.Class[] types,
                       boolean strict,
                       boolean safe)

DelimitedParser

public DelimitedParser(java.lang.String delimiter,
                       java.lang.String quote,
                       FieldTypeResolver fieldTypeResolver)

DelimitedParser

public DelimitedParser(java.lang.String delimiter,
                       java.lang.String quote,
                       java.lang.Class[] types,
                       boolean strict,
                       boolean safe,
                       FieldTypeResolver fieldTypeResolver)

DelimitedParser

public DelimitedParser(java.lang.String delimiter,
                       java.lang.String quote,
                       java.lang.Class[] types,
                       boolean strict,
                       boolean safe,
                       Fields sourceFields,
                       Fields sinkFields)

DelimitedParser

public DelimitedParser(java.lang.String delimiter,
                       java.lang.String quote,
                       java.lang.Class[] types,
                       boolean strict,
                       boolean safe,
                       Fields sourceFields,
                       Fields sinkFields,
                       FieldTypeResolver fieldTypeResolver)

Method Detail

reset

public void reset(Fields sourceFields,
                  Fields sinkFields)

reset

public void reset(java.lang.String delimiter,
                  java.lang.String quote,
                  java.lang.reflect.Type[] types,
                  boolean strict,
                  boolean safe,
                  Fields sourceFields,
                  Fields sinkFields,
                  FieldTypeResolver fieldTypeResolver)

getDelimiter

public java.lang.String getDelimiter()

getQuote

public java.lang.String getQuote()

createEscapePatternFor

public java.util.regex.Pattern createEscapePatternFor(java.lang.String quote)

Method createEscapePatternFor creates a regex Pattern cleaning quote escapes from a String.

If quote is null or empty, a null value will be returned;

Parameters:

quote - of type String

Returns:

Pattern

createCleanPatternFor

public java.util.regex.Pattern createCleanPatternFor(java.lang.String quote)

Method createCleanPatternFor creates a regex Pattern for removing quote characters from a String.

If quote is null or empty, a null value will be returned;

Parameters:

quote - of type String

Returns:

Pattern

createSplitPatternFor

public java.util.regex.Pattern createSplitPatternFor(java.lang.String delimiter,
                                                     java.lang.String quote)

Method createSplitPatternFor creates a regex Pattern for splitting a line of text into its component parts using the given delimiter and quote Strings. quote may be null.

Parameters:

delimiter - of type String

quote - of type String

Returns:

Pattern

createSplit

public java.lang.String[] createSplit(java.lang.String value,
                                      java.util.regex.Pattern splitPattern,
                                      int numValues)

Method createSplit will split the given value with the given splitPattern.

Parameters:

value - of type String

splitPattern - of type Pattern

numValues - of type int

Returns:

String[]

cleanSplit

public java.lang.Object[] cleanSplit(java.lang.Object[] split,
                                     java.util.regex.Pattern cleanPattern,
                                     java.util.regex.Pattern escapePattern,
                                     java.lang.String quote)

Method cleanSplit will return a quote free array of String values, the given split array will be updated in place.

If cleanPattern is null, quote cleaning will not be performed, but all empty String values will be replaces with a null value.

Parameters:

split - of type Object[]

cleanPattern - of type Pattern

escapePattern - of type Pattern

quote - of type String

Returns:

Object[] as a convenience

parseFirstLine

public Fields parseFirstLine(FlowProcess flowProcess,
                             Tap tap)

parseLine

public java.lang.Object[] parseLine(java.lang.String line)

cleanParsedLine

protected java.lang.Object[] cleanParsedLine(java.lang.Object[] split)

coerceParsedLine

protected java.lang.Object[] coerceParsedLine(java.lang.String line,
                                              java.lang.Object[] split)

onlyParseLine

protected java.lang.Object[] onlyParseLine(java.lang.String line)

joinFirstLine

public java.lang.Appendable joinFirstLine(java.lang.Iterable iterable,
                                          java.lang.Appendable buffer)

joinLine

public java.lang.Appendable joinLine(java.lang.Iterable iterable,
                                     java.lang.Appendable buffer)

joinWithQuote

protected java.lang.Appendable joinWithQuote(java.lang.Iterable tuple,
                                             java.lang.Appendable buffer)
                                      throws java.io.IOException

Throws:

java.io.IOException

joinNoQuote

protected java.lang.Appendable joinNoQuote(java.lang.Iterable tuple,
                                           java.lang.Appendable buffer)
                                    throws java.io.IOException

Throws:

java.io.IOException

inferTypes

protected java.lang.reflect.Type[] inferTypes(java.lang.Object[] result)

prepareFields

protected java.lang.Iterable prepareFields(java.lang.Iterable fields)

cleanFields

protected java.lang.Object[] cleanFields(java.lang.Object[] result)