org.apache.flink.fs.s3.common

Class FlinkS3FileSystem

java.lang.Object

org.apache.flink.core.fs.FileSystem

org.apache.flink.runtime.fs.hdfs.HadoopFileSystem

org.apache.flink.fs.s3.common.FlinkS3FileSystem

All Implemented Interfaces:

org.apache.flink.core.fs.EntropyInjectingFileSystem

public class FlinkS3FileSystem
extends org.apache.flink.runtime.fs.hdfs.HadoopFileSystem
implements org.apache.flink.core.fs.EntropyInjectingFileSystem

Implementation of the Flink FileSystem interface for S3. This class implements the common behavior implemented directly by Flink and delegates common calls to an implementation of Hadoop's filesystem abstraction.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.core.fs.FileSystem

org.apache.flink.core.fs.FileSystem.WriteMode

Field Summary

Fields

Modifier and Type	Field and Description
static long	S3_MULTIPART_MIN_PART_SIZE The minimum size of a part in the multipart upload, except for the last part: 5 MIBytes.

Constructor Summary

Constructors

Constructor and Description
FlinkS3FileSystem(org.apache.hadoop.fs.FileSystem hadoopS3FileSystem, String localTmpDirectory, String entropyInjectionKey, int entropyLength, S3AccessHelper s3UploadHelper, long s3uploadPartSize, int maxConcurrentUploadsPerStream) Creates a FlinkS3FileSystem based on the given Hadoop S3 file system.

Method Summary

Modifier and Type	Method and Description
org.apache.flink.core.fs.RecoverableWriter	createRecoverableWriter()
String	generateEntropy()
String	getEntropyInjectionKey()
org.apache.flink.core.fs.FileSystemKind	getKind()
String	getLocalTmpDir()

Methods inherited from class org.apache.flink.runtime.fs.hdfs.HadoopFileSystem

create, create, delete, exists, getDefaultBlockSize, getFileBlockLocations, getFileStatus, getHadoopFileSystem, getHomeDirectory, getUri, getWorkingDirectory, isDistributedFS, listStatus, mkdirs, open, open, rename, toHadoopPath

Methods inherited from class org.apache.flink.core.fs.FileSystem

create, get, getDefaultFsUri, getLocalFileSystem, getUnguardedFileSystem, initialize, initOutPathDistFS, initOutPathLocalFS

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

S3_MULTIPART_MIN_PART_SIZE

public static final long S3_MULTIPART_MIN_PART_SIZE

The minimum size of a part in the multipart upload, except for the last part: 5 MIBytes.

See Also:

Constant Field Values

Constructor Detail

FlinkS3FileSystem

public FlinkS3FileSystem(org.apache.hadoop.fs.FileSystem hadoopS3FileSystem,
                         String localTmpDirectory,
                         @Nullable
                         String entropyInjectionKey,
                         int entropyLength,
                         @Nullable
                         S3AccessHelper s3UploadHelper,
                         long s3uploadPartSize,
                         int maxConcurrentUploadsPerStream)

Creates a FlinkS3FileSystem based on the given Hadoop S3 file system. The given Hadoop file system object is expected to be initialized already.

This constructor additionally configures the entropy injection for the file system.

Parameters:

hadoopS3FileSystem - The Hadoop FileSystem that will be used under the hood.

entropyInjectionKey - The substring that will be replaced by entropy or removed.

entropyLength - The number of random alphanumeric characters to inject as entropy.

Method Detail

getEntropyInjectionKey

@Nullable
public String getEntropyInjectionKey()

Specified by:

getEntropyInjectionKey in interface org.apache.flink.core.fs.EntropyInjectingFileSystem

generateEntropy

public String generateEntropy()

Specified by:

generateEntropy in interface org.apache.flink.core.fs.EntropyInjectingFileSystem

getKind

public org.apache.flink.core.fs.FileSystemKind getKind()

Overrides:

getKind in class org.apache.flink.runtime.fs.hdfs.HadoopFileSystem

getLocalTmpDir

public String getLocalTmpDir()

createRecoverableWriter

public org.apache.flink.core.fs.RecoverableWriter createRecoverableWriter()
                                                                   throws IOException

Overrides:

createRecoverableWriter in class org.apache.flink.runtime.fs.hdfs.HadoopFileSystem

Throws:

IOException