org.apache.flink.fs.s3.common.writer

Class S3RecoverableWriter

java.lang.Object

org.apache.flink.fs.s3.common.writer.S3RecoverableWriter

All Implemented Interfaces:

org.apache.flink.core.fs.RecoverableWriter

@PublicEvolving
public class S3RecoverableWriter
extends Object
implements org.apache.flink.core.fs.RecoverableWriter

An implementation of the RecoverableWriter against S3.

This implementation makes heavy use of MultiPart Uploads in S3 to persist intermediate data as soon as possible.

This class partially reuses utility classes and implementations from the Hadoop project, specifically around configuring S3 requests and handling retries.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.core.fs.RecoverableWriter

org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable, org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable

Method Summary

Modifier and Type	Method and Description
boolean	cleanupRecoverableState(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable)
org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable>	getCommitRecoverableSerializer()
org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable>	getResumeRecoverableSerializer()
org.apache.flink.core.fs.RecoverableFsDataOutputStream	open(org.apache.flink.core.fs.Path path)
S3RecoverableFsDataOutputStream	recover(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable recoverable)
org.apache.flink.core.fs.RecoverableFsDataOutputStream.Committer	recoverForCommit(org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable recoverable)
boolean	requiresCleanupOfRecoverableState()
boolean	supportsResume()
static S3RecoverableWriter	writer(org.apache.hadoop.fs.FileSystem fs, org.apache.flink.util.function.FunctionWithException<File,RefCountedFile,IOException> tempFileCreator, S3AccessHelper s3AccessHelper, Executor uploadThreadPool, long userDefinedMinPartSize, int maxConcurrentUploadsPerStream)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

open

public org.apache.flink.core.fs.RecoverableFsDataOutputStream open(org.apache.flink.core.fs.Path path)
                                                            throws IOException

Specified by:

open in interface org.apache.flink.core.fs.RecoverableWriter

Throws:

IOException

recoverForCommit

public org.apache.flink.core.fs.RecoverableFsDataOutputStream.Committer recoverForCommit(org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable recoverable)
                                                                                  throws IOException

Specified by:

recoverForCommit in interface org.apache.flink.core.fs.RecoverableWriter

Throws:

IOException

recover

public S3RecoverableFsDataOutputStream recover(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable recoverable)
                                        throws IOException

Specified by:

recover in interface org.apache.flink.core.fs.RecoverableWriter

Throws:

IOException

requiresCleanupOfRecoverableState

public boolean requiresCleanupOfRecoverableState()

Specified by:

requiresCleanupOfRecoverableState in interface org.apache.flink.core.fs.RecoverableWriter

cleanupRecoverableState

public boolean cleanupRecoverableState(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable)
                                throws IOException

Specified by:

cleanupRecoverableState in interface org.apache.flink.core.fs.RecoverableWriter

Throws:

IOException

getCommitRecoverableSerializer

public org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable> getCommitRecoverableSerializer()

Specified by:

getCommitRecoverableSerializer in interface org.apache.flink.core.fs.RecoverableWriter

getResumeRecoverableSerializer

public org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable> getResumeRecoverableSerializer()

Specified by:

getResumeRecoverableSerializer in interface org.apache.flink.core.fs.RecoverableWriter

supportsResume

public boolean supportsResume()

Specified by:

supportsResume in interface org.apache.flink.core.fs.RecoverableWriter

writer

public static S3RecoverableWriter writer(org.apache.hadoop.fs.FileSystem fs,
                                         org.apache.flink.util.function.FunctionWithException<File,RefCountedFile,IOException> tempFileCreator,
                                         S3AccessHelper s3AccessHelper,
                                         Executor uploadThreadPool,
                                         long userDefinedMinPartSize,
                                         int maxConcurrentUploadsPerStream)