org.apache.flink.fs.s3.common.writer

Class S3RecoverableFsDataOutputStream

java.lang.Object

java.io.OutputStream

org.apache.flink.core.fs.FSDataOutputStream

org.apache.flink.core.fs.RecoverableFsDataOutputStream

org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream

All Implemented Interfaces:

Closeable, Flushable, AutoCloseable

@PublicEvolving
 @NotThreadSafe
public final class S3RecoverableFsDataOutputStream
extends org.apache.flink.core.fs.RecoverableFsDataOutputStream

A RecoverableFsDataOutputStream to S3 that is based on a recoverable multipart upload.

This class is NOT thread-safe. Concurrent writes tho this stream result in corrupt or lost data.

The close() method may be called concurrently when cancelling / shutting down. It will still ensure that local transient resources (like streams and temp files) are cleaned up, but will not touch data previously persisted in S3.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.core.fs.RecoverableFsDataOutputStream

org.apache.flink.core.fs.RecoverableFsDataOutputStream.Committer

Method Summary

Modifier and Type	Method and Description
void	close()
org.apache.flink.core.fs.RecoverableFsDataOutputStream.Committer	closeForCommit()
void	flush()
long	getPos()
static S3RecoverableFsDataOutputStream	newStream(org.apache.flink.fs.s3.common.writer.RecoverableMultiPartUpload upload, org.apache.flink.util.function.FunctionWithException<File,RefCountedFile,IOException> tmpFileCreator, long userDefinedMinPartSize)
org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable	persist()
static S3RecoverableFsDataOutputStream	recoverStream(org.apache.flink.fs.s3.common.writer.RecoverableMultiPartUpload upload, org.apache.flink.util.function.FunctionWithException<File,RefCountedFile,IOException> tmpFileCreator, long userDefinedMinPartSize, long bytesBeforeCurrentPart)
void	sync()
void	write(byte[] b, int off, int len)
void	write(int b)

Methods inherited from class java.io.OutputStream

write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

write

public void write(int b)
           throws IOException

Specified by:

write in class OutputStream

Throws:

IOException

write

public void write(byte[] b,
                  int off,
                  int len)
           throws IOException

Overrides:

write in class OutputStream

Throws:

IOException

flush

public void flush()
           throws IOException

Specified by:

flush in interface Flushable

Specified by:

flush in class org.apache.flink.core.fs.FSDataOutputStream

Throws:

IOException

getPos

public long getPos()
            throws IOException

Specified by:

getPos in class org.apache.flink.core.fs.FSDataOutputStream

Throws:

IOException

sync

public void sync()
          throws IOException

Specified by:

sync in class org.apache.flink.core.fs.FSDataOutputStream

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in class org.apache.flink.core.fs.RecoverableFsDataOutputStream

Throws:

IOException

persist

public org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable persist()
                                                                     throws IOException

Specified by:

persist in class org.apache.flink.core.fs.RecoverableFsDataOutputStream

Throws:

IOException

closeForCommit

public org.apache.flink.core.fs.RecoverableFsDataOutputStream.Committer closeForCommit()
                                                                                throws IOException

Specified by:

closeForCommit in class org.apache.flink.core.fs.RecoverableFsDataOutputStream

Throws:

IOException

newStream

public static S3RecoverableFsDataOutputStream newStream(org.apache.flink.fs.s3.common.writer.RecoverableMultiPartUpload upload,
                                                        org.apache.flink.util.function.FunctionWithException<File,RefCountedFile,IOException> tmpFileCreator,
                                                        long userDefinedMinPartSize)
                                                 throws IOException

Throws:

IOException

recoverStream

public static S3RecoverableFsDataOutputStream recoverStream(org.apache.flink.fs.s3.common.writer.RecoverableMultiPartUpload upload,
                                                            org.apache.flink.util.function.FunctionWithException<File,RefCountedFile,IOException> tmpFileCreator,
                                                            long userDefinedMinPartSize,
                                                            long bytesBeforeCurrentPart)
                                                     throws IOException

Throws:

IOException