A B C D E F G H I M N O P R S T U W

A

AbstractS3FileSystemFactory - Class in org.apache.flink.fs.s3.common

Base class for file system factories that create S3 file systems.

AbstractS3FileSystemFactory(String, HadoopConfigLoader) - Constructor for class org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory

apply(File) - Method in class org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator

Gets the next temp file and stream to temp file.

B

BackPressuringExecutor - Class in org.apache.flink.fs.s3.common.utils

An executor decorator that allows only a certain number of concurrent executions.

BackPressuringExecutor(Executor, int) - Constructor for class org.apache.flink.fs.s3.common.utils.BackPressuringExecutor

BUFFER_SIZE - Static variable in class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

C

cleanupRecoverableState(RecoverableWriter.ResumeRecoverable) - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableWriter

close() - Method in class org.apache.flink.fs.s3.common.utils.OffsetAwareOutputStream

close() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

close() - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream

closeForCommit() - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream

closeStream() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedFile

commit() - Method in class org.apache.flink.fs.s3.common.writer.S3Committer

commitAfterRecovery() - Method in class org.apache.flink.fs.s3.common.writer.S3Committer

commitMultiPartUpload(String, String, List<PartETag>, long, AtomicInteger) - Method in interface org.apache.flink.fs.s3.common.writer.S3AccessHelper

Finalizes a Multi-Part Upload.

configure(Configuration) - Method in class org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory

create(URI) - Method in class org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory

createHadoopFileSystem() - Method in class org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory

createRecoverableWriter() - Method in class org.apache.flink.fs.s3.common.FlinkS3FileSystem

D

deleteObject(String) - Method in interface org.apache.flink.fs.s3.common.writer.S3AccessHelper

Deletes the object associated with the provided key.

E

ENTROPY_INJECT_KEY_OPTION - Static variable in class org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory

The substring to be replaced by random entropy in checkpoint paths.

ENTROPY_INJECT_LENGTH_OPTION - Static variable in class org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory

The number of entropy characters, in case entropy injection is configured.

execute(Runnable) - Method in class org.apache.flink.fs.s3.common.utils.BackPressuringExecutor

F

FlinkS3FileSystem - Class in org.apache.flink.fs.s3.common

Implementation of the Flink FileSystem interface for S3.

FlinkS3FileSystem(FileSystem, String, String, int, S3AccessHelper, long, int) - Constructor for class org.apache.flink.fs.s3.common.FlinkS3FileSystem

Creates a FlinkS3FileSystem based on the given Hadoop S3 file system.

flush() - Method in class org.apache.flink.fs.s3.common.utils.OffsetAwareOutputStream

flush() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

flush() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedFile

flush() - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream

G

generateEntropy() - Method in class org.apache.flink.fs.s3.common.FlinkS3FileSystem

getCommitRecoverableSerializer() - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableWriter

getEntropyInjectionKey() - Method in class org.apache.flink.fs.s3.common.FlinkS3FileSystem

getFile() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedFile

getInitURI(URI, Configuration) - Method in class org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory

getInputFile() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

getInputFile() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedFSOutputStream

Gets the underlying File that allows to read the contents of the file.

getKind() - Method in class org.apache.flink.fs.s3.common.FlinkS3FileSystem

getLength() - Method in class org.apache.flink.fs.s3.common.utils.OffsetAwareOutputStream

getLength() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedFile

getLocalTmpDir() - Method in class org.apache.flink.fs.s3.common.FlinkS3FileSystem

getObject(String, File) - Method in interface org.apache.flink.fs.s3.common.writer.S3AccessHelper

Gets the object associated with the provided key from S3 and puts it in the provided targetLocation.

getObjectMetadata(String) - Method in interface org.apache.flink.fs.s3.common.writer.S3AccessHelper

Fetches the metadata associated with a given key on S3.

getObjectName() - Method in class org.apache.flink.fs.s3.common.writer.S3Recoverable

getOrLoadHadoopConfig() - Method in class org.apache.flink.fs.s3.common.HadoopConfigLoader

get the loaded Hadoop config (or fall back to one loaded from the classpath).

getPos() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

getPos() - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream

getRecoverable() - Method in class org.apache.flink.fs.s3.common.writer.S3Committer

getReferenceCounter() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

getResumeRecoverableSerializer() - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableWriter

getS3AccessHelper(FileSystem) - Method in class org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory

getStream() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedFile

H

HadoopConfigLoader - Class in org.apache.flink.fs.s3.common

This class lazily loads hadoop configuration from resettable Flink's configuration.

HadoopConfigLoader(String[], String[][], String, Set<String>, Set<String>, String) - Constructor for class org.apache.flink.fs.s3.common.HadoopConfigLoader

I

incompleteObjectLength() - Method in class org.apache.flink.fs.s3.common.writer.S3Recoverable

incompleteObjectName() - Method in class org.apache.flink.fs.s3.common.writer.S3Recoverable

inDirectories(File...) - Static method in class org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator

isClosed() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

isClosed() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedFSOutputStream

Checks if the file is closed for writes.

M

MAX_CONCURRENT_UPLOADS - Static variable in class org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory

N

newFile(File, OutputStream) - Static method in class org.apache.flink.fs.s3.common.utils.RefCountedFile

newStream(RecoverableMultiPartUpload, FunctionWithException<File, RefCountedFile, IOException>, long) - Static method in class org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream

numBytesInParts() - Method in class org.apache.flink.fs.s3.common.writer.S3Recoverable

O

OffsetAwareOutputStream - Class in org.apache.flink.fs.s3.common.utils

An OutputStream that keeps track of its current length.

open(Path) - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableWriter

openNew(FunctionWithException<File, RefCountedFile, IOException>) - Static method in class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

org.apache.flink.fs.s3.common - package org.apache.flink.fs.s3.common

org.apache.flink.fs.s3.common.utils - package org.apache.flink.fs.s3.common.utils

org.apache.flink.fs.s3.common.writer - package org.apache.flink.fs.s3.common.writer

P

PART_UPLOAD_MIN_SIZE - Static variable in class org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory

parts() - Method in class org.apache.flink.fs.s3.common.writer.S3Recoverable

persist() - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream

putObject(String, File) - Method in interface org.apache.flink.fs.s3.common.writer.S3AccessHelper

Uploads an object to S3.

R

recover(RecoverableWriter.ResumeRecoverable) - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableWriter

recoverForCommit(RecoverableWriter.CommitRecoverable) - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableWriter

recoverStream(RecoverableMultiPartUpload, FunctionWithException<File, RefCountedFile, IOException>, long, long) - Static method in class org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream

RefCounted - Interface in org.apache.flink.fs.s3.common.utils

Interface to simply add reference counting functionality.

RefCountedBufferingFileStream - Class in org.apache.flink.fs.s3.common.utils

A RefCountedFile that also uses an in-memory buffer for buffering small writes.

RefCountedBufferingFileStream(RefCountedFile, int) - Constructor for class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

RefCountedFile - Class in org.apache.flink.fs.s3.common.utils

A reference counted file which is deleted as soon as no caller holds a reference to the wrapped File.

RefCountedFSOutputStream - Class in org.apache.flink.fs.s3.common.utils

A FSDataOutputStream with the RefCounted functionality.

RefCountedFSOutputStream() - Constructor for class org.apache.flink.fs.s3.common.utils.RefCountedFSOutputStream

RefCountedTmpFileCreator - Class in org.apache.flink.fs.s3.common.utils

A utility class that creates local reference counted files that serve as temporary files.

release() - Method in interface org.apache.flink.fs.s3.common.utils.RefCounted

Decreases the reference counter.

release() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

release() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedFile

requiresCleanupOfRecoverableState() - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableWriter

restore(FunctionWithException<File, RefCountedFile, IOException>, File) - Static method in class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

restoredFile(File, OutputStream, long) - Static method in class org.apache.flink.fs.s3.common.utils.RefCountedFile

retain() - Method in interface org.apache.flink.fs.s3.common.utils.RefCounted

Increases the reference counter.

retain() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

retain() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedFile

S

S3_MULTIPART_MIN_PART_SIZE - Static variable in class org.apache.flink.fs.s3.common.FlinkS3FileSystem

The minimum size of a part in the multipart upload, except for the last part: 5 MIBytes.

S3AccessHelper - Interface in org.apache.flink.fs.s3.common.writer

An interface that abstracts away the Multi-Part Upload (MPU) functionality offered by S3, from the specific implementation of the file system.

S3Committer - Class in org.apache.flink.fs.s3.common.writer

Data object to commit an S3 MultiPartUpload.

S3Recoverable - Class in org.apache.flink.fs.s3.common.writer

Data object to recover an S3 MultiPartUpload for a recoverable output stream.

S3RecoverableFsDataOutputStream - Class in org.apache.flink.fs.s3.common.writer

A RecoverableFsDataOutputStream to S3 that is based on a recoverable multipart upload.

S3RecoverableWriter - Class in org.apache.flink.fs.s3.common.writer

An implementation of the RecoverableWriter against S3.

setFlinkConfig(Configuration) - Method in class org.apache.flink.fs.s3.common.HadoopConfigLoader

startMultiPartUpload(String) - Method in interface org.apache.flink.fs.s3.common.writer.S3AccessHelper

Initializes a Multi-Part Upload.

supportsResume() - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableWriter

sync() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

sync() - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream

T

toString() - Method in class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

toString() - Method in class org.apache.flink.fs.s3.common.writer.S3Recoverable

U

uploadId() - Method in class org.apache.flink.fs.s3.common.writer.S3Recoverable

uploadPart(String, String, int, File, long) - Method in interface org.apache.flink.fs.s3.common.writer.S3AccessHelper

Uploads a part and associates it with the MPU with the provided uploadId.

W

write(byte[], int, int) - Method in class org.apache.flink.fs.s3.common.utils.OffsetAwareOutputStream

write(int) - Method in class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

write(byte[], int, int) - Method in class org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream

write(byte[], int, int) - Method in class org.apache.flink.fs.s3.common.utils.RefCountedFile

write(int) - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream

write(byte[], int, int) - Method in class org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream

writer(FileSystem, FunctionWithException<File, RefCountedFile, IOException>, S3AccessHelper, Executor, long, int) - Static method in class org.apache.flink.fs.s3.common.writer.S3RecoverableWriter

A B C D E F G H I M N O P R S T U W