public abstract class YarnHighAvailabilityServices extends Object implements org.apache.flink.runtime.highavailability.HighAvailabilityServices
HighAvailabilityServices for YARN setups.
These high-availability services auto-configure YARN's HDFS and the YARN application's
working directory to be used to store job recovery data.
Note for implementers: This class locks access to and creation of services,
to make sure all services are properly shut down when shutting down this class.
To participate in the checks, overriding methods should frame method body with
calls to enter() and exit() as shown in the following pattern:
public LeaderRetrievalService getResourceManagerLeaderRetriever() {
enter();
try {
CuratorClient client = getCuratorClient();
return ZooKeeperUtils.createLeaderRetrievalService(client, configuration, RESOURCE_MANAGER_LEADER_PATH);
} finally {
exit();
}
}
| Modifier and Type | Field and Description |
|---|---|
protected org.apache.flink.runtime.blob.BlobStoreService |
blobStoreService
Blob store service to be used for the BlobServer and BlobCache.
|
static String |
FLINK_RECOVERY_DATA_DIR
The name of the sub directory in which Flink stores the recovery data.
|
protected org.apache.flink.core.fs.FileSystem |
flinkFileSystem
The Flink FileSystem object that represent the HDFS used by YARN.
|
protected org.apache.flink.core.fs.Path |
haDataDirectory
The directory for HA persistent data.
|
protected org.apache.hadoop.fs.FileSystem |
hadoopFileSystem
The Hadoop FileSystem object that represent the HDFS used by YARN.
|
protected static org.slf4j.Logger |
LOG
Logger for these services, shared with subclasses.
|
protected org.apache.flink.core.fs.Path |
workingDirectory
The working directory of this YARN application.
|
| Modifier | Constructor and Description |
|---|---|
protected |
YarnHighAvailabilityServices(org.apache.flink.configuration.Configuration config,
org.apache.hadoop.conf.Configuration hadoopConf)
Creates new YARN high-availability services, configuring the file system and recovery
data directory based on the working directory in the given Hadoop configuration.
|
| Modifier and Type | Method and Description |
|---|---|
void |
close() |
void |
closeAndCleanupAllData() |
org.apache.flink.runtime.blob.BlobStore |
createBlobStore() |
static YarnHighAvailabilityServices |
forSingleJobAppMaster(org.apache.flink.configuration.Configuration flinkConfig,
org.apache.hadoop.conf.Configuration hadoopConfig)
Creates the high-availability services for a single-job Flink YARN application, to be
used in the Application Master that runs both ResourceManager and JobManager.
|
static YarnHighAvailabilityServices |
forYarnTaskManager(org.apache.flink.configuration.Configuration flinkConfig,
org.apache.hadoop.conf.Configuration hadoopConfig)
Creates the high-availability services for the TaskManagers participating in
a Flink YARN application.
|
boolean |
isClosed()
Checks whether these services have been shut down.
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitgetCheckpointRecoveryFactory, getDispatcherLeaderElectionService, getDispatcherLeaderRetriever, getJobManagerLeaderElectionService, getJobManagerLeaderRetriever, getJobManagerLeaderRetriever, getResourceManagerLeaderElectionService, getResourceManagerLeaderRetriever, getRunningJobsRegistry, getSubmittedJobGraphStore, getWebMonitorLeaderElectionService, getWebMonitorLeaderRetrieverpublic static final String FLINK_RECOVERY_DATA_DIR
protected static final org.slf4j.Logger LOG
protected final org.apache.flink.core.fs.FileSystem flinkFileSystem
protected final org.apache.hadoop.fs.FileSystem hadoopFileSystem
protected final org.apache.flink.core.fs.Path workingDirectory
protected final org.apache.flink.core.fs.Path haDataDirectory
protected final org.apache.flink.runtime.blob.BlobStoreService blobStoreService
protected YarnHighAvailabilityServices(org.apache.flink.configuration.Configuration config,
org.apache.hadoop.conf.Configuration hadoopConf)
throws IOException
This class requires that the default Hadoop file system configured in the given Hadoop configuration is an HDFS.
config - The Flink configuration of this component / process.hadoopConf - The Hadoop configuration for the YARN cluster.IOException - Thrown, if the initialization of the Hadoop file system used by YARN fails.public org.apache.flink.runtime.blob.BlobStore createBlobStore()
throws IOException
createBlobStore in interface org.apache.flink.runtime.highavailability.HighAvailabilityServicesIOExceptionpublic boolean isClosed()
public void close()
throws Exception
close in interface AutoCloseableclose in interface org.apache.flink.runtime.highavailability.HighAvailabilityServicesExceptionpublic void closeAndCleanupAllData()
throws Exception
closeAndCleanupAllData in interface org.apache.flink.runtime.highavailability.HighAvailabilityServicesExceptionpublic static YarnHighAvailabilityServices forSingleJobAppMaster(org.apache.flink.configuration.Configuration flinkConfig, org.apache.hadoop.conf.Configuration hadoopConfig) throws IOException
flinkConfig - The Flink configuration.hadoopConfig - The Hadoop configuration for the YARN cluster.IOException - Thrown, if the high-availability services could not be initialized.public static YarnHighAvailabilityServices forYarnTaskManager(org.apache.flink.configuration.Configuration flinkConfig, org.apache.hadoop.conf.Configuration hadoopConfig) throws IOException
flinkConfig - The Flink configuration.hadoopConfig - The Hadoop configuration for the YARN cluster.IOException - Thrown, if the high-availability services could not be initialized.Copyright © 2014–2019 The Apache Software Foundation. All rights reserved.