org.apache.flink.yarn.highavailability

Class YarnPreConfiguredMasterNonHaServices

java.lang.Object

org.apache.flink.yarn.highavailability.YarnHighAvailabilityServices

org.apache.flink.yarn.highavailability.AbstractYarnNonHaServices

org.apache.flink.yarn.highavailability.YarnPreConfiguredMasterNonHaServices

All Implemented Interfaces:

AutoCloseable, org.apache.flink.runtime.highavailability.HighAvailabilityServices

public class YarnPreConfiguredMasterNonHaServices
extends AbstractYarnNonHaServices

These YarnHighAvailabilityServices are for use by the TaskManager in setups, where there is one ResourceManager that is statically configured in the Flink configuration.

Handled failure types

• User code & operator failures: Failed operators are recovered from checkpoints.
• Task Manager Failures: Failed Task Managers are restarted and their tasks are recovered from checkpoints.

Non-recoverable failure types

• Application Master failures: These failures cannot be recovered, because TaskManagers have no way to discover the new Application Master's address.

Internally, these services put their recovery data into YARN's working directory, except for checkpoints, which are in the configured checkpoint directory. That way, checkpoints can be resumed with a new job/application, even if the complete YARN application is killed and cleaned up.

A typical YARN setup that uses these HA services first starts the ResourceManager inside the ApplicationMaster and puts its RPC endpoint address into the configuration with which the TaskManagers are started. Because of this static addressing scheme, the setup cannot handle failures of the JobManager and ResourceManager, which are running as part of the Application Master.

See Also:

HighAvailabilityServices

Field Summary

Fields inherited from class org.apache.flink.yarn.highavailability.YarnHighAvailabilityServices

blobStoreService, FLINK_RECOVERY_DATA_DIR, flinkFileSystem, haDataDirectory, hadoopFileSystem, LOG, workingDirectory

Fields inherited from interface org.apache.flink.runtime.highavailability.HighAvailabilityServices

DEFAULT_JOB_ID, DEFAULT_LEADER_ID

Constructor Summary

Constructors

Constructor and Description
YarnPreConfiguredMasterNonHaServices(org.apache.flink.configuration.Configuration config, org.apache.hadoop.conf.Configuration hadoopConf, org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.AddressResolution addressResolution) Creates new YarnPreConfiguredMasterHaServices for the given Flink and YARN configuration.

Method Summary

Modifier and Type	Method and Description
org.apache.flink.runtime.leaderelection.LeaderElectionService	getDispatcherLeaderElectionService()
org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService	getDispatcherLeaderRetriever()
org.apache.flink.runtime.leaderelection.LeaderElectionService	getJobManagerLeaderElectionService(org.apache.flink.api.common.JobID jobID)
org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService	getJobManagerLeaderRetriever(org.apache.flink.api.common.JobID jobID)
org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService	getJobManagerLeaderRetriever(org.apache.flink.api.common.JobID jobID, String defaultJobManagerAddress)
org.apache.flink.runtime.leaderelection.LeaderElectionService	getResourceManagerLeaderElectionService()
org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService	getResourceManagerLeaderRetriever()
org.apache.flink.runtime.leaderelection.LeaderElectionService	getWebMonitorLeaderElectionService()
org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService	getWebMonitorLeaderRetriever()

Methods inherited from class org.apache.flink.yarn.highavailability.AbstractYarnNonHaServices

getCheckpointRecoveryFactory, getRunningJobsRegistry, getSubmittedJobGraphStore

Methods inherited from class org.apache.flink.yarn.highavailability.YarnHighAvailabilityServices

close, closeAndCleanupAllData, createBlobStore, forSingleJobAppMaster, forYarnTaskManager, isClosed

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

YarnPreConfiguredMasterNonHaServices

public YarnPreConfiguredMasterNonHaServices(org.apache.flink.configuration.Configuration config,
                                            org.apache.hadoop.conf.Configuration hadoopConf,
                                            org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.AddressResolution addressResolution)
                                     throws IOException

Creates new YarnPreConfiguredMasterHaServices for the given Flink and YARN configuration. This constructor parses the ResourceManager address from the Flink configuration and sets up the HDFS access to store recovery data in the YARN application's working directory.

Parameters:

config - The Flink configuration of this component / process.

hadoopConf - The Hadoop configuration for the YARN cluster.

Throws:

IOException - Thrown, if the initialization of the Hadoop file system used by YARN fails.

org.apache.flink.configuration.IllegalConfigurationException - Thrown, if the Flink configuration does not properly describe the ResourceManager address and port.

Method Detail

getResourceManagerLeaderRetriever

public org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService getResourceManagerLeaderRetriever()

getDispatcherLeaderRetriever

public org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService getDispatcherLeaderRetriever()

getResourceManagerLeaderElectionService

public org.apache.flink.runtime.leaderelection.LeaderElectionService getResourceManagerLeaderElectionService()

getDispatcherLeaderElectionService

public org.apache.flink.runtime.leaderelection.LeaderElectionService getDispatcherLeaderElectionService()

getJobManagerLeaderElectionService

public org.apache.flink.runtime.leaderelection.LeaderElectionService getJobManagerLeaderElectionService(org.apache.flink.api.common.JobID jobID)

getWebMonitorLeaderElectionService

public org.apache.flink.runtime.leaderelection.LeaderElectionService getWebMonitorLeaderElectionService()

getJobManagerLeaderRetriever

public org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService getJobManagerLeaderRetriever(org.apache.flink.api.common.JobID jobID)

getJobManagerLeaderRetriever

public org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService getJobManagerLeaderRetriever(org.apache.flink.api.common.JobID jobID,
                                                                                                    String defaultJobManagerAddress)

getWebMonitorLeaderRetriever

public org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService getWebMonitorLeaderRetriever()