> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hyperlane.xyz/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Validators

<Note>
  For an introduction on Validators in Hyperlane, you can check out the
  [Overview section](/docs/operate/overview-agents).
</Note>

As you follow this guide, you can run a Hyperlane validator on any of the existing [chains](https://github.com/hyperlane-xyz/hyperlane-registry/tree/main/chains) the protocol is live on. Hyperlane Validators are run on a per-origin-chain basis, and these instructions are written for a single chain.

## Requirements

* **Secure Signing Key**

  * Validators use this key to sign the `Mailbox`'s latest merkle root. Securing this key is important. If it is compromised, attackers can attempt to falsify messages, causing the Validator to be slashed.

  * The Hyperlane Validator agent currently supports signing with AWS KMS keys that are accessed via API keys/secrets as well as hexadecimal plaintext keys. See more under [agent keys](/docs/operate/set-up-agent-keys).

* **Publicly Readable Storage**

  * Validators write their signatures off-chain to publicly accessible, highly available, storage, so that they can be aggregated by the [Relayer](/docs/protocol/agents/relayer).

  * The Hyperlane Validator agent currently supports storing signatures on AWS S3 using the same AWS API key above, as well as storing signatures in the local filesystem for testing.

  * The design is [open source](https://github.com/hyperlane-xyz/hyperlane-monorepo/tree/56be527d691a11a1ff2de4a390fd0dae8bb77347/typescript/infra/src/agents) and generalizable to other storage and key solutions. There is a community-submitted work-in-progress [PR for GCS](https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3156).

* **Machine Requirements**

  * Validators can compile the Rust binary themselves or run a Docker image provided by Abacus Works. The binary can be run using your favorite cloud service. You can even run multiple instances of them in different regions for high availability, as Hyperlane has no notion of "double signing."

  * Hardware requirements & costs are minimal - validators often start with a 2-Core / 2GB RAM / 4GB Storage setup with typical costs around \$75/mo

* **RPC Node**

  * Validators make simple view calls to read merkle roots from the [Mailbox](/docs/protocol/core/mailbox) contract on the chain they are validating for. You must use your own RPC urls not any public ones.

  * Validators should configure multiple RPC URLs for redundancy and reliability. Different chains have different configurations—check the Configuration section for details.

<Warning>
  Operating a Validator for Polygon mainnet requires access to an archive node.
  This is because Validators should only sign roots once they've been finalized,
  and Polygon requires 256 block confirmations to achieve finality.
</Warning>

## Guide

### Pre-Requisites

As a recap - before running a production Validator you need to have:

1. Created a key for your Validator to sign with, see the [Agent Keys](/docs/operate/set-up-agent-keys) documentation.
2. Set up the destination for your Validator signatures to be posted, see the [AWS Signatures Bucket Setup](/docs/operate/validators/validator-signatures-aws) guide.

### Configuration

<Tip>
  Experienced operators may prefer to [deploy agents with
  Terraform](/docs/operate/guides/deploy-with-terraform) or a
  community-submitted [Ansible playbook](https://github.com/polkachu/hyperlane)
  instead. This process will automatically create agent keys, Validator buckets,
  permissions and any other auxiliary setup required to run a Validator cluster
  on AWS.
</Tip>

#### RPC Configuration

Hyperlane Validators and Relayers can use **multiple RPC URLs** to improve reliability and redundancy. The setup varies based on the chain type.

<Tabs>
  <Tab title="EVM Chains">
    EVM-based chains support configuring multiple RPC endpoints for redundancy and can specify how they are used.

    * **Configure multiple RPCs:** Use [`customRpcUrls`](/docs/operate/config/config-reference#chains-\<chain-name>-customrpcurls).
    * **RPC Selection Mode ([`rpcConsensusType`](/docs/operate/config/config-reference#chains-\<chain-name>-rpcconsensustype))**:
      * **Fallback**: The agent tries the first URL and switches to the next if needed.
      * **Quorum**: Requires a majority of the URLs to agree with the exception of submitting transactions; it will automatically coordinate the "latest" block if not specified to reduce sync errors.
  </Tab>

  <Tab title="Cosmos Chains">
    Cosmos-based chains require both **RPC and gRPC endpoints** for proper operation.

    * **Configure multiple RPCs:** Use [`customRpcUrls`](/docs/operate/config/config-reference#chains-\<chain-name>-customrpcurls).
    * **Configure gRPC endpoints:** Use [`customGrpcUrls`](/docs/operate/config/config-reference#chains-\<chain-name>-customrpcurls).
    * **Fallback Mechanism:** Cosmos agents always use **fallback mode**.
  </Tab>

  <Tab title="SVM Chains">
    SVM-based chains support configuring multiple RPC endpoints. Agents automatically use **fallback mode**: if the current RPC fails or times out, they switch to the next available endpoint.

    * **Configure RPCs:** Use [`customRpcUrls`](/docs/operate/config/config-reference#chains-\<chain-name>-customrpcurls).

    <Warning>
      For **SVM chains**, only **Fallback mode** is supported. SVM agents will automatically switch between configured RPC URLs when encountering errors or timeouts.
    </Warning>

    <Warning>
      **SVM Relayer Funding**: When relaying messages to SVM chains, make sure your Relayer key has sufficient funds on the destination chain. The Relayer performs view calls via simulations that require the sender to have funds for successful simulation, even for read operations. Without sufficient funds, you may encounter `No return data from InboxGetRecipientIsm instruction` errors.
    </Warning>
  </Tab>
</Tabs>

#### Configuration Settings

Like the local setup, there are a few base arguments you should provide when configuring your Validator.

| Parameter                                       | Description                                                                                             |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `--db`                                          | Path for writing persistent data to disk.                                                               |
| `--originChainName`                             | Name of the chain being validated. For example: `ethereum`.                                             |
| `--chains.[originChainName].customRpcUrls`      | Override the default RPC URLs used by the Validator for your origin chain.                              |
| `--chains.[originChainName].blocks.reorgPeriod` | Number of block confirmations the Validator needs to wait for before signing the `Mailbox` merkle root. |

<Info>
  Your Validator takes both command line arguments and environment variables as
  configuration. Take a look at the [agent
  configuration](/docs/operate/config/agent-config) page and the [configuration
  reference](/docs/operate/config/config-reference) for a full list of
  configuration possibilities.
</Info>

You can also provide the path to additional configuration files as a comma separated list with the [`CONFIG_FILES` environment variable](/docs/operate/config/config-reference#config_files). If you choose to run in Docker, see the docker section of [agent configuration](/docs/operate/config/agent-config) for tips on mounting your config files into your Docker container.

### Setup-specific configuration

These configurations requirements differ depending on which environment you are setting up.

#### Checkpoint signer configuration

<Tabs>
  <Tab title="Hex Key">
    | Argument                                | Description                                                                                                                                  |
    | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
    | `--validator.key`                       | Your Validator's private key, which is used to sign merkle roots.                                                                            |
    | `--chains.${localChainName}.signer.key` | Your Validator's private key, which will be used to submit a transaction onchain that publicly announces your Validator's checkpoint syncer. |
  </Tab>

  <Tab title="AWS KMS">
    | Argument             | Description                                                                                                                            |
    | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
    | `--validator.region` | The region of your AWS KMS key. For example: `us-east-1`.                                                                              |
    | `--validator.type`   | Set to the `aws` literal.                                                                                                              |
    | `--validator.id`     | The alias of your Validator's AWS KMS key, prefixed with `alias/`. For example: `alias/hyperlane-validator-signer-${originChainName}`. |
  </Tab>
</Tabs>

#### Transaction signer configuration

The key configured in this step needs a small amount of funds to send the initial announcement transaction.

<Tabs>
  <Tab title="Hex Key (EVM)">
    This is the same as the local configuration for the checkpoint signer.

    | Argument                                | Description                                                                                                                                  |
    | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
    | `--chains.${localChainName}.signer.key` | Your Validator's private key, which will be used to submit a transaction onchain that publicly announces your Validator's checkpoint syncer. |
  </Tab>

  <Tab title="AWS KMS (EVM)">
    | Argument                                    | Description                                                                                                                            |
    | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
    | `--chains.${originChainName}.signer.type`   | Set to the `aws` literal.                                                                                                              |
    | `--chains.${originChainName}.signer.region` | The region of your AWS KMS key. For example: `us-east-1`.                                                                              |
    | `--chains.${originChainName}.signer.id`     | The alias of your Validator's AWS KMS key, prefixed with `alias/`. For example: `alias/hyperlane-validator-signer-${originChainName}`. |
  </Tab>

  <Tab title="Hex Key (Cosmos)">
    | Argument                                    | Description                                                                                                                                      |
    | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
    | `--chains.${originChainName}.signer.type`   | Set to the `cosmosKey` literal.                                                                                                                  |
    | `--chains.${originChainName}.signer.prefix` | Set to the prefix literal associated with the chain's address format. Example: `osmo`.                                                           |
    | `--chains.${originChainName}.key`           | Your Validator's hex private key, which will be used to submit a transaction onchain that publicly announces your Validator's checkpoint syncer. |
  </Tab>
</Tabs>

#### Checkpoint syncer configuration

<Tabs>
  <Tab title="Local Setup">
    | Argument                  | Description                                                                                                                                                                                                                                                |
    | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | `--checkpointSyncer.type` | Set to `localStorage`.                                                                                                                                                                                                                                     |
    | `--checkpointSyncer.path` | The path to your local directory where Validator signatures will be written. This should be the value of `$MY_VALIDATOR_SIGNATURES_DIRECTORY` from the local setup. For example: `--checkpointSyncer.path='/tmp/hyperlane-validator-signatures-ethereum'`. |

    <Warning>
      Note that the Relayer **must** be configured with
      `--allowLocalCheckpointSyncers` to be able to read signatures from this
      Validator.
    </Warning>
  </Tab>

  <Tab title="Production Setup (AWS)">
    | Argument                    | Description                                                           |
    | --------------------------- | --------------------------------------------------------------------- |
    | `--checkpointSyncer.type`   | Set to `s3`.                                                          |
    | `--checkpointSyncer.bucket` | The AWS S3 bucket name.                                               |
    | `--checkpointSyncer.region` | The region of your AWS S3 bucket. For example: `us-east-1`.           |
    | `--checkpointSyncer.folder` | A folder name to use for this validator, can be the name of the chain |
  </Tab>
</Tabs>

## Start Validating

### Setup

The recommended installation method for a production environment is using a Docker image.

<Tabs>
  <Tab title="Docker image">
    First download the docker image:

    ```bash theme={null}
    docker pull --platform linux/amd64 ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0
    ```
  </Tab>

  <Tab title="Building from source">
    **Clone and setup**

    First, clone the Hyperlane monorepo:

    ```sh theme={null}
    git clone git@github.com:hyperlane-xyz/hyperlane-monorepo.git
    ```

    Then follow the [setup instructions](https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/README.md) in the `rust` directory. This should setup `rustup` as well as Rosetta 2 if you are on Apple Silicon.

    ```sh theme={null}
    # install rustup
    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

    # (apple silicon only) install rosetta 2
    softwareupdate --install-rosetta --agree-to-license
    ```

    Build the Validator:

    ```sh theme={null}
    cargo build --release bin validator
    ```
  </Tab>
</Tabs>

### Running the binary

For production Validators that write their signatures to an S3 bucket and have their keys configured with AWS KMS, you will have to provide the AWS access key and secret as environment variables.

| Environment variable    | Description                                             |
| ----------------------- | ------------------------------------------------------- |
| `AWS_ACCESS_KEY_ID`     | The access key ID of your Validator's AWS IAM user.     |
| `AWS_SECRET_ACCESS_KEY` | The secret access key of your Validator's AWS IAM user. |

For a refresher, check out the [Agent Keys](/docs/operate/set-up-agent-keys) guide.

<Tabs>
  <Tab title="Using Docker">
    Then start the container with the relevant arguments. For example, your configuration for AWS:

    ```sh theme={null}
    docker run \
      -it \
      -e AWS_ACCESS_KEY_ID=ABCDEFGHIJKLMNOP \
      -e AWS_SECRET_ACCESS_KEY=xX-haha-nice-try-Xx \
      --mount ... \
      ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0 \
      ./validator \
      --db /hyperlane_db \
      --originChainName <your_chain_name> \
      --reorgPeriod 1 \
      --validator.region us-east-1 \
      --checkpointSyncer.region us-east-1 \
      --validator.type aws \
      --chains.<your_chain_name>.signer.type aws \
      --validator.id alias/hyperlane-validator-signer-<your_chain_name> \
      --chains.<your_chain_name>.signer.id alias/hyperlane-validator-signer-<your_chain_name> \
      --checkpointSyncer.type s3 \
      --checkpointSyncer.bucket hyperlane-validator-signatures-<your_name> \
      --checkpointSyncer.folder <your_chain_name> \
    ```
  </Tab>

  <Tab title="Building from source">
    See these instructions for building from source without Docker.

    We can run the built binary from within the `hyperlane-monorepo/rust` directory:

    ```sh theme={null}
    # set AWS environment variables
    export AWS_ACCESS_KEY_ID=ABCDEFGHIJKLMNOP
    export AWS_SECRET_ACCESS_KEY=xX-haha-nice-try-Xx

    # run the Validator
    ./target/release/validator \
      --db /hyperlane_db \
      --originChainName <your_chain_name> \
      --reorgPeriod 1 \
      --validator.region us-east-1 \
      --checkpointSyncer.region us-east-1 \
      --validator.type aws \
      --chains.<your_chain_name>.signer.type aws \
      --chains.<your_chain_name>.signer.region<region_name> \
      --validator.id alias/hyperlane-validator-signer-<your_chain_name> \
      --chains.<your_chain_name>.signer.id alias/hyperlane-validator-signer-<your_chain_name> \
      --checkpointSyncer.type s3 \
      --checkpointSyncer.bucket hyperlane-validator-signatures-<your_chain_name>
    ```
  </Tab>
</Tabs>

### Announcing your Validator

The [Relayer](/docs/operate/relayer/run-relayer) needs to know where to find your Validator's signatures. Your Validator will automatically attempt to announce itself by writing to the `ValidatorAnnounce` contract on the chain that you're validating.

To do this, your Validator must have a small amount of tokens to pay for the gas for this transaction.

If your Validator has not yet announced itself, and does not have enough tokens to pay for gas, it will log a message specifying how many tokens are needed.

### Success!

The Validator will index the origin Mailbox contract for messages. If a message has been sent, you should see log messages that the Validator has signed them. If everything is configured correctly, you should see json files being written to your S3 bucket (if you followed the AWS setup) or to your local signatures directory if you followed the local setup. New json files get written every time a new outbound message is inserted into the mailbox.

## Running Multiple Validators

We encourage folks to validate on as many chains as they are interested in supporting. This section provides an overview of the considerations and instructions to set up multiple validators.

Running multiple validators improves security and redundancy. By deploying validators across different chains or regions, you reduce the risk of a single point of failure. If one validator encounters downtime or issues, others can continue to operate, ensuring the integrity and availability of the validation process.

### Running Multiple Validators: Key Considerations

* **Reusing Keys**: The same key can be used by multiple validators.
* **Shared AWS Account**: The same AWS account can be used by multiple validators.
* **Using a Shared S3 Bucket**: The same checkpoint syncer S3 bucket can be used by multiple validators, however each **must** use a different folder (e.g., `--checkpointSyncer.folder validator-1`, `--checkpointSyncer.folder validator-2`).

### Additional Steps for Multiple Validators

#### 1. Create an Agent Configuration

* Generate an agent [configuration](#configuration) file for each validator, specifying the chains they will validate and the corresponding RPC URLs.
* Ensure that each configuration file points to the correct signing key and S3 folder.

#### 2. Run Each Validator

* Ensure each Validator is assigned a unique database path (e.g., `--db /hyperlane_db_validator1`).
* Configure each validator to expose metrics on a unique port (e.g., `--metrics-port 9090`, `--metrics-port 9091`).
* If running multiple validators on the same machine, allocate sufficient system resources (CPU, memory, and storage).
* Ensure logs are directed to separate files or services for easier debugging and monitoring.
* If using Docker, assign unique container names and network configurations to avoid conflicts.

By following these additional steps, you can efficiently manage multiple Hyperlane Validators.

## Design Reference

```mermaid theme={null}
flowchart TB
    V(("Validator(s)"))
    Relayer((Relayer))

    subgraph Origin
      Sender
      M_O[(Mailbox)]
      POS[Proof of Stake]

      Sender -- "dispatch(destination, recipient, body)" --> M_O
    end

    subgraph Cloud
      aws[(Metadata<br>Database)]
    end

    M_O -. "indexing" .-> Relayer
    M_O -. "indexing" .-> V
    POS == "staking" === V

    V -- "signatures" --> aws

    subgraph Destination
      Recipient
      M_D[(Mailbox)]
      ISM[MultisigISM]

      M_D -- "verify(metadata, [origin, sender, body])" --> ISM
      M_D -- "handle(origin, sender, body)" --> Recipient
      ISM -. "interchainSecurityModule()" .- Recipient
    end

    Relayer -- "process()" --> M_D

    aws -. "metadata" .-> Relayer
    aws -. "moduleType()" .- ISM

    POS -. "validators()" .- ISM

    click ISM https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/solidity/contracts/interfaces/isms/IMultisigIsm.sol

    style Sender fill:#efab17
    style Recipient fill:#efab17
```