DVN Troubleshooting

This guide helps DVN operators diagnose and resolve common issues with DVN contracts, Gasolina services, and infrastructure deployments.

Contract Error Reference

DVN Contract Errors

These errors are emitted by the DVN smart contract when operations fail.

Error	Signature	Cause	Solution
`DVN_OnlySelf`	`DVN_OnlySelf()`	Calling a restricted function directly instead of through signed execute	Use the `execute` function with proper signatures to call `setSigner`, `setQuorum`, or role management functions
`DVN_InvalidRole`	`DVN_InvalidRole(bytes32 role)`	Attempting to grant/revoke an invalid role	Use only valid roles: `ADMIN_ROLE`, `MESSAGE_LIB_ROLE`, `ALLOWLIST`, or `DENYLIST`
`DVN_InstructionExpired`	`DVN_InstructionExpired()`	The signed instruction’s expiration timestamp has passed	Generate new signatures with a future expiration timestamp
`DVN_InvalidTarget`	`DVN_InvalidTarget(address target)`	Target address doesn’t match the DVN contract	Verify the target address in `ExecuteParam` matches the DVN contract address
`DVN_InvalidVid`	`DVN_InvalidVid(uint32 vid)`	VID in the instruction doesn’t match this DVN instance	Ensure the VID matches the DVN’s configured VID (check `vid()` on contract)
`DVN_InvalidSignatures`	`DVN_InvalidSignatures()`	Signature verification failed	Verify signers match those registered in the contract and signatures are correctly formatted
`DVN_DuplicatedHash`	`DVN_DuplicatedHash(bytes32 hash)`	Attempting to replay an already-executed instruction	Generate a new instruction with different parameters or expiration

DVNFeeLib Errors

These errors relate to fee calculation and configuration.

Error	Signature	Cause	Solution
`DVN_EidNotSupported`	`DVN_EidNotSupported(uint32 eid)`	Destination endpoint ID is not configured	Add the destination to `DstConfig` using `setDstConfig`
`DVN_INVALID_INPUT_LENGTH`	`DVN_INVALID_INPUT_LENGTH()`	Array lengths don’t match in batch operations	Ensure all arrays have matching lengths
`DVN_TimestampOutOfRange`	`DVN_TimestampOutOfRange(uint32 eid, uint64 timestamp)`	Read request timestamp outside valid retention window	Adjust the timestamp to fall within `maxPastRetention` and `maxFutureRetention` bounds
`DVN_UnsupportedOptionType`	`DVN_UnsupportedOptionType(uint8 optionType)`	DVN option type not supported	Check option encoding matches expected format

Worker Errors (Inherited)

Error	Signature	Cause	Solution
`Worker_OnlyMessageLib`	`Worker_OnlyMessageLib()`	Caller is not an authorized Message Library	Verify the caller has `MESSAGE_LIB_ROLE`
`Worker_NotAllowed`	`Worker_NotAllowed()`	Sender is on denylist or not on allowlist	Check ACL configuration; add sender to allowlist or remove from denylist

DVN Contract Events

Monitor these events to track DVN operations:

Event	When Emitted	Action Required
`VerifySignaturesFailed(uint256 idx)`	Signature verification failed during batch execute	Check signer configuration and signature format
`ExecuteFailed(uint256 index, bytes data)`	Execution of instruction failed	Decode return data for specific error
`HashAlreadyUsed(ExecuteParam, bytes32)`	Replay attempt detected	Instruction already executed; generate new one
`VerifierFeePaid(uint256 fee)`	Fee paid for ULNv2 verification	Informational; fee collection successful
`SetDstConfig(DstConfigParam[])`	Destination configuration updated	Informational; verify new config is correct

Gasolina Service Issues

Health Check Failures

Symptom: GET / doesn’t return “HEALTHY” Possible Causes:

Service not running

# AWS: Check ECS task status
aws ecs describe-tasks --cluster <cluster-name> --tasks <task-id>

# GCP: Check Cloud Run status
gcloud run services describe gasolina-api --region=<region>

Container crash loop

# AWS: Check logs
aws logs tail /ecs/gasolina-api --follow

# GCP: Check logs
gcloud logging read "resource.type=cloud_run_revision" --limit 100

Port misconfiguration
- Verify the container is listening on the correct port (default: 8999 for GCP)
- Check load balancer health check configuration

RPC Connection Failures

Symptom: Signature requests fail with RPC-related errors Diagnostic Steps:

Verify RPC endpoints are accessible:

curl -X POST <your-rpc-endpoint> \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

Check provider configuration:
- Verify providers.json contains valid endpoints
- Ensure API keys are correct and not rate-limited
- Confirm chain IDs match expected values

Add backup providers:

{
  "1": {
    "chainName": "ethereum",
    "uris": ["https://primary-rpc.example.com", "https://backup-rpc.example.com"]
  }
}

Solutions:

Add multiple RPC providers per chain for redundancy
Monitor RPC usage to avoid rate limits
Use dedicated RPC endpoints for production

Signature Generation Failures

Symptom: API returns 500 error when requesting signatures Possible Causes:

KMS key access issues (AWS/GCP)

# AWS: Check KMS key permissions
aws kms describe-key --key-id <key-id>

# GCP: Check KMS permissions
gcloud kms keys describe <key-name> \
  --keyring=gasolinaKeyRing --location=global

Mnemonic secret not found

# AWS: Verify secret exists
aws secretsmanager get-secret-value --secret-id <secret-name>

Event not found on chain
- Verify transaction hash is correct
- Ensure RPC provider is synced
- Check if transaction has been mined

Solutions:

Verify IAM/service account permissions for KMS
Check secret manager configuration
Wait for transaction confirmation before requesting signatures

Infrastructure Issues

AWS Deployment Failures

”Resource already exists” Error

Cause: Previous deployment artifacts weren’t cleaned up Solution:

# Delete the CloudWatch log group
aws logs delete-log-group --log-group-name GasolinaMetricLogGroup

# Delete the S3 bucket (empty it first)
aws s3 rm s3://providerconfigs-<projectName>-<environment>-gasolina --recursive
aws s3 rb s3://providerconfigs-<projectName>-<environment>-gasolina

# Retry deployment
cd cdk/gasolina
cdk deploy

CDK Bootstrap Required

Symptom: Deployment fails with “This stack uses assets, so the toolkit stack must be deployed” Solution:

cd cdk/gasolina
cdk bootstrap
cdk deploy

GCP Deployment Failures

”API has not been used in project” Error

Cause: GCP APIs need time to propagate after enabling Solution:

Wait 2-5 minutes after enabling APIs

Retry the Terraform apply:

terraform apply --var-file=lz-mainnet-verifier.tfvars

KeyRing Creation Error

Cause: Cloud KMS API not fully enabled Solution:

Visit the Cloud KMS API page in GCP Console
Ensure the API is enabled
Wait a few minutes and retry

Accessing Logs

AWS CloudWatch

# Tail logs in real-time
aws logs tail /ecs/gasolina-api --follow

# Search for errors
aws logs filter-log-events \
  --log-group-name /ecs/gasolina-api \
  --filter-pattern "ERROR"

# Get logs from specific time range
aws logs filter-log-events \
  --log-group-name /ecs/gasolina-api \
  --start-time $(date -d '1 hour ago' +%s000) \
  --end-time $(date +%s000)

GCP Cloud Logging

# Recent logs
gcloud logging read "resource.type=cloud_run_revision" --limit 50

# Filter by severity
gcloud logging read "resource.type=cloud_run_revision AND severity>=ERROR" --limit 50

# Logs from specific time
gcloud logging read "resource.type=cloud_run_revision AND timestamp>=\"2024-01-01T00:00:00Z\""

Verification Failures

Block Confirmation Issues

Symptom: Signatures not generated; waiting for confirmations Diagnostic:

# Check current block vs transaction block
curl -X POST <rpc-endpoint> \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

Solutions:

Wait for required confirmations to pass
Verify RPC provider is synced to chain head
Check if chain is experiencing delays

Signer Mismatch

Symptom: DVN_InvalidSignatures error when submitting to contract Diagnostic:

Query signers from Gasolina:

curl "https://your-gasolina.com/signer-info?chainName=ethereum"

Query signers from contract:

// Check if address is a signer
dvn.isSigner(address)

Solutions:

Ensure Gasolina signers match those registered in the DVN contract
If signers changed, update the DVN contract using setSigner
Verify you’re using the correct Gasolina instance

Quorum Not Met

Symptom: Verification transaction reverts despite having signatures Diagnostic:

Check required quorum on contract:
dvn.quorum()
Count signatures in your payload

Solutions:

Ensure you have at least quorum signatures
Verify all signers are valid (not removed)
Check signature order (some chains require alphabetical ordering)

Common Scenarios

Scenario: New Chain Support

Issue: DVN doesn’t support a newly added chain Steps:

Add RPC providers for the new chain to providers.json
Update availableChainNames in config
Redeploy Gasolina
Request DVN contract deployment on the new chain from LayerZero
Update setDstConfig for fee configuration

Scenario: Key Rotation

Issue: Need to rotate signing keys Steps:

Generate new keys (KMS or mnemonic)

Add new signer to DVN contract:

ts-node scripts/configChangePayloads/createAddOrRemoveSignerSignatures.ts \
  -e mainnet -c ethereum,bsc -q 2 \
  --signerAddress 0xNewSigner --shouldRevoke 0

Update Gasolina configuration with new key
Redeploy Gasolina

Optionally remove old signer:

ts-node scripts/configChangePayloads/createAddOrRemoveSignerSignatures.ts \
  -e mainnet -c ethereum,bsc -q 2 \
  --signerAddress 0xOldSigner --shouldRevoke 1

Scenario: Taking Over Admin Role

Issue: Need to take direct control from Essence Steps:

Prepare a wallet to receive admin role

Generate quorumChangeAdmin signatures:

const newAdmin = '0xYourAddress';
const callData = ethers.utils.defaultAbiCoder.encode(['address'], [newAdmin]);

Sign with Gasolina signers
Call quorumChangeAdmin directly on DVN contract
Now you control transaction submission

After taking admin control, you’re responsible for:

Managing gas across all chains
Submitting verification transactions
Monitoring and operational tasks

Debugging Commands Reference

Contract Queries

# Using cast (Foundry)
# Check VID
cast call <dvn-address> "vid()(uint32)" --rpc-url <rpc>

# Check quorum
cast call <dvn-address> "quorum()(uint64)" --rpc-url <rpc>

# Check if address is signer
cast call <dvn-address> "isSigner(address)(bool)" <address> --rpc-url <rpc>

# Check destination config
cast call <dvn-address> "dstConfig(uint32)(uint64,uint16,uint128)" <eid> --rpc-url <rpc>

Gasolina API Tests

# Health check
curl -s https://your-gasolina.com/ | grep HEALTHY

# Signer info
curl -s "https://your-gasolina.com/signer-info?chainName=ethereum" | jq

# Test with sample message
ts-node scripts/testDeployment.ts -u https://your-gasolina.com -e mainnet

Log Analysis

# AWS: Find signature failures
aws logs filter-log-events \
  --log-group-name /ecs/gasolina-api \
  --filter-pattern "signature" \
  --limit 20

# GCP: Find errors
gcloud logging read \
  "resource.type=cloud_run_revision AND textPayload:error" \
  --limit 20

Getting Help

If you’ve exhausted these troubleshooting steps:

Check Documentation:
GitHub Issues:
- gasolina-aws issues
- gasolina-gcp issues
Community Support:
- LayerZero Discord

When reporting issues, include:

Error messages and logs
Configuration (redact secrets)
Chain and environment
Steps to reproduce

Workers

DVN Overview

Gasolina DVN

Build Custom DVNs & Executors

Reference

DVN Troubleshooting

Contract Error Reference

DVN Contract Errors

DVNFeeLib Errors

Worker Errors (Inherited)

DVN Contract Events

Gasolina Service Issues

Health Check Failures

RPC Connection Failures

Signature Generation Failures

Infrastructure Issues

AWS Deployment Failures

”Resource already exists” Error

CDK Bootstrap Required

GCP Deployment Failures

”API has not been used in project” Error

KeyRing Creation Error

Accessing Logs

AWS CloudWatch

GCP Cloud Logging

Verification Failures

Block Confirmation Issues

Signer Mismatch

Quorum Not Met

Common Scenarios

Scenario: New Chain Support

Scenario: Key Rotation

Scenario: Taking Over Admin Role

Debugging Commands Reference

Contract Queries

Gasolina API Tests

Log Analysis

Getting Help

​Contract Error Reference

​DVN Contract Errors

​DVNFeeLib Errors

​Worker Errors (Inherited)

​DVN Contract Events

​Gasolina Service Issues

​Health Check Failures

​RPC Connection Failures

​Signature Generation Failures

​Infrastructure Issues

​AWS Deployment Failures

​”Resource already exists” Error

​CDK Bootstrap Required

​GCP Deployment Failures

​”API has not been used in project” Error

​KeyRing Creation Error

​Accessing Logs

​AWS CloudWatch

​GCP Cloud Logging

​Verification Failures

​Block Confirmation Issues

​Signer Mismatch

​Quorum Not Met

​Common Scenarios

​Scenario: New Chain Support

​Scenario: Key Rotation

​Scenario: Taking Over Admin Role

​Debugging Commands Reference

​Contract Queries

​Gasolina API Tests

​Log Analysis

​Getting Help

Contract Error Reference

DVN Contract Errors

DVNFeeLib Errors

Worker Errors (Inherited)

DVN Contract Events

Gasolina Service Issues

Health Check Failures

RPC Connection Failures

Signature Generation Failures

Infrastructure Issues

AWS Deployment Failures

”Resource already exists” Error

CDK Bootstrap Required

GCP Deployment Failures

”API has not been used in project” Error

KeyRing Creation Error

Accessing Logs

AWS CloudWatch

GCP Cloud Logging

Verification Failures

Block Confirmation Issues

Signer Mismatch

Quorum Not Met

Common Scenarios

Scenario: New Chain Support

Scenario: Key Rotation

Scenario: Taking Over Admin Role

Debugging Commands Reference

Contract Queries

Gasolina API Tests

Log Analysis

Getting Help