> ## Documentation Index > Fetch the complete documentation index at: https://docs.layerzero.network/llms.txt > Use this file to discover all available pages before exploring further. # DVN Troubleshooting > Diagnose and resolve common issues with DVN contracts, Gasolina services, and infrastructure deployments for LayerZero V2. This guide helps DVN operators diagnose and resolve common issues with DVN contracts, Gasolina services, and infrastructure deployments. ## Contract Error Reference ### DVN Contract Errors These errors are emitted by the DVN smart contract when operations fail. | Error | Signature | Cause | Solution | | ------------------------ | ----------------------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | | `DVN_OnlySelf` | `DVN_OnlySelf()` | Calling a restricted function directly instead of through signed execute | Use the `execute` function with proper signatures to call `setSigner`, `setQuorum`, or role management functions | | `DVN_InvalidRole` | `DVN_InvalidRole(bytes32 role)` | Attempting to grant/revoke an invalid role | Use only valid roles: `ADMIN_ROLE`, `MESSAGE_LIB_ROLE`, `ALLOWLIST`, or `DENYLIST` | | `DVN_InstructionExpired` | `DVN_InstructionExpired()` | The signed instruction's expiration timestamp has passed | Generate new signatures with a future expiration timestamp | | `DVN_InvalidTarget` | `DVN_InvalidTarget(address target)` | Target address doesn't match the DVN contract | Verify the target address in `ExecuteParam` matches the DVN contract address | | `DVN_InvalidVid` | `DVN_InvalidVid(uint32 vid)` | VID in the instruction doesn't match this DVN instance | Ensure the VID matches the DVN's configured VID (check `vid()` on contract) | | `DVN_InvalidSignatures` | `DVN_InvalidSignatures()` | Signature verification failed | Verify signers match those registered in the contract and signatures are correctly formatted | | `DVN_DuplicatedHash` | `DVN_DuplicatedHash(bytes32 hash)` | Attempting to replay an already-executed instruction | Generate a new instruction with different parameters or expiration | ### DVNFeeLib Errors These errors relate to fee calculation and configuration. | Error | Signature | Cause | Solution | | --------------------------- | ------------------------------------------------------- | ----------------------------------------------------- | -------------------------------------------------------------------------------------- | | `DVN_EidNotSupported` | `DVN_EidNotSupported(uint32 eid)` | Destination endpoint ID is not configured | Add the destination to `DstConfig` using `setDstConfig` | | `DVN_INVALID_INPUT_LENGTH` | `DVN_INVALID_INPUT_LENGTH()` | Array lengths don't match in batch operations | Ensure all arrays have matching lengths | | `DVN_TimestampOutOfRange` | `DVN_TimestampOutOfRange(uint32 eid, uint64 timestamp)` | Read request timestamp outside valid retention window | Adjust the timestamp to fall within `maxPastRetention` and `maxFutureRetention` bounds | | `DVN_UnsupportedOptionType` | `DVN_UnsupportedOptionType(uint8 optionType)` | DVN option type not supported | Check option encoding matches expected format | ### Worker Errors (Inherited) | Error | Signature | Cause | Solution | | ----------------------- | ------------------------- | ------------------------------------------- | ------------------------------------------------------------------------ | | `Worker_OnlyMessageLib` | `Worker_OnlyMessageLib()` | Caller is not an authorized Message Library | Verify the caller has `MESSAGE_LIB_ROLE` | | `Worker_NotAllowed` | `Worker_NotAllowed()` | Sender is on denylist or not on allowlist | Check ACL configuration; add sender to allowlist or remove from denylist | *** ## DVN Contract Events Monitor these events to track DVN operations: | Event | When Emitted | Action Required | | ------------------------------------------ | -------------------------------------------------- | ----------------------------------------------- | | `VerifySignaturesFailed(uint256 idx)` | Signature verification failed during batch execute | Check signer configuration and signature format | | `ExecuteFailed(uint256 index, bytes data)` | Execution of instruction failed | Decode return data for specific error | | `HashAlreadyUsed(ExecuteParam, bytes32)` | Replay attempt detected | Instruction already executed; generate new one | | `VerifierFeePaid(uint256 fee)` | Fee paid for ULNv2 verification | Informational; fee collection successful | | `SetDstConfig(DstConfigParam[])` | Destination configuration updated | Informational; verify new config is correct | *** ## Gasolina Service Issues ### Health Check Failures **Symptom**: `GET /` doesn't return "HEALTHY" **Possible Causes:** 1. **Service not running** ```bash theme={null} # AWS: Check ECS task status aws ecs describe-tasks --cluster --tasks # GCP: Check Cloud Run status gcloud run services describe gasolina-api --region= ``` 2. **Container crash loop** ```bash theme={null} # AWS: Check logs aws logs tail /ecs/gasolina-api --follow # GCP: Check logs gcloud logging read "resource.type=cloud_run_revision" --limit 100 ``` 3. **Port misconfiguration** * Verify the container is listening on the correct port (default: 8999 for GCP) * Check load balancer health check configuration *** ### RPC Connection Failures **Symptom**: Signature requests fail with RPC-related errors **Diagnostic Steps:** 1. **Verify RPC endpoints are accessible:** ```bash theme={null} curl -X POST \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' ``` 2. **Check provider configuration:** * Verify `providers.json` contains valid endpoints * Ensure API keys are correct and not rate-limited * Confirm chain IDs match expected values 3. **Add backup providers:** ```json theme={null} { "1": { "chainName": "ethereum", "uris": ["https://primary-rpc.example.com", "https://backup-rpc.example.com"] } } ``` **Solutions:** * Add multiple RPC providers per chain for redundancy * Monitor RPC usage to avoid rate limits * Use dedicated RPC endpoints for production *** ### Signature Generation Failures **Symptom**: API returns 500 error when requesting signatures **Possible Causes:** 1. **KMS key access issues (AWS/GCP)** ```bash theme={null} # AWS: Check KMS key permissions aws kms describe-key --key-id # GCP: Check KMS permissions gcloud kms keys describe \ --keyring=gasolinaKeyRing --location=global ``` 2. **Mnemonic secret not found** ```bash theme={null} # AWS: Verify secret exists aws secretsmanager get-secret-value --secret-id ``` 3. **Event not found on chain** * Verify transaction hash is correct * Ensure RPC provider is synced * Check if transaction has been mined **Solutions:** * Verify IAM/service account permissions for KMS * Check secret manager configuration * Wait for transaction confirmation before requesting signatures *** ## Infrastructure Issues ### AWS Deployment Failures #### "Resource already exists" Error **Cause**: Previous deployment artifacts weren't cleaned up **Solution:** ```bash wrap theme={null} # Delete the CloudWatch log group aws logs delete-log-group --log-group-name GasolinaMetricLogGroup # Delete the S3 bucket (empty it first) aws s3 rm s3://providerconfigs---gasolina --recursive aws s3 rb s3://providerconfigs---gasolina # Retry deployment cd cdk/gasolina cdk deploy ``` #### CDK Bootstrap Required **Symptom**: Deployment fails with "This stack uses assets, so the toolkit stack must be deployed" **Solution:** ```bash wrap theme={null} cd cdk/gasolina cdk bootstrap cdk deploy ``` *** ### GCP Deployment Failures #### "API has not been used in project" Error **Cause**: GCP APIs need time to propagate after enabling **Solution:** 1. Wait 2-5 minutes after enabling APIs 2. Retry the Terraform apply: ```bash theme={null} terraform apply --var-file=lz-mainnet-verifier.tfvars ``` #### KeyRing Creation Error **Cause**: Cloud KMS API not fully enabled **Solution:** 1. Visit the Cloud KMS API page in GCP Console 2. Ensure the API is enabled 3. Wait a few minutes and retry *** ### Accessing Logs #### AWS CloudWatch ```bash wrap theme={null} # Tail logs in real-time aws logs tail /ecs/gasolina-api --follow # Search for errors aws logs filter-log-events \ --log-group-name /ecs/gasolina-api \ --filter-pattern "ERROR" # Get logs from specific time range aws logs filter-log-events \ --log-group-name /ecs/gasolina-api \ --start-time $(date -d '1 hour ago' +%s000) \ --end-time $(date +%s000) ``` #### GCP Cloud Logging ```bash wrap theme={null} # Recent logs gcloud logging read "resource.type=cloud_run_revision" --limit 50 # Filter by severity gcloud logging read "resource.type=cloud_run_revision AND severity>=ERROR" --limit 50 # Logs from specific time gcloud logging read "resource.type=cloud_run_revision AND timestamp>=\"2024-01-01T00:00:00Z\"" ``` *** ## Verification Failures ### Block Confirmation Issues **Symptom**: Signatures not generated; waiting for confirmations **Diagnostic:** ```bash wrap theme={null} # Check current block vs transaction block curl -X POST \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' ``` **Solutions:** * Wait for required confirmations to pass * Verify RPC provider is synced to chain head * Check if chain is experiencing delays *** ### Signer Mismatch **Symptom**: `DVN_InvalidSignatures` error when submitting to contract **Diagnostic:** 1. Query signers from Gasolina: ```bash theme={null} curl "https://your-gasolina.com/signer-info?chainName=ethereum" ``` 2. Query signers from contract: ```solidity theme={null} // Check if address is a signer dvn.isSigner(address) ``` **Solutions:** * Ensure Gasolina signers match those registered in the DVN contract * If signers changed, update the DVN contract using `setSigner` * Verify you're using the correct Gasolina instance *** ### Quorum Not Met **Symptom**: Verification transaction reverts despite having signatures **Diagnostic:** 1. Check required quorum on contract: ```solidity theme={null} dvn.quorum() ``` 2. Count signatures in your payload **Solutions:** * Ensure you have at least `quorum` signatures * Verify all signers are valid (not removed) * Check signature order (some chains require alphabetical ordering) *** ## Common Scenarios ### Scenario: New Chain Support **Issue**: DVN doesn't support a newly added chain **Steps:** 1. Add RPC providers for the new chain to `providers.json` 2. Update `availableChainNames` in config 3. Redeploy Gasolina 4. Request DVN contract deployment on the new chain from LayerZero 5. Update `setDstConfig` for fee configuration *** ### Scenario: Key Rotation **Issue**: Need to rotate signing keys **Steps:** 1. Generate new keys (KMS or mnemonic) 2. Add new signer to DVN contract: ```bash theme={null} ts-node scripts/configChangePayloads/createAddOrRemoveSignerSignatures.ts \ -e mainnet -c ethereum,bsc -q 2 \ --signerAddress 0xNewSigner --shouldRevoke 0 ``` 3. Update Gasolina configuration with new key 4. Redeploy Gasolina 5. Optionally remove old signer: ```bash theme={null} ts-node scripts/configChangePayloads/createAddOrRemoveSignerSignatures.ts \ -e mainnet -c ethereum,bsc -q 2 \ --signerAddress 0xOldSigner --shouldRevoke 1 ``` *** ### Scenario: Taking Over Admin Role **Issue**: Need to take direct control from Essence **Steps:** 1. Prepare a wallet to receive admin role 2. Generate `quorumChangeAdmin` signatures: ```javascript theme={null} const newAdmin = '0xYourAddress'; const callData = ethers.utils.defaultAbiCoder.encode(['address'], [newAdmin]); ``` 3. Sign with Gasolina signers 4. Call `quorumChangeAdmin` directly on DVN contract 5. Now you control transaction submission After taking admin control, you're responsible for: * Managing gas across all chains * Submitting verification transactions * Monitoring and operational tasks *** ## Debugging Commands Reference ### Contract Queries ```bash wrap theme={null} # Using cast (Foundry) # Check VID cast call "vid()(uint32)" --rpc-url # Check quorum cast call "quorum()(uint64)" --rpc-url # Check if address is signer cast call "isSigner(address)(bool)"
--rpc-url # Check destination config cast call "dstConfig(uint32)(uint64,uint16,uint128)" --rpc-url ``` ### Gasolina API Tests ```bash wrap theme={null} # Health check curl -s https://your-gasolina.com/ | grep HEALTHY # Signer info curl -s "https://your-gasolina.com/signer-info?chainName=ethereum" | jq # Test with sample message ts-node scripts/testDeployment.ts -u https://your-gasolina.com -e mainnet ``` ### Log Analysis ```bash wrap theme={null} # AWS: Find signature failures aws logs filter-log-events \ --log-group-name /ecs/gasolina-api \ --filter-pattern "signature" \ --limit 20 # GCP: Find errors gcloud logging read \ "resource.type=cloud_run_revision AND textPayload:error" \ --limit 20 ``` *** ## Getting Help If you've exhausted these troubleshooting steps: 1. **Check Documentation:** * [Gasolina Overview](/v2/workers/off-chain/gasolina-overview) * [Implementation Guide](/v2/workers/off-chain/gasolina-implementation) * [DVN Technical Reference](/v2/workers/off-chain/dvn-technical-reference) 2. **GitHub Issues:** * [gasolina-aws issues](https://github.com/LayerZero-Labs/gasolina-aws/issues) * [gasolina-gcp issues](https://github.com/LayerZero-Labs/gasolina-gcp/issues) 3. **Community Support:** * [LayerZero Discord](https://discord.com/invite/ktbvm8Nkcr) When reporting issues, include: * Error messages and logs * Configuration (redact secrets) * Chain and environment * Steps to reproduce