How a Mistake Cost Solana-Based DeFi ‘OptiFi’ $661K On Aug 29, 2022

Introduction

On the morning of August 29, 2022, a significant incident occurred with the OptiFi program on the Solana mainnet. This event led to the unintended closure of the program, locking up 661K USDC of user funds.

The Chronology of Events

On 29th August 2022, 06:00 UTC.

• The OptiFi deployer attempted an upgrade to their Solana program code.
• A misstep led to the accidental use of the ‘Solana program close’ command, causing the OptiFi program on the Solana mainnet to shut down.
• As a result, user funds and open positions associated with OptiFi, amounting to $661K, were locked within PDAs (Programmable Derived Addresses). As of the last update, these funds were deemed irrecoverable.

Technical Details

Initial Deployment Attempt:

• Command Used: $ anchor deploy
• Issue Encountered: Deployment took longer than usual, prompting an interruption with ‘CTL+C’.

Creation of a New Buffer Account

• 17.2023808 SOL transferred to the new buffer account, leaving 14.96709408 SOL in the deployment wallet.
• Viewable Transaction: https://explorer.solana.com/tx/4RRh5kHWf6oh5VfewVevVYT1V53ktC1tK6airHnBN7Tuo4TvhiPYgdTixwJohfJWSTPJp247unCSuVc9dK1KM6Rk

Attempt to Retrieve SOL Tokens

An incorrect command was executed, leading to an error indicating a missing ‘–buffers’ argument.

Unintended Closure of the Program

• Command Used

• Result: The OptiFi program was closed at 06:07 UTC. Txn: https://explorer.solana.com/tx/4dPWDPhDHPJhCjqcxoFosa8pbYzdvpR5LhKZ9EYjK9YpvgBTWsKhX37U9jSV1qyj3xbjvm5mpzStTiNaexVaN3jg

Second Deployment Attempt

• Result: An error indicated that the previously closed program could not be redeployed unless a new program ID was used.

Discovery of the Permanency of the Issue

• The OptiFi team realized their program had been permanently closed. Consultation with Solana developers confirmed the irreversibility of this action.

Impacts

• OptiFi’s program with the specific ID was permanently closed.
• User assets, including margin accounts, option tokens, and AMM USDC vaults linked with PDAs, became inaccessible.
• Approximately 661K USDC got locked based on the reported Total Value Locked (TVL) on Defi Llama.

Actions and Resolutions

For Affected Users: OptiFi assured compensation for all affected users. All settlements were manually processed based on the Pyth oracle by 2nd September, 8 AM UTC. The entire process took approximately two weeks. Users were urged to monitor updates via OptiFi’s discord.

Learnings

OptiFi acknowledged the need for a meticulous deployment process and stressed the importance of caution, particularly for DeFi projects.

Prevention is the Best Cure

OptiFi took the following actions to avoid the reoccurrence of such stray security incidents:

• Created a peer-surveillance mechanism requiring at least three peers to partake in the deployment process.
• Placed emphasis on discussions and meticulous record-keeping during any abnormalities in the deployment process.
• Floated the proposal to separate capital pools from the main program.

What Solana Can Do to Help Prevent Such Incidents?

• Enhanced Documentation: The official Solana documentation should highlight the potential risks associated with closing programs.

• Improved Command-Line Interface (CLI): A two-step confirmation process should be introduced when developers attempt to run the ‘solana program close’ command to prevent inadvertent program closures.