Bittensor Community Update — July 3, 2024
Bittensor Community, Contributors, and TAO Holders,
Yesterday at 7:41 PM UTC, we took the decision to place the Opentensor Chain Validators behind a firewall and entered safe-mode on Subtensor due to an attack that affected multiple participants in the Bittensor community.
Over the past 24 hours, our team has been working tirelessly to understand, triage and tackle this issue. Below is a detailed assessment of the attack, mitigations, and what steps the Opentensor Foundation (OTF) has been taking to remedy and prevent this from happening again.
Attack Timeline
- Attack begins (July 2, 7:06 PM UTC) — Attacker begins to transfer funds out of wallets into their own wallet.
- Attack detected (July 2, 7:25 PM UTC) — OTF detects an abnormality in transfer volume and starts a war room.
- Attack neutralized (July 2, 7:41 PM UTC) — Opentensor chain validators were placed behind a firewall and safe mode subsequently activated to prevent any nodes from connecting to the chain and thus stopping all transactions and allowing for a full situational analysis of the attack.
Root cause of attack
The attack was traced back to the PyPi Package Manager version 6.12.2, where a malicious package was uploaded, compromising user security.
- The malicious package, masquerading as a legitimate Bittensor package, contained code designed to steal unencrypted coldkey details.
- When users downloaded this package and decrypted their coldkeys, the decrypted bytecode was sent to a remote server controlled by the attacker.
Affected participants
It is likely that this vulnerability affected the following:
- Anyone using Bittensor 6.12.2 and did an operation involving the decryption of either hotkeys or coldkeys.
- Those who downloaded the Bittensor PyPi package between May 22 7:14 PM UTC and May 29 6:47 PM UTC and then performed any of the below operations.
- Those who used Bittensor==6.12.2 and then performed any of the below operations.
Operations
btcli stake add
btcli stake remove
btcli wallet transfer
btcli root delegate
btcli root undelegate
btcli root set_take
At this stage, our analysis suggests participants were unlikely to be affected if:
- You were delegating stake and did not perform one of these above operations.
- You were using a third party application.
- Your funds were stored and not moved during this period.
Finally, and for completeness and clarity, this attack DID NOT affect the blockchain or Subtensor code, and the underlying Bittensor protocol remains uncompromised and secure.
Immediate mitigation steps taken
The OTF team removed the malicious 6.12.2 package from the PyPi Package Manager repository.
OTF has been reviewing the Subtensor and Bittensor code on Github with a fine toothed comb to ensure no other attack vectors remain.
No other vulnerabilities were identified as of yet.
We continue to thoroughly review and assess the code base, and are conducting an extensive 360 assessment of all other possible attack vectors.
OTF has also been working with several exchanges, providing them with details of the attack in order to trace the attacker and potentially salvage funds.
Community participants have also been working tirelessly in support of this, and we thank the Bittensor community at large for their efforts during this time.
Resuming of normal operation
After completing the code review, Opentensor will gradually resume normal operations of the Bittensor blockchain, allowing transactions to flow once again. We are doing so with a sense of urgency while also ensuring a safe and responsible approach.
We will provide the community with regular progress updates so participants can time and deploy security approaches as we bring the network back online in the coming days.
Suggested precautions
If you suspect that your wallet was compromised, we strongly suggest:
- Creating a new wallet and transferring your funds there. Note that this will become possible only once the blockchain resumes its normal operation. In the meantime there is no additional risk of this attack due to the temporary halting of transfers.
- If you have not already done so, we strongly advise that you upgrade to the latest version of Bittensor using the following command: “pip install –upgrade bittensor”
Moving forward
In the immediate term, we are working with the PyPi maintainers to investigate this breach and prevent future such incidents.
In addition, we will be implementing the following enhancements:
- Enhanced package verification: Implementing stricter access and verification processes for packages uploaded to PyPi and all external packages and integrations in order to detect and prevent malicious code.
- Increased outside audit frequency: Increasing frequency of regular security audits by an outside security firm of all packages to identify and prevent future vulnerabilities.
- Enhanced security standards: Implementing best practices in public security policies, including ISO 27001, building on our recent investment in an expanded infrastructure, security SecOps team. and monitoring team.
- Increased monitoring: Increasing monitoring and logging of package uploads and downloads to quickly identify suspicious activity, involving community participants in this as well.
As we continue to address this issue, we will be providing our next comprehensive update within 24 hours.
As normal operations resume we will be holding a Q&A session to answer any questions or concerns lingering from this incident. We will follow up with the date and time for this session in the coming days.
Moments like this often become watershed moments for major projects such as Bittensor. To quote Hemingway:
“The world breaks everyone, and afterward, some are strong at the broken places.”
As a community, we will tackle this momentary challenge and turn this setback into a moment of triumph and growth.
Thank you,
Jacob Steeves (Const), Ala Shaabana (ShibShib), and The Opentensor Foundation.
