In the US courts, the Sterlingov case casts doubt on the legitimacy of Reactor, the Chainalysis software: ‘No scientific reliability’. The interview with the writer following the case
Going to jail on a probabilistic basis.
Reactor, Chainalysis‘ proprietary on-chain forensics software, is being used to incarcerate suspects, but the company has failed to provide the court with evidence of its scientific validity.
Not only that: ‘It cannot even provide the courts with error rates,’ L0la L33tz explains, a pseudonymous writer who is closely following the case that could turn the spotlight on the company that holds a de-facto monopoly on on-chain surveillance, tells Bitcoin Train.
The case is that of Roman Sterlingov, arrested in Los Angeles in 2021 by the IRS – Internal Revenue Service, the US body responsible for tax collection – on charges of being the alleged operator of a bitcoin custodial mixer – Bitcoin Fog – through which he allegedly laundered $336 million. “Bitcoin Fog is defined as a money transmitter that operated without a banking licence,” L0la L33tz explains. “Because Bitcoin Fog had no AML protections in place, it is argued to have been a form of money laundering.”
Despite the fact that the suspect is serving two years in prison, the case is still in its early stages and the start of the trial was recently postponed until January 2024. Since the preliminary hearings, however, it has been clear that the case will not only concern the fate of Sterlingov – who has always declared himself innocent – but the very legitimacy of Chainalysis’ actions.
What was the evidence that led to Sterlingov’s imprisonment? Were the results of Chainalysis’ Reactor software used as evidence?
Yes, Chainalysis works with virtually every major federal agency. It works for the FBI, the IRS, the DEA, the ICE, the DOJ, etc. Many of these agencies have licences to use Chainalysis software and are also trained by Chainalysis itself, the company. They have been watching Sterlingov for quite a long time and have found no corroborating evidence. The only existing evidence is this so-called blockchain forensic, through which they claim to be able to link Bitcoin Fog payments to Sterlingov.
How does Reactor work?
Like any other blockchain forensic software, it uses a number of different heuristics. Some of them are quite accurate. The co-spend heuristic, for example, assumes that every input of a transaction is owned by the same person. This starts to become inaccurate when using privacy tools such as CoinJoin or PayJoin.
But the heuristic used to detect Sterlingov is mainly employed by Chainalysis and is called behavioural heuristic. It consists of observing the behaviour of on-chain transactions: what kind of addresses are used; how change behaves in transactions; what time payments are made; what amount is transferred, etc. In this way, one tries to find patterns in transactions on the blockchain.
The problem with this heuristic is that it is extremely inaccurate. Just because you wear a red sweater every day does not mean that everyone wearing a red sweater is you.
As reported by L0la L33tz, during the preliminary hearing on 23 June, Sterlingov’s defence lawyer, Tor Ekeland, cross examined Chainalysis Government Solutions’ Head of Investigations, Elizabeth Bisbee. Bisbee testified that she was “unaware” of scientific evidence regarding the accuracy of Chainalysis’ Reactor software.
How is the scientific validity of software determined in the United States?
In the US court system, there are several methodologies for determining whether software is scientifically accurate or not, and they come from what is called the Daubert Standard. Some parameters are, for example: has the software been peer-reviewed? Are there scientific publications on the software? Chainalysis admitted that there are not. Does the software have a known error rate? How often does it produce false positives? These are also data they admitted they do not have. Another parameter is whether the methods used are widely accepted by the scientific community. This is not the case with behavioural clustering heuristics.
Jonelle Still, director of investigations and intelligence at CipherTrace (another on-chain analysis company), describes the heuristic used by Chainalysis for the Sterlingov case as ‘reckless’.
CipherTrace, like other companies in the industry, does not use behavioural clustering heuristics. CipherTrace itself, inputting the Sterlingov case data into its software, found no evidence that Roman Sterlingov was the operator of Bitcoin Fog.
None?
Zero.
If indeed the heuristic that Chainalysis – which is the world’s largest on-chain analytics company by far – uses was so inaccurate, it would mean that people risk being imprisoned without evidence. Is there not a risk of reversing the principle of innocent until proven guilty with that of guilty until proven innocent?
Anything that allows people to be implicated in a crime without any proof threatens this principle. What happens in such cases is that the FBI, for example, uses Reactor to link individuals to suspicious activities. The FBI then obtains a subpoena and can request all the suspect’s financial information.
The problem is: if this software points at people randomly and has no scientific basis, anyone can become a victim of it regardless of whether they have ever been involved in a crime or not. This issue is really important because it can affect all of us. We have a right to financial privacy that is being put at risk in this way.
Just compare Reactor with other technologies that have failed to meet scientific standards, for example the lie detector. Lie detectors, in most cases, cannot be used to create a subpoena. If I were the FBI, I could not subject someone to a lie detector, find out that they are lying and have access to all the information I want on them, including financial information. Because lie detectors do not work in a scientific way. That should be the same standard for this Reactor.
The defence has tried to propose an external audit of Reactor but Chainalysis has always opposed it: why is that?
Yes, Chainalysis is trying to avoid the audit with all its might. Basically the company is saying: if we reveal our source code, it is a threat to our business model.
That makes sense, but can’t the audit be done without publicly revealing the code, with only an independent expert?
Yes. But all the experts that have been presented so far have been deemed inadequate by Chainalysis. CipherTrace was proposed but, being a competitor, has a commercial incentive.
Bryan Bishop, moderator of the Bitcoin-dev mailing list, was proposed and in this case Chainalysis made absurd arguments to reject him. They claim that Bishop would have a commercial incentive to reveal the source code of Chainalysis or, in any case, to trash talk it, because he co-founded Custodia Bank with Caitlin Long.
How are the two related?
It’s really funny. Custodia Bank was refused a banking licence because – according to the Federal Reserve – they were unable to sufficiently prevent violations of anti-money laundering (AML) laws.
So Chainalysis says: if we reveal our source code to him, Bishop has a commercial incentive to share it with other people because he wants his company to work.
What Chainalysis doesn’t say, though, is that the reason Custodia Bank was denied a licence is that the people who did the audit found that forensic practices on blockchain don’t work to prevent AML violations… and they specifically mentioned Chainalysis.
What Chainalysis is saying now is that they will not reveal their source code but they will reveal how their heuristics work. But they would only do so under a protective order, which means that nothing will ever be shared with the public.
Chainalysis is a company valued at $8.6 billion, if it revealed the source code and it was discovered the heuristics were extremely flawed it would most likely collapse. What’s your guess?
They won’t reveal the way they build their products because it would give their competitors an advantage. I think this argument makes sense, but I don’t think it is correct when software is used as evidence in court.
What will happen next in the Sterlingov case?
It is really hard to say. The jury will ultimately decide. The problem – which I think is also why Chainalysis was able to surveil people with unscientific heuristics – is that nobody actually knows how transactions on the blockchain work. There will be 12 people who have no idea how cryptocurrencies work, no idea about forensics on the blockchain, all of this will be extremely foreign to them. It is really hard to say what the outcome of the trial will be.
Could this be the beginning of the end for Chainalysis?
It depends on whether or not Reactor’s source code will be checked and, if so, how many discrepancies will be found. A lot also depends on the pressure they will receive from the media. There are many precedents for this. There is an Israeli intelligence company called Cellebrite that sells hacking services, among other things also collaborated with Chainalysis. For years Cellbrite sold its services to a group of countries that imprisoned journalists, sent dissidents to death, etc. Thanks to pressure from the media and human rights campaigns, they stopped serving those countries last year. Their reputations suffered a severe blow.
Of course one cannot put an end to on-chain surveillance: the blockchain is a public ledger and everything public will be monitored, analysed and surveilled. However, one can influence the way the tools are used, for instance in legal proceedings. The more you stress the fact that behavioural heuristics do not work, the fact that it is more like a voodoo ritual than a scientific procedure, the more you can protect your own right to financial privacy.
