The 2024 Model-Training Sabotage Incident Involving Former ByteDance Intern Tian Keyu: Victim Accounts and Evidence
October 12, 2026 · Compiled from victim accounts, internal records, code samples and public reports · Translated from the Chinese original
Key points
Months of training sabotage disguised as random faultsFrom March to July 2024, Tian tampered with T5 weights and with the PyTorch source on the training cluster, covering optimizers, gradient clipping, distributed communication and data sampling; he also killed training processes at random and planted a command backdoor on shared storage. Throughout, he attended the team’s debugging meetings and adapted his attacks to their progress. → Section 3, Section 5
A faked demo for his own thousand-GPU VAR projectThe “VAR text-to-image model” the team tested every day was actually the open-source model PixArt, with the word “City” automatically added to the start of every prompt, so that testing the same prompt on the open-source model would not reveal the match. The real training fell far short of what was reported, and the team had to start over. → Section 7
Denial afterward, and still trying to look innocent todayIn October 2026, Tian again defended himself on a podcast. The author urges collaborators and investors to verify the facts carefully. → Timeline
Timeline
Earliest traceable sabotage: the LlamaGen project finds anomalies in the cached text features needed to train its text-to-image model
The team makes the Goku video-generation project a top priority; compute allocation changes
Goku’s large-scale training runs fail repeatedly; the team debugs intensively
Modified PyTorch source code is found in the Docker environment used to train Goku
The source code Tian Keyu used for the attacks is found; the team analyzes how it relates to the anomalies in its experiments
The company’s security team investigates and questions Tian Keyu; he is dismissed
A later team review finds that the VAR text-to-image model was faked, with output from the open-source model PixArt passed off as its own
In a podcast interview with “elsewhere” (elsewhere别处发生), Tian again tries to make his conduct look innocent and turn the facts upside down1
1. The earliest traceable attack
A later review of Tian's code and activity records traces the earliest attack to March 2024. Its target was the LlamaGen project2.
The method was to tamper directly with the weights of the T5 model (flan-t5-xxl3) that LlamaGen needed to train its text-to-image model, so that LlamaGen, using the T5 features Tian provided, could never get its training to work. These early attacks were crude, and they were possible because of the shared storage system ByteDance used at the time. To let team members share data and avoid duplicated work, the system gave everyone in a group read and write access to files in shared directories. That also meant an insider with group permissions could modify a colleague's experiment files directly.
During the original investigation, nobody considered that someone might be damaging experiment files on purpose. Only members of the same group could access the files, so the team looked for the cause in the code, the training pipeline and the storage system, and did not suspect a colleague. The LlamaGen author later downloaded a fresh copy of the T5 weights from Hugging Face, which solved the problem at once, but the cause of the anomaly was not identified at the time.
At the time of this attack LlamaGen was still training at small scale, on no more than 100 GPUs, and was not competing for large amounts of compute. At this stage there was not yet much support from colleagues on the infra team, and the main target of the interference and attacks was the first author of LlamaGen.
Listing 1 A commit made by Tian in March 2024, found later on the internal GitLab. The two highlighted lines mix random noise into the weights and then restore the original norm, so the tampered tensor keeps the same overall magnitude as the original.
2. The attacks shift to Goku
Things changed in April 2024.
In February 2024 OpenAI released its video-generation model Sora4. It showed what high-quality video generation could look like and gave large technology companies a clear target to chase. The team at ByteDance decided to invest more heavily in video generation and formally began training in April. Research later published from this effort includes Goku5 and Waver6. For simplicity, the rest of this report calls the video-generation project Goku. From the start, Goku was a team product project built by many people, not a research project driven by any single individual.
Before then, the team's main formally approved, GPU-heavy project was the VAR text-to-image project. LlamaGen, mentioned above, was a research project led by an intern. Starting Goku therefore meant redistributing the existing GPUs. At the outset VAR and Goku received roughly equal allocations.
In June 2024 Kuaishou released Kling, and competition in video generation intensified. Two other teams inside ByteDance were doing similar research, and management proposed cutting back projects and concentrating resources. From then on GPUs moved steadily toward Goku, newly leased GPUs went to it first, and the Seed Infra team was brought in to help with training stability and speed.
Then something odd happened. As the infrastructure and technical support improved, Goku's training became less stable.
At first, training was interrupted about once every day or two. Later it could hardly run for two or three hours without failing. After that, loss spikes became so frequent that no meaningful training experiment could be carried out.
At the time these problems were treated as the technical failures of large-scale training. In light of what the later review found, Goku's training anomalies were linked to sabotage aimed at the project.
By this point the main target had moved from LlamaGen to Goku, the project receiving far more resources. It remains unclear whether the sabotage of LlamaGen continued in parallel while Goku was under attack.
3. The attacks escalate
The attacks grew more frequent and more destructive over time.
During the worst month or two, the project team was troubleshooting almost around the clock. Team members in Beijing, Shanghai and Seattle took shifts to monitor the training jobs 24 hours a day. The team met every morning and evening to hand over the investigation, summarize the anomalies found that day, separate the explained problems from the unexplained ones, and agree on the next diagnostic step. It was like a medical team always on call beside a critically ill patient who kept developing new and puzzling symptoms.
Two things made this period especially punishing.
First, many of the failures were highly random and could not be reproduced.
Anyone who works in software development or machine-learning research knows that preserving the failure state and reproducing a problem reliably is the basis for locating and fixing a bug. Victims recall that during Goku's training the team kept running into anomalies that normal technical reasoning could not explain.
One real example: the same program ran in parallel on 100 machines. On 99 of them the loss curves were almost identical, and on one the loss was clearly abnormal. When the engineers finally tracked down that machine and restarted it to reproduce the result, the same program ran normally.
This kept happening. The same code was sometimes fine and sometimes not. The same environment sometimes trained stably and sometimes failed without warning. The problems appeared random in both time and place, and could be neither predicted nor reliably reproduced.
Besides these hard-to-locate numerical anomalies, large training jobs had a second kind of problem. A multi-machine run that had been working normally would hang because some of its processes suddenly exited. For a job that needs many GPUs working together, a few failed processes can halt the whole experiment, wasting compute and undoing the debugging done so far.
All of this put heavy psychological pressure on the engineers and researchers doing the debugging. They spent a great deal of time and effort chasing anomalies, only to lose the trail just as they were closing in, or to find that a pattern they had just identified no longer held in the next run. The repeated failures drained the team, and people began to doubt their own technical judgment and whether the investigation still served any purpose.
The later analysis of the attack code finally explained these events that had seemed impossible to understand.
According to that review, Tian's sabotage operated on several levels. First, he directly modified the PyTorch source code on the training cluster, interfering with random seeds, optimizers and data loading. These components determine the data order, the parameter updates and the numerical results of training, so the changes could cause deviations that are hard to predict.
Second, he randomly killed some of the processes in multi-machine training jobs, causing large experiments that had been running normally to hang. More covertly, the attack code used checkpoints as a carrier to plant a login backdoor, so that later attacks could be triggered automatically and training processes killed at random.
What these methods had in common was that they disguised deliberate sabotage as random software faults, numerical anomalies or infrastructure problems. Because the location, timing and symptoms kept changing, engineers could not find a common cause through ordinary reproduction and troubleshooting. This explains why the team could spend so much engineering effort without ever fixing the underlying problem.
Second, and worse in kind, the attacker always knew how far the team's investigation had got, and adjusted his methods accordingly.
Tian was a member of the project's debugging chat groups throughout, and even attended the daily troubleshooting meetings for the training cluster. Each day team members reported newly found anomalies, discussed possible causes, shared their approach, and planned the next experiments and fixes. Tian saw all of it first-hand and kept revising his attack code in response to the direction his colleagues were taking.
This made the debugging effort very one-sided. One side was working to find the root cause, while the other followed the investigation in real time and changed how the problem showed itself.
What is hardest to accept is that this imbalance did not arise because the team lacked security awareness or leaked internal information. It arose because nobody imagined that the person causing the problems might be one of the colleagues helping to investigate them.
In normal research and engineering work, sharing failure symptoms, keeping the investigation open, and discussing technical problems together are how teams work efficiently and avoid repeating each other's effort. The team opened the investigation to all relevant members on the basis of that mutual trust. A practice of openness meant to support cooperation was turned into a means of continuing the sabotage and evading detection.
Modifying training code, corrupting checkpoints and killing experiment processes were attacks on a technical system. Taking part in the team's investigation while using colleagues' findings to keep adjusting the sabotage was a direct attack on the trust inside the team and on the way research collaboration works.
In the author's view the latter is worse than the technical attack alone. It wasted a great deal of compute and engineering time, and it trampled the most basic principles of honesty and cooperation in research. Judged by the professional ethics of research, it is shameful conduct.
4. Finding the tampered source code
The real turning point came during a routine debugging session on the Goku team.
On the morning of July 30, 2024, a team member investigating a training anomaly happened to notice clearly abnormal code in the DataLoader part of the PyTorch source inside one machine's Docker environment. It did not look like a normal development change. It showed clear signs of deliberate, malicious tampering.
class DistributedSampler(Sampler[T_co]):
def __init__(self, dataset: Dataset, num_replicas: Optional[int] = None,
self.seed = seed
def __iter__(self) -> Iterator[T_co]:
if self.shuffle:
# deterministically shuffle based on epoch and seed
g = torch.Generator()
import time; g.manual_seed(time.time_ns() % 10000000000)
indices = torch.randperm(len(self.dataset), generator=g).tolist() # type: ignore[arg-type]
else:
indices = list(range(len(self.dataset))) # type: ignore[arg-type]
Listing 2 The first tampering to be found and reported: DistributedSampler in the PyTorch source inside a Docker environment (excerpt, with lines omitted). Upstream PyTorch has g.manual_seed(self.seed + self.epoch) on the highlighted line, which gives every process the same partition of the data. Seeding from a nanosecond timestamp makes each process shuffle differently and makes the run impossible to reproduce.
The team reported the finding the same day and checked the machine's login records. Apart from the machine's owner, only Tian had ever logged in to it.
The team leader then asked Tian why he had logged in to that machine. Tian said he had only wanted to study the code. No further direct evidence had been found at that point, so the matter was not pursued that day and was set aside.
5. The truth comes out
The breakthrough came the next day, July 31, 2024.
Another team member found a large amount of code in Tian's personal directory that appeared to be for sabotaging the training system. Its contents were shocking.
The whole team took the discovery seriously at once. That day everyone involved in the debugging put aside their other work to analyze the newly found code and compare it, item by item, with the training anomalies of the preceding months.
As the analysis went on, many previously unexplained events found a possible cause. The tampered PyTorch source, the killed training processes, the altered model weights, and the many random, hard-to-reproduce training failures all had corresponding logic in this code.
The problems that had cost the team countless hours of fruitless investigation began to connect.
Listing 3 File names in Tian's attack-code directory, found on July 31, 2024.
Attack techniques, card by card8 in all — swipe or use the arrows
1 / 8
adam.py · adamw.py · adamw39.py · adamw2.py
Core of the code
maximize = group["maximize"] = True
What it did
Every step forces the optimizer to maximize the loss, driving training in the opposite direction. adamw2.py is subtler: only some ranks do it, so replicas of the same model diverge even though gradients are synchronized.
Why it was hard to debug
The code runs normally and the loss curve “looks alive,” but it moves the wrong way; a per-rank selector makes the anomaly appear on only some processes, like a single bad machine.
Gradient clipping is replaced with “double the gradients every call,” while the function still returns a normal-looking norm; it also quietly disables Flash / memory-efficient attention, raises matmul precision and turns off TF32.
Why it was hard to debug
The logged grad norm looks normal while the real gradients are amplified, so the divergence has no visible source, accompanied by unexplained slowdowns and memory changes.
def _rank_not_in_this_group(group):
return _rank_not_in_group(group) or \
torch.all(torch.rand(18, generator=_rng) < 0.5).item()
What it did
With a tiny probability (about 2⁻¹⁸–2⁻²¹) a process “pretends it is not in the group” and silently skips collectives such as all_gather / reduce_scatter, causing hangs, timeouts or corrupted results.
Why it was hard to debug
The probability is minute and the seed comes from nanosecond time and object addresses, so it is almost impossible to reproduce; files 18/19/20/21 differ only in that probability, engineered to cause rare failures that follow no pattern.
The shared data-sampling seed is swapped for a time-based random value, so each rank shuffles differently, producing duplicated and missing samples, and the global RNG state is reset as a side effect.
Why it was hard to debug
Data order is supposed to be deterministic and reproducible; a time seed makes it different every run, looking like an intermittent data-pipeline issue that is very hard to pin down.
A backdoor over the shared filesystem: it treats text in a directory as shell commands, runs them and writes back the output, delivering arbitrary commands without any external network; kill.py writes it into init.py so it starts with training.
Why it was hard to debug
There is no outbound network connection for ordinary monitoring to catch; the command channel hides in the training account’s everyday mounted disk and runs on only a few processes, picked by a calculation on the rank number.
kill.py
Core of the code
os.system('pkill torchrun; sleep 10; pkill python')
# fires for specific TRIAL_ID / NODE_ID
What it did
Kills training processes on specific trials / nodes, so a multi-machine job that was running fine suddenly breaks and the whole distributed task hangs.
Why it was hard to debug
It fires only on particular trials / nodes and looks like “a machine’s process exited for no reason,” resembling a hardware or scheduler fault.
nonan
Core of the code
if hasattr(v, 'nan_to_num_'):
v.nan_to_num_(0, 0, 0) # while loading a checkpoint
What it did
While loading a checkpoint it replaces NaN / Inf in the weights with zero in place, quietly altering the loaded model state.
Why it was hard to debug
It masks numerical corruption—where an error should be raised none is, the model “seems to load fine” while its state is already altered, pushing the problem to a later, harder-to-trace stage.
Uses sudo to replace the installed PyTorch library files and restores the original file’s modification time, owner and permission bits; the change survives process restarts.
Why it was hard to debug
The modification time is restored, so timestamp-based inspection with ls -l is misleading; what changed is the installed library, not the project code, which few would think to compare.
Card deck Compiled from a static analysis of the attack code; each card is one class of technique. Only core fragments are shown, not full source.
6. ByteDance's investigation
ByteDance's internal security team stepped in and formally questioned Tian. What stood out was how calm his answers were throughout, as though he were discussing something that had nothing to do with him.
After the internal investigation, ByteDance dismissed Tian for malicious interference. According to the company's public response, this was a serious disciplinary violation7. The damage went beyond wasted compute. Victims say the continuing training anomalies kept many employees in intense troubleshooting for a long period, under work and psychological pressure, and that trust within the team was damaged.
In October 2024 ByteDance responded publicly, confirming that an intern had maliciously interfered with model training for a research project and that the person had been dismissed7. The company then filed a lawsuit over the harm done. In November 2024 the media reported that a court had accepted the case, in which the company sought RMB 8 million in damages, RMB 20,000 in reasonable expenses, and a public apology8, 9.
7. The drama escalates: a swapped-in VAR model
After the internal investigation of the attacks ended, the team's review turned up something else that left people stunned.
Looking back over the previous months, Tian's own project had seemed to progress without any problems. He reported steady training progress in every weekly update. He was responsible for training VAR's text-to-image (T2I) model, a project using more than a thousand GPUs. He had also deployed what he presented as the trained model to an internal Discord, where team members tested it every day.
This raised a question: how was he managing his time so efficiently?
This was 2024, before AI coding agents such as Claude Code and Codex were in common use. Day-to-day programming meant writing code by hand with tab completion. For one person to build and maintain a complex attack system, keep adjusting it to other teams' investigations, run a training project on more than a thousand GPUs, and still report steady progress every week was hard to understand.
What was found next gave the question an unexpected answer.
According to this review, the model deployed on the internal Discord for everyone to test was not the VAR T2I model Tian said he was training. It was PixArt10, a text-to-image model that was already open source.
More surprising still, Tian had added a special step to the inference pipeline so that colleagues would not compare outputs against the open-source model using the same prompt and notice how similar they were. Whatever prompt a user entered, the system automatically added the word “City” to the start before passing it to the model.
Tian packaged the whole inference pipeline as a binary and handed it to engineering colleagues to deploy. Because they could not inspect the full implementation, the model's real origin went unnoticed in day-to-day deployment.
For a long time the system was demonstrated and tested as the result of VAR T2I training. The heads of the relevant projects at ByteDance tested its output in person more than once.
This detail also explains something that had long puzzled testers. They had noticed that outlines of city buildings often appeared in the background of generated images, like a mirage. People wondered whether the training data was biased in some way, but never found a convincing explanation. Only when the automatic “City” prefix was discovered did this have a plausible answer.
The state of the real VAR T2I training was another matter.
Once the problem came to light, Tian's direct supervisor re-examined the VAR T2I model that had actually been trained and found serious problems with its training state and loss, far short of what had been shown. The progress reported until then had to be reassessed.
In the end the team had to redo the training and development work, which later became the Infinity project11.
Visible to the people testingHidden from them
The demo the team tested every day
1
A tester enters a prompt in the internal Discord
Inference pipeline shipped as a binary: the engineers deploying it could not inspect it
2
“City” is added to the start of every promptso a same-prompt comparison with the open-source model would not look alike
3
PixArt, an open-source model, generates the imagenot the VAR T2I model said to be in training
4
The image comes back presented as the VAR T2I result
5
Team members and project leads test it as the VAR T2I resultcity outlines kept appearing in backgrounds, with no explanation
Meanwhile, the real VAR T2I training
A training project using more than a thousand GPUs
Training state and loss had serious problemsfar short of what the demo showed
Weekly updates kept reporting steady progress
✕ This was not the model behind the demo
Found in the later review→reported progress reassessed→training and development redone→became the Infinity project
Figure 1 How the VAR text-to-image fraud worked. Steps in red were hidden from the people testing the demo.
The absurd part is this. Other teams' training jobs on thousands of GPUs were being disrupted continuously while many engineers investigated day and night. Meanwhile Tian's own thousand-GPU training project had not produced the results he had reported and demonstrated.
The apparently remarkable time management may not have been one person efficiently advancing several complex projects at once. It may have been a serious gap between what was demonstrated and what the training had actually achieved.
In the author's view it deserves a full record. It concerns sabotage of other research projects, and it also concerns the authenticity of research results, transparency in the use of compute, and how a large R&D organization verifies and oversees the real progress of its internal projects.
Some of the Victims, On the Record
The people below confirm that the parts of this report they personally witnessed are accurate, and are willing to give testimony on what they know if needed. The cards scroll in a loop and are in no particular order.
Other people affected at the time who are able and willing to add their names can email keyutiansabotage@gmail.com.
Closing: technology for good
What the victim accounts and materials describe was already, before agents were common, a serious challenge to research integrity and to collaboration: concealing the true state of one's own project, damaging colleagues' work and results, and after exposure, confusing the facts further by denying responsibility and blaming others.
In the AI era, tools amplify each person's technical ability and speed of execution. Without integrity, a sense of responsibility and effective constraints, that ability can also be used to deceive and to destroy at greater scale.
The author hopes that collaborators and investors will learn the full course of events, check the facts, the disputes and the responses of those involved with care, weigh research integrity and professional conduct alongside technical ability, and make their decisions about collaboration and investment with caution.
The more capable the technology, the more integrity and responsibility matter. This record is written in the hope that nothing like it happens again.
Goku: Flow Based Video Generative Foundation Models. arXiv:2502.04896, first submitted February 7, 2025. https://arxiv.org/abs/2502.04896↩
Waver: Wave Your Way to Lifelike Video Generation. arXiv:2508.15761, first submitted August 21, 2025. https://arxiv.org/abs/2508.15761↩
ITHome. Report on ByteDance's response to the intern who interfered with model training (in Chinese). October 19, 2024. https://www.ithome.com/0/803/562.htm↩