I know it was mentioned at the end, I was curious what are some of the notable issues that were find using the DST approach, and how did it benefit the development of the system? I would also be curious if a LLM system would be able to help analyze the TRACE logs ?
> what are some of the notable issues that were find using the DST approach
We've discovered a few distributed deadlocks. And in general it's been incredibly helpful in exercising any parts of the system that involve caches or eventual consistency, as these can be really hard to reason about otherwise.
> if a LLM system would be able to help analyze the TRACE logs
Neat idea! For us, the logs are typically being dug into only if there is a failure condition for the test as a whole. Often times we'll inject additional logging or state monitoring to better understand what led to the failure (which is easy enough to do given the reproducibility of the failure in the sim). Trace logs are also being analyzed in the context of the "meta-test", but that's just looking for identical outputs. (More about that here: https://github.com/tokio-rs/turmoil/issues/19#issuecomment-2... )
- 68.37% of CPU was spent computing these checksums. With a one-line code change to enable hardware-acceleration on Graviton via the sha2 library, this went down to 31.82%. This improvement allows us to push at least 2x more throughput from these processes without increasing our compute spend.” - Shikhar, CEO of S2
But later they say:
- Checksum Processing Efficiency: The AWS S3 Rust SDK was found to be unnecessarily recomputing CRC32C checksums. Identifying this issue led to the implementation of a simple workaround, further improving efficiency.
Thank you for the feedback! Quickly worked with the S2 team to get the screenshot from the change added (it's just enabling the hardware acceleration feature in the sha2 crate)!
