I have been workong a lot on explainable AI (XAI) techniques as a part of my proposed speech in Py Delhi.
I found that there is a lot of gaps in our understanding of AI models, most of our understanding is guesswork.
Some smart people told me that it is by design, that we are not supposed to be aware of how models operate, but I am not satisfied. There is some research to unveil the inference paths and token influences - but most of it is through observation. We dont have a lot of real insight into how complicated models operate. I have been reading and reading and I looking at code of libraries like transformerlens and circuit tracer. I think we need to have better tools - so I started collecting some work under a website I launched - modelrecon.com. I am very positive about it and hope to make some impact through some tools and techniques I might end up identifyng and implementing. This is the closest I have gotten to building a company!!