For mathematicians studying Transformers, a very nice concise, precise and complete definition: apronus.com/math/transform…
Lukasz Kaiser
1,346 posts
Lukasz Kaiser
@lukaszkaiser
- I'm so happy to see o1 launch! Leading this research with my colleagues for almost 3 years and working on related ideas even longer convinced me: it's a new paradigm. Models that train hidden CoTs are more powerful than raw Transformers, learn from less data, generalize better.We're releasing a preview of OpenAI o1—a new series of AI models designed to spend more time thinking before they respond. These models can reason through complex tasks and solve harder problems than previous models in science, coding, and math. openai.com/index/introduc…
- OpenAI is nothing without its people
- Believe in MIRAclesWe have reached an agreement in principle for Sam Altman to return to OpenAI as CEO with a new initial board of Bret Taylor (Chair), Larry Summers, and Adam D'Angelo. We are collaborating to figure out the details. Thank you so much for your patience through this.
- Replying to @apples_jimmyOn the 12th day of shipmas Santa brought to me... AGI in a model called ooo3
- o1 is the start of a new paradigm; a lot of work remains which makes it so exciting to do research in this domainReplying to @rao2ztldr; LRM o1's AlphaGo-like RL-training & inference, if only on pseudo CoT moves rather than task specific ones, does seem to lift it beyond the approximate retrieval nature of LLMs to a sort of approximate reasoning. This is however sans guarantees, and orders higher cost. 9/
- So long AIME, we hardly used you as a benchmark and now you're gone... At least in a good company with ARC-AGI.Day 12: Early evals for OpenAI o3 (yes, we skipped a number) openai.com/12-days/?day=12
- When you know the right CoT you can compute anything. As for learning good CoTs, o1 is a start :)What is the performance limit when scaling LLM inference? Sky's the limit. We have mathematically proven that transformers can solve any problem, provided they are allowed to generate as many intermediate reasoning tokens as needed. Remarkably, constant depth is sufficient.
- Honored to have been awarded the 2024 @NEC C&C Prize together with my wonderful friends and Transformer coauthors @ashVaswani@NoamShazeer@nikiparmar09@ilblackdragon@aidangomez@kyosu@YesThisIsLion Learn more:
- Would 500/700 employees of your company sign to quit to fight for it? Between 1am and 5am on a Sunday night before a company wide holiday on Thanksgiving week?
- We're only beginning to understand this new paradigm of CoT-LLMs. There're so many new phenomena to study, research on it will be very exciting. You know it's a start of something good when your first model (with extra tuning) gets 93% on AIME’24 and does IOI-level coding :).introducing our new reasoning models: o1-preview and o1-mini, available in the API today (no waitlist) for tier 5 users: openai.com/index/learning…
- This is one of my favorite ways to evaluate models and a capability that I'm very much looking forward to have in our models.We’re releasing a new benchmark, MLE-bench, to measure how well AI agents perform at machine learning engineering. The benchmark consists of 75 machine learning engineering-related competitions sourced from Kaggle. openai.com/index/mle-benc…
